Case Study

Yahoo View Experiments

Site

view.yahoo.com

Yahoo View was launched in the summer of 2016, and since then, the team has been building the core features to provide the best TV-watching experience online. Through user research, we found that community ideas could be the differentiator for Yahoo View. This project is to form testable hypotheses on the community ideas, create the corresponding concepts or designs, test the concepts in a production environment or UX research lab and learn from the experiments.

My role

I joined as a design lead in this initiative. My responsibilities include:

 

HIGH-LEVEL USER FLOW DIAGRAMS

Before jumping into designs, I created high-level user flows to illustrate where we can make the most impact on the user journey and how we can test the hypotheses.

 

EXPERIMENT PLANNING

My job was to find the best way which requires the smallest amount of effort and time to test the ideas. The experiments could be a fake door test or A/B test in production, or a user test in our UX research lab. I then created the appropriate wireframes and interactive prototypes for the tests. Before launching the experiments, the engineering lead and I decided on the metrics we needed to track.

 

EXPERIMENT TRACKING

After the experiments are implemented, I tracked the results in Google Analytics with the appropriate metrics. I captured the metrics in the spreadsheet and made product decisions based on the results.

Community ideas

Under the community ideas, we had three themes:
 

  1. Conversation
    Hypothesis: New generated content (comments or otherwise) is another source of user value.

     

  2. Watch together
    Hypothesis: Bringing people into a digital space to watch videos together could drive more conversations.

     

  3. Trending right now
    Hypothesis: User engagement-based videos help users get a sense of people and add organic validation.

 

Under each theme, we had several product ideas. All these experiments are supposed to use the smallest amount of effort.

Theme #1: Conversation

Allow comments

Yahoo View has the homepage, show page, episode page, and movie page. Where should we add the commenting tool on Yahoo View? I created the user journey map below to illustrate the pros and cons of each place. 

 

Based on this user journey map, the team and I thought adding the commenting tool on the episode page would have the strongest impact on our experiment. As a result, we added Canvass , a commenting tool developed by Yahoo, to the episode page of Yahoo View. 

 

In this experiment, the A version did not have a commenting tool, and the B version had a commenting tool. Then, we split the traffic 50/50. We tracked the percentage of users who engage with the commenting system, as well as the users who saw the commenting system, and see if they have a longer time on site and more videos per session.

The verdict

Though some users really enjoyed leaving comments, the usage was very low. Moreover, we did not see an increase in engagement for the users who saw the commenting tool. We further tested different titles to encourage users to leave comments. However, the numbers remained flat. Considering the engineering maintenance cost and the low usage, as a team, we decided to drop the commenting feature.

 

For other ideas under the conversation theme, we interviewed users in the UX lab. Users ranked them low on the product feature list. We decided not to further test them in production.

 

Based on the evidence, "conversation" may not be the right product direction at this moment.

Theme #2: Watch together

Watch party

This idea ranked the highest on the product feature list in our user research, and much academic research has supported this idea. Therefore, we decided to test it in production to verify if this is a feature that users would use. Since this feature can take engineers months to build, I created a fake door test. A fake door test is to show users options that don't exist, and we track the clicks that users take.

Public watch party fake door
public_watch_together_sketch.png

Sketch of the public watch party fake door

public_watch_together_wireframe.png

Wireframe of the public watch party fake door

Private watch party fake door

One layout exploration of having the private watch party window open and sticky to the bottom right window.

Final wireframe. To maximize the effectiveness of the fake door test, we decided to go with this layout. In this layout, the live chat window covers 30% of the viewing video and is open by default.

The verdict

We opened 100% of traffic to the experiment for 24 hours and tracked the click-through rate (CTR) on the buttons of the public watch party and the share buttons of the private watch party. The public watch party had a higher CTR than the private watch party. One possible reason for it was that users did not expect to be able to start a private watch party on Yahoo View. Also, users would need to coordinate the schedule with other viewers before beginning a private watch party, which might not be convenient for users at the time. As for the public watch party experiment, we had about 5% of users click on the button. It wasn't low enough to dismiss the feature but also not high enough to justify the engineering cost. As a result, the team and I decided to put this feature on the backlog with a lower priority.

Theme #3: Trending right now

Watching now

To test the idea of "what people are watching right now," we had three buckets. Bucket A was the control. Bucket B was adding a strip called "What People Are Watching Right Now!" with the number of current watchers. Bucket C added the same strip as bucket B but instead had the total number of views. We compared all three buckets to see which ones get more user engagement.

View_Desktop_TrendingInComm_v1.jpg

Bucket B and bucket C

Top videos

All the strips on the Yahoo View homepage were horizontal, which couldn't convey the idea of ranking. Therefore, I explored different layouts to test the concept of top videos. We also tested this idea on the homepage.

View_Desktop_TrendingInComm_Phase2_List_

We added this section on the homepage and ran as an experiment.

The verdict

Surprisingly, none of the experiments moved the metrics. One possible reason was that users might only be interested in the shows they've already seen. Therefore, most watched videos by others don't affect users' watching decisions. Further user studies need to be done to understand the reason why it didn't perform as we expected.​ The team and I decided to put this idea in the backlog and explore other ways to test it in the future.

Takeaways

Based on the experiments, we see that users are interested in sharing and discussing the shows at a personal level, but they wouldn't be drawn to the shows when others watch them. Overall, we see some tractions in the community ideas, but not enough to call it the next product direction.

We still have a few community ideas that can be further tested, but the plan now is to go back to the UX research lab and explore a different direction. As a designer, it's important to know that some ideas may be tested well in the UX research lab, but it doesn't necessarily mean the features would perform well in production. I wouldn't call this community initiative a success, but that is ok. The key to innovation is "test, fail, learn and repeat!"