Copyright © Julia Wang, 2019

Case Study

Yahoo View Experiements

Site

view.yahoo.com

Yahoo View was launched in the summer of 2016, and since then, the team had been building the core features to provide the best TV watching experience online. Through user research, we found the community ideas could be the differentiator for Yahoo View. This project is to form testable hypotheses on the community ideas, create the corresponding concepts or designs, test the concepts in production or UX research lab and learn from the experiments.

My role

I joined as a design lead in this initiative. My responsibilities include:

 

HIGH-LEVEL USER FLOW DIAGRAMS

Before jumping into designs, I created high-level user flows to illustrate where we can make the most impact on the user journey, and how we can test the hypotheses.

 

EXPERIMENT PLANNING

My job was to find the best way which requires the smallest amount of effort and time to test the ideas. The experiments could be fake door test or A/B test in production, or user test in our UX research lab. I then created the appropriate wireframes and interactive prototypes for the tests. Before launching the experiments, the engineering lead and I decided on the metrics we needed to track.

 

EXPERIMENT TRACKING

After the experiments are implemented, I tracked the results in Google Analytics with the appropriate metrics. I captured the metrics in the spreadsheet and made product decisions based on the results.

Community ideas

Under the community ideas, we had three themes:

  1. Conversation
    Hypothesis: New generated content (comments or otherwise) are another source of value for the users.

  2. Watch together
    Hypothesis: Bringing people into a digital space to watch videos together could drive more conversations.

  3. Trending right now
    Hypothesis: User engagement based videos help users get a sense of people as well as add organic validation.

 

Under each theme, we had several product ideas. All these experiments are supposed to use the smallest amount of effort

Theme #1: Conversation

Allow comments

Yahoo View has the homepage, show page, episode page and movie page. Where should we add the commenting tool on Yahoo View? I created the user journey map below to illustrate the pros and cons of each place. 

 

Based on this user journey map, the team and I thought adding the commenting tool at the episode page would have the strongest impact in our experiment. As a result, we added Canvass , a commenting tool developed by Yahoo, to the episode page of Yahoo View. 

 

In this experiment, the A version did not have a commenting tool, and the B version had a commenting tool. Then, we split the traffic 50/50. We tracked the percentage of users who engage with the commenting system, as well as the users who saw the commenting system and see if they have longer time on site and more video per session.

The verdict

Though some users really enjoyed leaving comments, the usage was very low. Moreover, we did not see an increase in the engagement for the users who saw the commenting tool. We furthered tested different titles to encourage users to leave comments. However, the numbers remained flat. Considering the engineering maintenance cost and the low usage, as a team, we decided to drop the commenting feature.

 

For other ideas under the conversation theme, we interviewed users in the UX lab. Users ranked them low on the product feature list. We decided not to further test them in production.

 

Based on the evidence, "conversation" may not be the right product direction at this moment.

Theme #2: Watch together

Watch party

This idea ranked the highest on the product feature list in our user research, and there had been much academic research to support this idea. Therefore, we decided to test it in production to verify if this is a feature that users would use. Since this feature can take engineers months to build, I decided to create a fake door test. A fake door test is to show users options that don't really exist, and we track the clicks that users take.

Public watch party fake door

Sketch of the public watch party fake door

Wireframe of the public watch party fake door

Private watch party fake door

One layout exploration of having the private watch party window open and sticky to the bottom right window.

Final wireframe. To maximize the effectiveness of the fake door test, we decided to go with this layout. In this layout, the live chat window covers 30% of the viewing video and is open by default.

The verdict

We opened 100% of traffic to the experiment for 24 hours and tracked the click-through rate (CTR) on the buttons of public watch party and the share buttons of the private watch party. The public watch party had a higher CTR than the private watch party. One possible reason for it was because users did not expect to be able to start a private watch party on Yahoo View. Also, users would need to coordinate the schedule with other viewers before starting a private watch party, which might not be convenient for users at the time. As for the public watch party experiment, we had about 5% of users clicked on the button. It wasn't low enough to dismiss the feature, but it was also not high enough to justify the engineering cost. As a result, the team and I decided to put this feature on the backlog with a lower priority.

Theme #3: Trending right now

Watching now

To test the idea of "what people are watching right now", we had three buckets. Bucket A was the control. Bucket B was adding a strip called "What People Are Watching Right Now!" with the number of current watchers. Bucket C was adding the same strip as bucket B but instead having the total number of views. We compared all three buckets to see which one get more user engagement.

Bucket B and bucket C

Top videos

All the strips on Yahoo View homepage were horizontal, which couldn't convey the idea of ranking. Therefore, I explored different layouts to test the idea of top videos. We also tested this idea on the homepage.

We added this section on the homepage and ran as an experiment.

The verdict

Surprisingly, none of the experiments moved the metrics. One possible reason was users may only be interested in the shows they've already seen. Therefore, most watched videos by others don't affect users' watching decision. Further user studies need to be done to understand the reason why it didn't perform as we expected.​ The team and I decided to put this idea in the backlog and explore other ways to test it in the future.

Takeaways

Based on the experiments, we see users are interested in sharing and discussing the shows at the personal level, but they wouldn't be drawn to the shows when others watch it. Overall, we see some tractions in the community ideas, but not enough to call it the next product direction.

We still have a few community ideas that can be further tested, but the plan now is to go back to the UX research lab and explore a different direction. As a designer, it's important to know that some ideas may be tested really well in the UX research lab, but it doesn't necessarily mean the features would perform well in production. I wouldn't call this community initiative is a success, but that is ok. The key to innovation is "test, fail, learn and repeat!"