Recommendation Engines: Going Beyond the Social Graph

Hunter Walk. Leads product team at YouTube.
Tom Conrad. Product engineering at Pandora.
Garrett Camp. Cofounder of Stumble Upon.
Lior Ron from Google’s Hotpot project. Works on local recommendation engine.
Liz Gannes, journalist, moderating.

Conrad: Pandora has 8 billion thumbs up/thumbs down data, completely contextualized.

Walk says that YouTube knows a lot about where their videos are embedded. Talks about could personally review videos, or use algorithm to analyze videos, but they are also look at what the top blogs/sites are pulling from YouTube to understand what videos are popular with whom.

Walk: About 50% of searches on YouTube are “broad,” meaning the person is looking for an experience, not a particular video. Google has to figure out what the best videos are to help someone understand/experience a topic. It’s very different from trying to answer a question, like we think of in traditional search.

Camp: We want to get away from 10 blue links. We want to be surprised, have serendipitous experience.

Conrad: Looking at most common starting points a couple of years ago. One of top ones was called Christmas. The station was seeded with an indie rock band called Christmas. Oops. So then they started playing the station to see what happened, and it was playing all holiday music. The crowd had very quickly weeded out the data error by thumbing down the band on the holiday station.

Walk shares story of a teenager who told them that she wanted to know what her friends hadn’t watched yet on YouTube, so she knew what to share. It’s a hard problem, but they want to figure out what’s not yet spreading, but will.

Camp: StumbleUpon tests new, non-socially-recommended stuff in streams to figure out this kind of question. When you’re just looking at the social graph, you’re in a closed loop.

Ron: Social is really important in recommendations because of the trust factor. Getting a friend recommendation still beats the site telling you, hey “other people like you” like this.

Walk: Early on, they just trusted what the uploader said about what the video is. Now, they use a lot of technology to understand a piece of content. What does perfect metadata look like? What’s everything I COULD know about it? And then the challenge is, you take all that data to try to create an experience, not just spit out data.

Walk: One of biggest changes in perceived search relevance was when they started showing context for recommendations. Immediately, people thought recommendations were more relevant. And two, if the recommendation was wrong, they blamed themselves, not YouTube.

Conrad: Pandora has broken out of the PC into mobile and now car implementations. The difference in environment between listening at work via headphones to listening in the car with the whole family and lots of people making the music decisions is very complex.

Camp: StumbleUpon is often a free-time application, so their new mobile app [6mo old] is doing well.

Ron: Interesting patterns in how people are following people for recommendations. Some people follow only celebrities, for instance.

Camp: For analytics, they look at thumbs up/thumbs down, length of time on resource, comparing time to type of resource. SU has an 80-85% thumbs up rate.

Walk: You have to be careful with analytics. You don’t want to introduce features that push up your positive stats to the detriment of user experience.

Conrad: They religiously test all changes to the algorithms now, after making several changes in early days that “everyone” agreed would be great, that instead tanked numbers.

Ron: Recommendation is very vertical-oriented. The required data is so specific that it’s hard to have a general recommendation engine.

Camp: Also, UI affects what kind of data you get a lot, so that’s part of why people build the engines themselves.

Ron: We’re not living in a world yet where we’re bombarded by awesome recommendations and we have to tune them. Part of the problem right now is getting coverage for everywhere.

Camp: We do a combination of social and similarity in your recommendation list.