Building turnkey recommendations for 5% of internet video

Description

JW Player—the world’s largest network-independent video platform, representing 5% of global internet video—provides on-demand recommendations as a service to thousands of media publishers, driving higher engagement and retention among their viewers. For the thousands of publishers using this service, sites such as Business Insider, Refinery29, Hearst, and USA Today, this translates directly to increased advertising dollars and is thus a major focus for algorithmic improvement on the part of JW Player’s data science team.

Nir Yungster and Kamil Sindi explain how the company is systematically improving model performance while navigating the many engineering challenges and unique needs of the diverse publishers it serves.

Topics include:

• Building an MVP service that is scalable and tuned to each publisher’s content and use case, including both associative-based models built on a Kafka-Storm-Redis pipeline and a content-based approach that relies on Elasticsearch for delivery, with a high-availability service handling 20K requests a second.
• Building an offline-online iteration process to systematically improve performance: Some of the more interesting discussion areas here include how JW Player defines model performance metrics for thousands of publishers with differing content and viewer contexts and A/B testing that shows that changes in algorithms can lead to dramatically different outcomes depending on the publisher. Some of those changes are inherent to the content type (e.g., visitors to a news site like USA Today will naturally be more sensitive to the age of content than visitors to a general information site like Investopedia, where the relevance of content is stable over time). Other differences arise from implementation differences (e.g., the success of content-based recommendations can depend on the diversity of a publisher’s library or the quality of titles and descriptions from its editors).
• Using a deep learning framework to build personalized recommendations: The reasons for making this change include helping unify content and association-based approaches (instead of separate models, they are features/inputs); providing a flexible platform for adding new features (e.g., NLP-based video categorization and viewer preferences and NLP/audio/visual-based psychographic features for videos, such as valence, arousal, and dominance—and viewer preferences for each of these); and using A/B testing to help refine offline training metrics for deep learning models.