Building a robust content recommendation platform for 60 million news readers

Description

In 2016, Tribune Publishing began built an in-house data science team to better leverage its vast datasets with new machine learning and analytics technologies. One of the primary successes of this team was its content recommendation system (“RecSys”), developed entirely in house on top of existing open source systems and new open source libraries created and released by Tribune.

Requirements for the RecSys included the ability to perform A/B/n testing against legacy human-edited and algorithmic recommendations, support multiple publications with both shared and exclusive content, support “real-time” online machine learning at scale, scale without limit in the face of traffic spikes, and gracefully degrade when responses can’t be delivered within a given time limit.

Matt Chapman leads a walkthrough of the lifecycle of the request from the web browser of a news-reading end user to the backend algorithms that generates up-to-the moment, personalized recommendations for what the user might want to read next. Along the way, Matt reviews the challenges that the team faced, the open source solutions used at each step, and the new framework and libraries developed by the team to make development of algorithms and of the system itself fast, flexible, and scalable.

Topics include:

• Docker container orchestration with DCOS: What’s wrong with Kubernetes? (Nothing.)
• Why so much Python? Isn’t it too slow? (No. Well, yes, but it doesn’t matter.)
• Cassandra, Scylla, Memcached, and Redis: Who won the data store shoot-out? (No one.)
• How to pickle TensorFlow models (or anything at all)
• Kafka and when to avoid using a message broker (usually)
• Operations monitoring with Graphite: What metrics matter? (Not many.)
• ZMQ and why microservice architectures aren’t too slow (You’re too slow.)