Build & maintain complex distributed systems
October 1–2, 2017: Training
October 2–4, 2017: Tutorials & Conference
New York, NY

Scaling the machine learning systems powering Quora's home feed

Nikhil Garg (Quora), Neeraj Agrawal (Quora)
4:45pm–5:25pm Wednesday, October 4, 2017
Resilience Engineering, Systems Engineering
 Location: Gramercy

Who is this presentation for?

  • Senior software engineers and engineering leaders

Prerequisite knowledge

  • Experience working with distributed systems

What you'll learn

  • Learn strategies for building and scaling machine learning-based ranking and recommendation systems and tricks and data structures for low-level but important optimizations
  • Explore how Quora runs various open source systems in production and the company's architecture choices

Description

Millions of people visit Quora’s home feed to find engaging high-quality content personalized to their interests. The home feed ranks hundreds of thousands of stories in every request using sophisticated machine learning algorithms that simultaneously optimize for engagement, quality, personalization, and content generation.

Nikhil Garg and Neeraj Agrawal describe the evolution of the home feed’s architecture and share several lessons from building and scaling this system.

Topics include:

  • A high-level overview of the system’s architecture
  • The pros and cons of various approaches of distributing data
  • Various open source systems that Quora used at different points in time
  • Some CPU-cache-aware data structures Quora built in metal-efficient C++
  • Efficient ways of serving very complex machine learning models
Photo of Nikhil Garg

Nikhil Garg

Quora

Nikhil Garg is an engineering manager at Quora, where he has led the quality, ads, and ML platform teams, among others. He is interested in the intersection of machine learning, distributed systems, and human psychology.

Photo of Neeraj Agrawal

Neeraj Agrawal

Quora

Neeraj Agrawal is an engineering manager on Quora’s infrastructure team, where he leads company-wide efforts to make Quora fast and responsive. He started as an engineer working on server-side performance and has since revamped how the company thinks about performance. In his role as the speed lead, he has built the speed team from the ground up and created a culture of “performance matters” at Quora. Previously, he worked on other parts of the product, including writing the first version of Quora for iPhone and building a distributed, real-time ranking service for the personalized news feed product.