Build & maintain complex distributed systems
October 1–2, 2017: Training
October 2–4, 2017: Tutorials & Conference
New York, NY

Drinking from the fire hose: Building a massive-scale monitoring stack

Robert Claire (Pinterest)
11:35am12:15pm Wednesday, October 4, 2017
Real time, events, streams & scale
Location: Grand Ballroom West
Average rating: ****.
(4.00, 1 rating)

Who is this presentation for?

  • Software, DevOps, and infrastructure engineers

Prerequisite knowledge

  • Familiarity with distributed systems, monitoring, and site availability

What you'll learn

  • Understand best practices for managing, storing, and distributing a heavy volume of real-time monitoring data
  • Learn how technologies like Spark Streaming, Kafka, and HBase are used to address specific problems
  • Discover the trade-offs between cost, performance, and storage requirements


At companies like Pinterest, every minute of downtime results in a bad customer experience for thousands of users. Rob Claire explores the technical challenges and lessons learned in building a monitoring stack that can reliably process millions of events per second, covering specific technologies—including Spark Streaming, Kafka, and HBase—and best practices for managing and monitoring data.

Topics include:

  • Real-time data processing with Kafka and Spark Streaming
  • Building a distributed and robust data tier from multiple HBase clusters
  • Managing a cloud-based monitoring stack
  • Understanding the trade-offs between cost, performance, and storage requirements
Photo of Robert Claire

Robert Claire


Rob Claire is an engineer on the visibility team at Pinterest, where he focuses on extracting insight from real-time operational data. Rob has more than 17 years of experience in the fields of data engineering, DevOps, and performance tuning. His career has included stints at One King’s Lane, Slide, Ning, and eBay.