Presented By O’Reilly and Cloudera
Make Data Work
March 5–6, 2018: Training
March 6–8, 2018: Tutorials & Conference
San Jose, CA

Deploying and monitoring interactive machine learning applications with Clipper

Dan Crankshaw (UC Berkeley RISELab)
11:50am12:30pm Wednesday, March 7, 2018
Average rating: ****.
(4.25, 4 ratings)

Who is this presentation for?

  • Data scientists, data engineers, and anyone working with machine learning platforms

Prerequisite knowledge

  • Basic experience developing and training machine learning models and building interactive applications (such as interactive web apps)
  • Familiarity with Docker and Kubernetes (useful but not required)

What you'll learn

  • Understand the systems and DevOps challenges of deploying low-latency machine learning applications
  • Learn how Clipper helps solve these challenges and how to incorporate Clipper into existing serving infrastructure


Machine learning is being deployed in a growing number of applications that demand real-time, accurate, and robust predictions under heavy serving loads. However, most machine learning frameworks and systems only address model training, not deployment.

Clipper is an open source, general-purpose model-serving system that addresses these challenges. Interposing between applications that consume predictions and the machine learning models that produce predictions, Clipper simplifies the model deployment process by adopting a modular serving architecture and isolating models in their own containers, allowing them to be evaluated using the same runtime environment as that used during training. Clipper’s modular architecture provides simple mechanisms for scaling out models to meet increased throughput demands and performing fine-grained physical resource allocation for each model. Further, by abstracting models behind a uniform serving interface, Clipper allows developers to compose many machine learning models within a single application to support increasingly common techniques such as ensemble methods, multiarmed bandit algorithms, and prediction cascades.

Dan Crankshaw offers an overview of the Clipper serving system and explains how to use it to serve Apache Spark and TensorFlow models on Kubernetes. Dan concludes by discussing some recent work on statistical performance monitoring for machine learning models.

Photo of Dan Crankshaw

Dan Crankshaw

UC Berkeley RISELab

Dan Crankshaw is a PhD student in the CS Department at UC Berkeley, where he works in the RISELab. After cutting his teeth doing large-scale data analysis on cosmology simulation data and building systems for distributed graph analysis, Dan has turned his attention to machine learning systems. His current research interests include systems and techniques for serving and deploying machine learning, with a particular emphasis on low-latency and interactive applications.