Put AI to work
June 26-27, 2017: Training
June 27-29, 2017: Tutorials & Conference
New York, NY

Strategies for integrating people and machine learning in online systems

Jason Laska (Clara Labs)
11:55am12:35pm Thursday, June 29, 2017
Implementing AI
Location: Sutton Center/North Level: Intermediate
Secondary topics:  Machine Learning, Natural Language
Average rating: *****
(5.00, 1 rating)

Prerequisite Knowledge

  • A basic working knowledge of how machine learning problems are set up—namely, what is meant by annotations for training data

What you'll learn

  • Learn some of the common pitfalls in building a real-time constrained human-in-the-loop system and approaches to these problems


Clara Labs is an email-based scheduling service for busy people. Simply “cc” Clara on an email to a person you want to meet with, and Clara handles the back-and-forth game of email tag for you. To build a robust and accurate system that gracefully handles nuanced requests, Clara Labs combined machine learning with a distributed human labor force. This service, available 24/7, consistently responds within 30 minutes or less and enables a single person to do work for an unbounded number of customers.

A hybrid person-machine system has clear benefits, such as increased accuracy and decreased cost (i.e., increased scalability) via partial automation. Further, human input to the system leads to new annotations for retraining algorithms. There are great advantages to vertically integrating the ML annotation process directly with the product (e.g., the fidelity of labeled data increases when the annotator understands what actions will be derived directly from their work).

Despite these advantages, there are several distinct challenges to building such a system: annotators are noisy and may be biased by bad ML predictions (if displayed). There also tends to be an inverse relationship between speed of data entry and annotator accuracy, and the learning curve for using a unique expert system may be high. In fact, simply measuring accuracy in the system may be challenging depending on time and cost constraints.

Jason Laska explores the challenges of building a real-time(ish) knowledge workforce, how to integrate automation, and key strategies Clara Labs learned that enable scale. Along the way, Jason discusses incentives and algorithms for increasing both the accuracy and speed of human operators and for measuring their performance, strategies for dealing with task ambiguity, and tricks for building an effective ramping system to onboard workers. Jason also covers the “automation spectrum” (i.e., the integration points where machine learning predictions can be used to dramatically enhance human performance).

Photo of Jason Laska

Jason Laska

Clara Labs

Jason Laska is the head of engineering at Clara Labs. Previously, he spearheaded the computer vision program at Dropcam (acquired by Google in 2014), developing massive scale online vision systems for the product. Jason holds a PhD in electrical engineering from Rice University, where he made contributions to inverse problems, dimensionality reduction, and optimization. He briefly dabbled in publishing as a cofounder and editor of Rejecta Mathematica, a publication for previously rejected mathematics articles.