Clara Labs is an email-based scheduling service for busy people. Simply “cc” Clara on an email to a person you want to meet with, and Clara handles the back-and-forth game of email tag for you. To build a robust and accurate system that gracefully handles nuanced requests, Clara Labs combined machine learning with a distributed human labor force. This service, available 24/7, consistently responds within 30 minutes or less and enables a single person to do work for an unbounded number of customers.
A hybrid person-machine system has clear benefits, such as increased accuracy and decreased cost (i.e., increased scalability) via partial automation. Further, human input to the system leads to new annotations for retraining algorithms. There are great advantages to vertically integrating the ML annotation process directly with the product (e.g., the fidelity of labeled data increases when the annotator understands what actions will be derived directly from their work).
Despite these advantages, there are several distinct challenges to building such a system: annotators are noisy and may be biased by bad ML predictions (if displayed). There also tends to be an inverse relationship between speed of data entry and annotator accuracy, and the learning curve for using a unique expert system may be high. In fact, simply measuring accuracy in the system may be challenging depending on time and cost constraints.
Jason Laska explores the challenges of building a real-time(ish) knowledge workforce, how to integrate automation, and key strategies Clara Labs learned that enable scale. Along the way, Jason discusses incentives and algorithms for increasing both the accuracy and speed of human operators and for measuring their performance, strategies for dealing with task ambiguity, and tricks for building an effective ramping system to onboard workers. Jason also covers the “automation spectrum” (i.e., the integration points where machine learning predictions can be used to dramatically enhance human performance).
Jason Laska is the head of engineering at Clara Labs. Previously, he spearheaded the computer vision program at Dropcam (acquired by Google in 2014), developing massive scale online vision systems for the product. Jason holds a PhD in electrical engineering from Rice University, where he made contributions to inverse problems, dimensionality reduction, and optimization. He briefly dabbled in publishing as a cofounder and editor of Rejecta Mathematica, a publication for previously rejected mathematics articles.
©2017, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org