Presented By O’Reilly and Intel AI
Put AI to work
Sep 4-5, 2018: Training
Sep 5-7, 2018: Tutorials & Conference
San Francisco, CA

How Captricity built a human-level handwriting recognition engine using data-driven AI

Ramesh Sridharan (Captricity)
4:50pm-5:30pm Thursday, September 6, 2018
Implementing AI
Location: Yosemite BC
Secondary topics:  Computer Vision, Platforms and infrastructure

Who is this presentation for?

  • Engineers (machine learning, application development, DevOps, etc.), product managers, and engineering managers

Prerequisite knowledge

  • Familiarity with AI, machine learning, and the software development lifecycle

What you'll learn

  • Discover how Captricity built a machine learning pipeline that can read handwriting at human-level accuracy
  • Learn how to use metrics and data to guide productionizing machine learning and how to use metrics and data to identify problems worth solving with machine learning


The last few years have seen an explosion in products and startups seeking to harness the promise of AI. Building such products presents many challenges during model development, from curating training data and investigating model architectures to training models. Once models are prototyped and developed, productionizing them is equally challenging: excellent performance on held-out test data does not always translate to production environments. Each of these steps can take weeks or even months and can cost tens of thousands of dollars.

Captricity has iterated on this process through the development of many incremental models to solve the problem of recognizing structure and transcribing content of paper forms. This has culminated in a handwriting recognition system that achieves human-level accuracy and speed. A key observation for this approach is that not all problems require machine learning to solve on day one. Human feedback loops and reciprocal data applications are powerful tools to fill in gaps that ML can’t solve yet, and the judicious use of data can identify areas that maximize ML’s impact on a product while minimizing development time and effort. Similarly, human feedback loops are critical after deployment: a well-instrumented model with observable performance metrics is much easier to debug than an opaque one. For example, by identifying key metrics such as customer-level accuracy before deploying, Captricity was able to reduce the development lifecycle time from months to weeks and quickly identify failure cases to drive the next round of research improvements.

Ramesh Sridharan outlines the journey from simple binary classification to handwriting recognition with near-human levels of accuracy at superhuman speeds and shares the successes and failures encountered on the road to this development. Along the way, he discusses the big ideas Captricity learned building and deploying this system, using data to identify specific problems to solve using AI and to evaluate and validate the algorithm itself and the overall system once deployed. Ramesh focuses on how Captricity used data to break down this large problem into manageable subproblems to solve with ML, how instrumenting an ML system with well-chosen metrics can enable debugging and implementation, reducing time to ship, and how the same infrastructure can enable ongoing evaluation and retraining.

Photo of Ramesh Sridharan

Ramesh Sridharan


Ramesh Sridharan is a machine learning engineering manager at Captricity. Ramesh is passionate about using technology for social good, and his research has helped enable a cross-collaboration between researchers and doctors to understand large, complex medical image collections, particularly in predicting the effects of diseases such as Alzheimer’s on brain anatomy. He holds a PhD in electrical engineering and computer science from MIT’s Computer Science and Artificial Intelligence Lab (CSAIL), where his thesis focused on developing machine learning and computer vision technologies to enhance medical image analysis.