Presented By O’Reilly and Intel AI
Put AI to Work
April 29-30, 2018: Training
April 30-May 2, 2018: Tutorials & Conference
New York, NY

Executive Briefing: Lean AI product development (and common pitfalls)

John Lewin (Microsoft)
2:35pm–3:15pm Wednesday, May 2, 2018
AI Business Summit, AI in the Enterprise
Location: Regent Parlor
Average rating: *****
(5.00, 2 ratings)

What you'll learn

  • Understand that to build a great product that people love, the metrics you define and how you organize your team will make as much of an impact on success as the technology


Great AI products are more than technology; they are built on a clear model of customer success. Getting that model right can be more challenging than building the AI models themselves, and getting it wrong can be very expensive. Shane Lewin outlines common pitfalls in defining AI products and explains how to organize teams to solve them.

Shane shares a process for defining what you are trying to optimize in a way that optimizes customer success but is also computationally tractable. It’s not just the data scientists who need to be thinking about this but also whomever is leading product, because that the choice impacts the kind of product you end up building. Shane also points out that the quality of a model is often not as important as the speed at which a model improves, and that the latter is as much of a function of data infrastructure and measurement as it is about team size. A good solution is not to try to make a perfect AI but an AI that can perform as good as a human, as Facebook is (re)popularizing with the Wizard of Oz studies. But putting a human in the mix has implications on the model choice and architecture that need to be considered carefully. There are no obvious solutions here, but Shane explores some of the current thinking.

Examples include:

  • An image search engine with metrics that kept getting better even though the customers were not being retained
  • A medical AI company with an entirely different problem but an identical experience
  • A five-person team with moderate AI experience that outperformed a 30-person team of researchers who mostly held relevant PhDs
  • An artificially intelligent interior designer
Photo of John Lewin

John Lewin


Shane Lewin is principal manager of AI and research at Microsoft. Shane has been shipping products in technology and AI for over 10 years, with experience spanning everything from large companies such as Netflix and Microsoft to early-stage startups, including Gliimpse and OpenTalent, to companies making the transition from smaller company to large organization, such as Powerset and Shutterstock. Most recently, he was vice president of product for data science at Lumiata. His portfolio includes a medical AI engine that identifies high-risk patients to enable early intervention; an image search engine that allows customers to search by emotion, mood, and context; a self-learning customer communication and email platform that increased retention while reducing total emails; a data optimization platform that saved over $1M per year on a nearly 10x cost reduction; and the massive distributed document summarization engine that generates all the text you see on Bing. Shane holds an MS in computational and mathematical engineering from Stanford University and degrees in molecular biology and mathematics from the University of Colorado at Boulder.