Executive Briefing: Lean AI product development (and common pitfalls)

Description

Great AI products are more than technology; they are built on a clear model of customer success. Getting that model right can be more challenging than building the AI models themselves, and getting it wrong can be very expensive. Shane Lewin outlines common pitfalls in defining AI products and explains how to organize teams to solve them.

Shane shares a process for defining what you are trying to optimize in a way that optimizes customer success but is also computationally tractable. It’s not just the data scientists who need to be thinking about this but also whomever is leading product, because that the choice impacts the kind of product you end up building. Shane also points out that the quality of a model is often not as important as the speed at which a model improves, and that the latter is as much of a function of data infrastructure and measurement as it is about team size. A good solution is not to try to make a perfect AI but an AI that can perform as good as a human, as Facebook is (re)popularizing with the Wizard of Oz studies. But putting a human in the mix has implications on the model choice and architecture that need to be considered carefully. There are no obvious solutions here, but Shane explores some of the current thinking.

Examples include:

• An image search engine with metrics that kept getting better even though the customers were not being retained
• A medical AI company with an entirely different problem but an identical experience
• A five-person team with moderate AI experience that outperformed a 30-person team of researchers who mostly held relevant PhDs
• An artificially intelligent interior designer