Presented By O’Reilly and Intel Nervana
Put AI to work
September 17-18, 2017: Training
September 18-20, 2017: Tutorials & Conference
San Francisco, CA

Next-generation intelligent applications require cognitive design.

John Whalen (Brilliant Experience)
4:00pm–4:40pm Tuesday, September 19, 2017
Interacting with AI
Location: Franciscan AB
Secondary topics:  Data and training, Visualization and Interfaces
Average rating: ****.
(4.25, 4 ratings)

Prerequisite Knowledge

  • Experience designing or using an AI system such as Siri, Cortana, Alexa, or Google Assistant

What you'll learn

  • Learn how to "usability test" AI interactions with humans and measure success
  • Understand the two distinct ways that humans construct commands to AI systems and how, using physiological measurements, you can measure the human response to the AI system responses


John Whalen explores the concept of cognitive design, describing how humans structure their commands to AI systems (syntax, word usage, prosody) and how to measure human reactions to AI responses using biometrics (facial emotion recognition, heart rate, GSR). Along the way, John shares insights into how to optimally architect the customer experience.

John offers an overview of the results of an evaluation of four major AI systems (Siri, Cortana, Alexa, and Google Assistant), tested by the young and old, those new to AI systems and those that use these tools every day, native and non-native speakers, and techies and non-techies. Each were asked to interact with the systems to request facts, complex information, jokes, commands, and calendar information while the evaluators recorded their commands, the AI response, and the human’s physiological response to the AI response (facial emotion, heart rate, and GSR).

There were several intriguing findings:

  1. There were two distinct ways humans constructed commands for the AI systems.
  2. The success of the systems did vary by age, but not in the way you might expect.
  3. The testers’ favorite AI systems were not always the ones that performed the best in terms of giving correct answers.
  4. There was a distinct physiological signature associated with a positive experience.

John explains how these findings can help you determine how you should measure the success of your AI system or chatbot and suggests new ways to predict market success that go beyond AI answer accuracy.

Photo of John Whalen

John Whalen

Brilliant Experience

John Whalen leads Brilliant Experience, a consultancy that supports intra- and entrepreneurs to ensure the success of mission-critical innovation projects, using a unique blend of user-centered design, psychology, design thinking, and Lean Startup techniques. John draws on his over 15 years of user-centered design experience to help large enterprises integrate brain science into Agile, design thinking, and UCD projects. His specialty is providing businesses with competitive advantages using a mix of user research insights and expert knowledge of human vision, attention, and memory. John has experience (and great stories to tell from) working with Fortune 500 clients in the ecommerce, financial, healthcare, and government verticals. He holds a PhD in cognitive science.