Put open source to work
July 16–17, 2018: Training & Tutorials
July 18–19, 2018: Conference
Portland, OR

A voice interface for electronic health records

Ton Ngo (IBM), Yi-Hong Wang (IBM)
11:30am12:00pm Tuesday, July 17, 2018
Location: B115-116
Tags: tensorflow
Average rating: ***..
(3.00, 1 rating)

Electronic health records (EHRs), mandated in the US by the Affordable Care Act, have become an integral component of our medical care system. Ton Ngo and Yi-Hong Wang explain how they are using TensorFlow.js to build a voice interface for a web-based electronic health record system. This project is a part of the Call for Code Initiative announced in May 2018, which calls for using technology to help disaster relief.

The goal of this project is to leverage the advantages of EHRs when a major natural disaster hits a large population in remote area. In these situations, survivors require medical care while dealing with chaotic, primitive conditions. To help nurses and doctors be more effective in providing care, Ton and Yi-Hong envision deploying a local EHR system in the field. However, one of the challenges in operating in this situation is entering patient data into the EHR. Laptops may not be convenient in the field and are likely not available, but smart phones will likely be ubiquitous. Since these devices have microphones and can run a browser, a voice interface to the EHR system should make it possible to enable easy data entry.

To implement this voice interface, TensorFlow.js in a web-based EHR is used to load a speech-to-text model and manage the interaction between the user and the app. The JS interface is specifically designed to fit an appropriate workflow for the medical care providers in the field. The project leverages publicly available deep learning models and datasets, such as the Google tensor2tensor code, the Mozilla Deep Voices project, and the Figure-Eight medical transcription dataset. The model initially will support English, with the possibility for other languages later. Ton and Yi-Hong describe the design, progress, and the challenges they encountered.

Photo of Ton Ngo

Ton Ngo


Ton Ngo is a senior software developer in the IBM Cognitive OpenTech Group at the IBM Silicon Valley Lab. Previously, he was with the IBM Research Lab at Yorktown and Almaden. He’s been active in the open source community for four years and is working on TensorFlow and deep learning. He was a core contributor in OpenStack for Magnum and Heat-Translator, focusing on the networking and storage support for container orchestrator such as Kubernetes. Ton frequently gives talks and programming tutorials on TensorFlow in San Francisco, Seattle, and New York and at OpenStack Summits worldwide. He has published papers on a wide range of subjects.

Photo of Yi-Hong Wang

Yi-Hong Wang


Yi-Hong Wang is a software developer with the Cognitive OpenTech Group at the IBM Silicon Valley Lab, where he focuses on Node.js and JanusGraph open technologies. Recently, he’s been working on TensorFlow, focusing on the build package and TensorFlow JavaScript.