Digging for gold: Developing AI in healthcare against unstructured text data

Description

Healthcare faces significant growth in both structured and unstructured data, with challenges including diverse data formats, interoperability, regulatory requirements, and the need for advanced analytics. Today the industry is a $3 trillion market employing the use of AI, big data, and data across all formats and varieties being to drive innovative solutions that can improve patient care, increase efficiencies, and financial outcomes.

Over the years, electronic healthcare records (EHRs) have become more sophisticated and feature rich, but despite these advances, doctors still enjoy the simplicity of summarizing their patient feedback in freeform clinical notes. Unfortunately, many of these notes are coded properly after the fact, leaving critical information forgotten and potentially locked away in the patient record. This includes valuable information such as medications, symptoms, and diagnoses—all of which could be used to inform future care. The unstructured nature of clinical notes makes it difficult to process this data, join it with other high-value healthcare datasets such as pathology, genomics or billing records, and produce vital analytics such as key performance indicators (KPIs) or risk predictions. All of this can result in poor patient outcomes and lost opportunities for revenue or operational efficiencies.

Chiny Driscoll and Jawad Khan offer an overview of a solution by Cloudera and MetiStream that lets healthcare providers automate the extraction, processing, and analysis of clinical notes within an electronic health record in batch or real time, improving care, identifying errors, and recognizing efficiencies in billing and diagnoses. The solution leverages key industry open source solutions such as cTAKES, a natural language processing (NLP) suite developed for the extraction of information from clinical free-text in EHR solutions. cTAKES uses the Unstructured Information Management Architecture (UIMA) framework for implementing its NLP pipeline, which annotates free text to discern clinical terms and then normalizes these terms to well-known ontology codes (notably UMLS CUI, Snomed-CT, and RxNorm).

Used in conjunction with NLP and Apache Spark, years of unstructured text can be processed within hours. Integration with Spark also allows the processed notes to capitalize on the powerful and inherent machine learning capabilities available through Spark MLlib and ML Pipelines, enabling annotated clinical data to be used to train a model and develop risk predictions. The output of insight from the predictive models and AI can then be leveraged to easily distribute and operationalize these models throughout the enterprise.

Topics include:

• How to automatically structure, annotate, and index clinical notes at scale
• How to process data in real time or batch, including multiyear historical loads
• How to conduct free-form text search with millisecond response times
• How to build machine learning models against the output of the structured text
• How to build and deploy advanced analytics within clinical workflows