Spark NLP in action: Improving patient flow forecasting at Kaiser Permanente

Description

Applying natural language processing in practice is nontrivial for two reasons. First, human language is nuanced, fuzzy, and highly contextual, requiring domain-specific models to be trained for most tasks. Second, NLP is usually just part of a bigger machine learning or information retrieval pipeline that solves for a real business use case. Putting together a complete, scalable, performant, measurable, and reproducible pipeline traditionally requires significant engineering compromises.

David Talby and Santosh Kulkarni explain how Kaiser Permanente uses the open source NLP library for Apache Spark to tackle one of the most common challenges with applying natural language process in practice—integrating domain-specific NLP as part of a scalable, performant, measurable, and reproducible machine learning pipeline—and improve the accuracy of forecasting the demand for hospital beds. Accurate forecasting is critical to ensuring that enough beds and nurses are available to take care of incoming patients. While some predictive features are structured, many relevant features are locked in free-text clinical notes. Along the way, David and Santosh explain how Kaiser Permanente’s systems meet the highest standards of robustness, scale, and compliance.

Topics include:

• Extending Spark NLP with domain-specific word embeddings, named entity recognition, and assertion polarity models to match the vocabulary, grammar, and context of clinical notes
• Using the Spark ML Pipeline APIs to define a unified, frictionless pipeline that runs NLP algorithms and then uses them as features when training the forecasting model
• Leveraging Spark’s built-in optimizations and features to deliver a fully distributed, highly performant, serializable, and easily measurable learning pipeline