Feature engineering with Spark NLP to accelerate clinical trial recruitment

Description

Recruiting patients for clinical trials is a major challenge in drug development. Finding patients requires an in-depth understanding of their medical histories and current health statuses while the majority of patient data is unstructured and spread across physician notes, pathology, imaging, genomic, and other reports. For this reason, clinical trial recruitment is a slow and manual process.

Saif Addin Ellafi and Scott Hoch dive into a case study that describes how Deep 6 uses the Spark natural language processing (NLP) platform to apply state-of-the-art deep learning to accurately extract the relevant clinical facts from unstructured text. These facts are then used in subsequent data science pipelines in constructing patients’ medical histories.

John Snow Labs’s NLP library for Apache Spark is an open source library that provides natural language understanding capabilities with state-of-the-art accuracy, performance, and scale. It provides deep learning-based NLP algorithms for named entity recognition, spell checking, sentiment analysis, assertion status detection, entity resolution, optical character recognition (OCR), and sentence segmentation, and it enables highly efficient training of domain-specific machine learning and deep learning NLP models.

They explain how Deep 6 uses Spark NLP to scale its training and inference pipelines to millions of patients while achieving state-of-the-art accuracy. They explore the technical challenges, the architecture of the full solution, and the lessons Deep 6 learned that you can directly apply to your next natural language understanding project.