Transforming a machine learning prototype to a deployable solution leveraging Spark in healthcare

Description

Extensive research has been conducted at the intersection of machine learning and healthcare. With an anticipated 48% annual growth in healthcare data, building scalable healthcare solutions is more crucial now than ever before. Fortunately, there has been an equivalent surge in the algorithms, software packages, optimization techniques, and hardware available for obtaining insights on healthcare data.

However, despite these advances, minimal research has been conducted to understand the challenges and considerations associated with transforming research prototypes into real-world healthcare solutions. Often machine learning research is conducted in silos: researchers build prototypes, which are subsequently picked up by software developers who develop scalable solutions for the prototype. There is a gap in the understanding of trade-offs to consider to transition from a research prototype to a deployable healthcare solution.

Rachita Chandra outlines challenges and considerations for transforming a research prototype built for a single machine to a deployable healthcare solution that leverages Spark in a distributed environment. The original prototype, which tackled prediction of healthcare costs, worked well for a dataset of 5 million users in a nondistributed environment. It utilized several Python data science libraries and machine learning models. However, as the dataset grew larger (> 1 TB), computational resources became the bottleneck, and the need to adopt Spark became apparent. Since the research prototype was in its mature phases, porting components of the existing pipeline to leverage Spark was more effective than building a Spark codebase from scratch. The deployable solution is an end-to-end multitenant enterprise application comprising of several components: user authentication, request handling, data transformations, quality checks, analytics, machine learning modules, a visualization interface, and error handling.

Topics include:

• The process of finding compatible modules and mapping machine learning libraries from a nondistributed environment to a distributed environment
• Architectural changes and additional modules needed in the codebase to enable the application to be deployed
• A comparison of the machine learning model implementations across the distributed and non-distributed system and the performance benchmarks obtained in both environments
• The debugging challenges faced and tools utilized to identify and circumvent these errors