Building a large-scale machine learning application using Amazon SageMaker and Spark

Description

Machine learning’s popularity has grown tremendously in recent years. The drive to integrate machine learning into every solution has never been more pronounced, but the path from investigation to model development to implementation in production can be difficult. Amazon SageMaker AWS’s new machine learning platform seeks to make this process easier.

Machine learning starts with data, and Spark is one of the most popular and flexible solutions for handling large datasets for ETL, ad hoc analysis, and advanced machine learning. However, using Spark for production machine learning use cases can create inconsistencies in algorithm scale, conflicts over cluster resources, and prediction latencies. By offloading training to Amazon SageMaker’s highly scalable algorithms and distributed, managed training environment and deploying with SageMaker’s real-time production endpoints, implementing machine learning in production is easier and more reliable.

David Arpin walks you through building a machine learning application, from data manipulation to algorithm training to deployment to a real-time prediction endpoint, using Spark and Amazon SageMaker.