Presented By
O’Reilly + Cloudera
Make Data Work
March 25-28, 2019
San Francisco, CA

Serverless workflows for orchestration hybrid cluster-based and serverless processing

Rustem Feyzkhanov (Instrumental)
5:10pm5:50pm Wednesday, March 27, 2019
Secondary topics:  AI and Data technologies in the cloud, Data Integration and Data Pipelines
Average rating: ***..
(3.50, 8 ratings)

Who is this presentation for?

  • Architects, engineers, and executives

Level

Beginner

Prerequisite knowledge

  • A basic familiarity of AWS

What you'll learn

  • Learn how to use serverless workflows to architect hybrid processing systems in your cloud pipeline
  • Understand existing solutions and how they work as well as hidden problems and tricks when using serverless workflows

Description

Serverless is becoming increasingly popular as a means to organize core processing. However, companies with existing pipelines can find it hard to move completely serverless. Serverless workflows unite the serverless and cluster worlds, with the benefits of both approaches.

Serverless workflows enable hybrid architecture with both cluster-based processing with longtime processing or high CPU load jobs and serverless functions for the rest of operations because they’re scalable, simple, and cheap. Additionally serverless workflows enable completely modular processing, where the developer defines the method of implementation, whether third-party services, open source libraries, or native cloud services.

Rustem Feyzkhanov compares AWS Step Functions and Azure Logic Apps, discussing their pricing, features, and pros and cons. He also demonstrates a number of applications and their architectures, including an ML/DL pipeline, load testing, image processing, and report generation. You’ll then learn how serverless workflows allow you to conduct production tasks such as A/B testing modules, canary deployments, and error handling.

Photo of Rustem Feyzkhanov

Rustem Feyzkhanov

Instrumental

Rustem Feyzkhanov is a machine learning engineer at Instrumental, where he creates analytical models for the manufacturing industry. Rustem is passionate about serverless infrastructure (and AI deployments on it) and has ported several packages to AWS Lambda from TensorFlow, Keras, and scikit-learn for ML to PhantomJS, Selenium, and WRK for web scraping.

Comments on this page are now closed.

Comments

Picture of Rustem Feyzkhanov
Rustem Feyzkhanov | MACHINE LEARNING ENGINEER
03/28/2019 3:15am PDT

Link to presentation: http://bit.ly/2Wx2LKV
Link to code repository: https://github.com/ryfeus/stepfunctions2processing