Engineering the Future of Software
16–18 October 2017: Conference & Tutorials
18–19 October 2017: Training
London, UK

Apache Spark and machine learning on microservices

Stepan Pushkarev (hydrosphere.io)
13:1514:05 Monday, 16 October 2017
Microservices, pros and cons
Location: King's Suite - Sandringham Level: Intermediate
Secondary topics:  Best Practice, Hands-on
Average rating: **...
(2.48, 21 ratings)

Prerequisite Knowledge

  • Basic experience building big data analytics services and plugging them into enterprise architecture

What you'll learn

  • Learn how to redesign your big data analytics platform as microservices

Description

Usually data scientists find it challenging to create a clean REST API; likewise, web developers find it almost impossible to understand machine learning internals. And big data engineers tend to use clunky Hadoop distributions with dozens of tightly coupled tools and then continue to follow this design, developing data processing scripts that communicate through unmanageable state and shared flags. Hydrosphere.io helps data scientists and big data engineers plug into modern reactive and microservices architectures that have already been adopted by traditional web and enterprise teams.

Hadoop-based data platforms that power ETL jobs and machine learning pipelines are great examples of monolithic architectures that could be redesigned with microservices. Stepan Pushkarev walks you through building and deploying data processing, reporting services, training, and prediction pipelines as decoupled microservices connected with the rest of the enterprise architecture.

Topics include:

  • ETL in microservices
  • Machine learning pipelines as microservices
  • Low-latency prediction services
  • Reactive prediction services
  • Asynchronous batch analytics services
Photo of Stepan Pushkarev

Stepan Pushkarev

hydrosphere.io

Stepan Pushkarev is CTO at hydrosphere.io, where he is responsible for product vision, sales strategy, and architecture and is still heavily involved in development. A technology entrepreneur, Stepan has cofounded two product startups so far. Previously, he spent 13 years in the software industry as a consulting engineer, architect, and CTO.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)

Comments

Picture of Stepan Pushkarev
Stepan Pushkarev | CTO
13/10/2017 9:25 BST

Preview blog post for ML FaaS concept to be presented during the talk!

Picture of Stepan Pushkarev
Stepan Pushkarev | CTO
25/09/2017 18:10 BST

Please check out background information for the 1st part of the talk Statesafe Data Pipelines
Looking forward to receive a feedback and questions before or after the talk.