Engineering the Future of Software
18-19 October 2016: Training
19-21 October 2016: Tutorials & Conference
London, UK

How to apply big data analytics and machine learning to real-time processing of microservice events

Kai Wähner (Confluent)
10:45–12:15 Thursday, 20/10/2016
Integration architecture
Location: Blenheim Room Level: Beginner
Average rating: *....
(1.67, 3 ratings)

Prerequisite knowledge

  • A basic familiarity with big data, event and stream processing, and analytics (useful but not required)

What you'll learn

  • Understand how event processing uses analytic models (without redeveloping) to take action in real time and how microservices can be leveraged to solve the Agile requirements of your projects
  • Learn about different frameworks for machine learning and event processing
  • Gain exposure to real-world case studies from different industries


With the growth of mobile, the cloud, and the Internet of Things, the world is becoming more connected every year. Much-hyped big data is currently the popular method to deal with the data that is created. Large amounts of historical data are stored in Hadoop or other platforms, and big data frameworks leverage machine-learning frameworks, such as R, Apache Spark, or H2O, to draw new knowledge and to find patterns from this data (for example, for promotions, cross-selling, or fraud detection). The key challenge is how these findings can be integrated from historical data into new transactions in real time to make customers happy, increase revenue, or prevent fraud.

“Fast data” via stream processing is the solution to embed patterns—obtained from analyzing historical data—into future transactions in real time. Kai Wähner explores “fast data” frameworks and explains how they are strongly related to microservices. Kai uses several real-world success stories to explain the concepts behind stream processing and its relation to Hadoop and other big data platforms and discusses how the patterns and statistical models of R, Spark MLlib, H2O, and other technologies can be integrated into real-time processing. Along the way, Kai points out why a microservices architecture helps solve the Agile requirements for these kind of projects, offers a brief overview of available open source frameworks and commercial products, such as Apache Storm, Apache Flink, Spark Streaming, IBM InfoSphere Streams, or TIBCO StreamBase, and shares a live demo showing how to implement stream processing, how to integrate machine learning, and how human operations can be enabled in addition to the automatic processing via a web UI and push events.

Photo of Kai Wähner

Kai Wähner


Kai Wähner is a technology evangelist at Confluent. Kai’s areas of expertise include big data analytics, machine learning, deep learning, messaging, integration, microservices, the internet of things, stream processing, and blockchain. He’s regular speaker at international conferences such as JavaOne, O’Reilly Software Architecture, and ApacheCon and has written a number of articles for professional journals. Kai also shares his experiences with new technologies on his blog.