Engineering the Future of Software
November 13–14, 2016: Training
November 14–16, 2016: Tutorials & Conference
San Francisco, CA

How to apply big data analytics and machine learning to real-time processing

Kai Wähner (Confuent)
10:45am–12:15pm Tuesday, 11/15/2016
Integration architecture
Location: Tower Salon A Level: Beginner
Average rating: ***..
(3.17, 6 ratings)

Prerequisite knowledge

  • A basic familiarity with big data, event and stream processing, and analytics (useful but not required)

What you'll learn

  • Understand how event processing uses analytic models (without redeveloping) to take action in real time and how microservices can be leveraged to solve the Agile requirements of your projects
  • Learn about different frameworks for machine learning and event processing
  • Gain exposure to real-world case studies from different industries

Description

With the growth of mobile, the cloud, and the Internet of Things, the world is becoming more connected every year. Much-hyped big data is currently the popular method to deal with the data that is created. Large amounts of historical data are stored in Hadoop or other platforms, and big data frameworks leverage machine-learning frameworks, such as R, Apache Spark, or H2O, to draw new knowledge and to find patterns from this data (for example, for promotions, cross-selling, or fraud detection). The key challenge is how these findings can be integrated from historical data into new transactions in real time to make customers happy, increase revenue, or prevent fraud.

“Fast data” via stream processing is the solution to embed patterns—obtained from analyzing historical data—into future transactions in real time. Kai Wähner explores “fast data” frameworks and explains how they are strongly related to microservices. Kai uses several real-world success stories to explain the concepts behind stream processing and its relation to Hadoop and other big data platforms and discusses how the patterns and statistical models of R, Spark MLlib, H2O, and other technologies can be integrated into real-time processing. Along the way, Kai points out why a microservices architecture helps solve the Agile requirements for these kind of projects, offers a brief overview of available open source frameworks and commercial products, such as Apache Storm, Apache Flink, Spark Streaming, IBM InfoSphere Streams, or TIBCO StreamBase, and shares a live demo showing how to implement stream processing, how to integrate machine learning, and how human operations can be enabled in addition to the automatic processing via a web UI and push events.

Photo of Kai Wähner

Kai Wähner

Confuent

Kai Waehner works as Technology Evangelist at Confluent. Kai’s main area of expertise lies within the fields of Big Data Analytics, Machine Learning / Deep Learning, Messaging, Integration, Microservices, Internet of Things, Stream Processing and Blockchain. He is regular speaker at international conferences such as JavaOne, O’Reilly Software Architecture or ApacheCon, writes articles for professional journals, and shares his experiences with new technologies on his blog (www.kai-waehner.de/blog). Contact and references: kontakt@kai-waehner.de / @KaiWaehner / www.kai-waehner.de