Realtime Data Analysis Patterns

Mikio Braun (Zalando SE)
Data Science
Location: 115
Average rating: ****.
(4.19, 16 ratings)

Processing huge volume event streams in realtime in a robust and efficient fashion poses quite some challenges. Throwing raw processing power at the problem is one way to solve them, but there are more efficient ways, in particular when the specific analysis task focusses on interesting points or allows to deal with approximate results. In this talk we’ll cover what we call realtime data analysis patterns, covering all aspects from data acquisition, processing, to storage of historic data, always making sure that the resulting system can provide constant performance. The resulting architecture uses approximative algorithms at its core, and uses a combination of in-memory and disk based storage. We deal with such questions like: How to make sure we can ingest several 10k events per second? How to keep track of millions of objects with bounded resources? How to integrate with existing infrastructure? Finally, we will discuss several use cases, including social media data, user realtime profiling and recommendation, and realtime analytics.

Photo of Mikio Braun

Mikio Braun

Zalando SE

Mikio Braun is co-founder of Streamdrill, a startup focused on approximative approaches for real real-time big data. He holds a Ph.D. in Machine Learning and has worked in research for a number of years, before becoming interested in putting research results into good use in the industry. His current interests focus on anything to do with real-time data analysis, in particular using approximative approaches beyond scaling.