Presented By O'Reilly and Cloudera
Make Data Work
Sept 29–Oct 1, 2015 • New York, NY

Elastic stream processing without tears

Michael Hausenblas (Red Hat)
11:20am–12:00pm Thursday, 10/01/2015
IoT & Real-time
Location: 3D 02/11 Level: Non-technical
Average rating: *....
(1.00, 1 rating)

Walmart handles more than 1 million customer transactions every hour. The average Boeing 737 engine generates 10 terabytes of data every 30 minutes in flight. By 2020, researchers estimate there will be 100 million internet connected devices. To process this data in real time—whether from mobile phones or jet engines—will be the new normal. How are companies today adapting to this new real-time stream of data, using open source projects that allow them to do this kind of stream processing at scale, including Apache Kafka, Apache Storm, Apache Samza, Apache Spark, and so on?

At the same time, how are organizations adapting the compute power depending on business needs or to accommodate the relentless growth in inbound traffic? True elastic stream processing can be achieved by combining a highly-scalable platform like Apache Mesos, with stream-processing frameworks built on top of it such as Marathon, Spark, Kafka, and new emerging solutions. In this talk we will discuss the use cases and requirements, and demonstrate a Mesos-based solution for elastically processing data streams.

Photo of Michael Hausenblas

Michael Hausenblas

Red Hat

Michael Hausenblas is a developer advocate for Go, OpenShift, and Kubernetes at Red Hat, where he helps app ops engineers build and operate distributed services. Michael shares his experience with distributed systems and large-scale data processing through demos, blog posts, and public speaking engagements and contributes to open source software such as OpenShift and Kubernetes. Previously, Michael was a developer advocate at Mesosphere, chief data engineer at MapR Technologies, and a research fellow at the National University of Ireland, Galway, where he researched large-scale data integration and the internet of things and gained experience in advocacy and standardization (World Wide Web Consortium, IETF). In his free time, Michael contributes to open source software (mainly using Go), blogs, and hangs out on Twitter too much.