Presented By O'Reilly and Cloudera
Make Data Work
September 25–26, 2017: Training
September 26–28, 2017: Tutorials & Conference
New York, NY

Why containers and microservices need streaming data

Paul Curtis (Weaveworks)
4:35pm5:15pm Wednesday, September 27, 2017
Data engineering, Data Engineering & Architecture
Location: 1A 15/16/17 Level: Intermediate
Secondary topics:  Architecture, Streaming
Average rating: ****.
(4.67, 3 ratings)

Who is this presentation for?

  • Data engineers, developers, system managers, and enterprise architects

Prerequisite knowledge

  • A basic understanding of containers and streaming technologies

What you'll learn

  • New strategies for utilizing containers and streams globally in a big data environment


Containerization has enabled the convenient, predictable, and portable deployment of applications. Similarly, microservices are very useful for implementing complex systems. Finally, message streaming is rapidly becoming a standard way to build distributed systems.

The surprise here is the synergistic way these three different ideas interact. No matter which approach you start with, the others will enhance it. If you are trying to containerize the parts of your system, microservices turn out to be a great way to organize those parts. And message stream transport, with the right capabilities, is a great way to connect microservices. Each additional piece adds its own value while enhancing the others.

For example, the agility and flexibility of microservices is useful for projects that make use of machine learning models. Containers provide a predictable environment for running the applications such that behavior observed during the testing phase will be stable once the applications are deployed in production. And the additional choice of running different components in a decoupled microservices style connected by message streams that support multiple independent consumers not only lets you evaluate new models without interfering with production but also enables other projects to make use of the same data without interference.

Paul Curtis explores these infrastructure components and discusses the design of highly scalable real-world systems that take advantage of this powerful triad of containers, microservices, and message streaming. Paul also shares real-world customer case histories that highlight how these approaches interact. You’ll learn how to adopt the technologies and architecture needed to take advantage of this combination of approaches.

Photo of Paul Curtis

Paul Curtis


Principal Solutions Architect at Weaveworks, previously a Principal Engineer at MapR.