Presented By O’Reilly and Cloudera
Make Data Work
September 11, 2018: Training & Tutorials
September 12–13, 2018: Keynotes & Sessions
New York, NY

Streaming big data in the cloud: What to consider and why

Bill Chambers (Databricks)
2:55pm–3:35pm Wednesday, 09/12/2018
Streaming systems & real-time applications
Location: 1E 07/08 Level: Intermediate
Average rating: ***..
(3.00, 1 rating)

Who is this presentation for?

  • Software engineers, data engineers, and streaming data engineers

Prerequisite knowledge

  • A basic understanding of Spark, streaming and stream processing concepts, and big data

What you'll learn

  • Learn how to choose the best streaming engine and architecture for your use case
  • Explore considerations for leveraging Apache Spark's Structured Streaming processing engine


Running streaming workloads successfully is a challenge, whether you’re deploying on-premises or in the cloud. While buying a managed service is an option, it’s usually quite expensive. Therefore, many companies opt for open source streaming engines like Apache Spark’s Structured Streaming.

Apache Spark’s Structured Streaming consolidates all big data processing under a unified API. Built on the foundation of the Spark SQL engine, not only does Structured Streaming allow developers to express the same queries for batch as for streaming, but it also allows for different execution strategies for streaming processing, including microbatching for high throughput or continuous processing for low latency.

Bill Chambers shares a decision making framework for determining the best tools and technologies for successfully deploying and maintaining streaming data pipelines to solve business problems. Bill then offers an overview of Apache Spark’s Structured Streaming processing engine and shares lessons learned running hundreds of Structured Streaming workloads in the cloud. Along the way, Bill dives into the internals of the Structured Streaming engine and explains why it’s suitable for a variety of uses cases.

Topics include:

  • How to successfully create business value with streaming
  • What makes a successful streaming use case and what doesn’t
  • A decision framework for choosing a streaming engine and architecture
  • The best advantages of streaming in the cloud (both storage and compute)
  • How to leverage cloud storage like S3 and Azure Blob Store for streaming workloads
  • How to successfully monitor and maintain your streaming applications
  • Future development of Structured Streaming
Photo of Bill Chambers

Bill Chambers


William Chambers is a product manager at Databricks, where he works on Structured Streaming and data science products. He is lead author of Spark: The Definitive Guide, coauthored with Matei Zaharia. Bill also created as a way to teach Apache Spark basics. Bill holds a master’s degree in information management and systems from UC Berkeley’s School of Information. During his time at school, Bill was also creator of the Data Analysis in Python with pandas course for Udemy and cocreator of and first instructor for Python for Data Science, part of UC Berkeley’s Masters of Data Science program.