Mar 15–18, 2020

Redis plus Spark Structured Streaming: The perfect way to scale out your continuous app

Dave Nielsen (Redis Labs)
4:15pm4:55pm Tuesday, March 17, 2020
Location: 230 A
Secondary topics:  Streaming and IoT

Who is this presentation for?

Data engineers, data architects, developers




Continuous applications have three things in common: they collect data from sources (e.g., IoT devices), process them in real time (e.g., ETL), and deliver them to the machine learning serving layer for decision making. Continuous applications face many challenges as they grow to production. Often, due to the rapid increase in the number of devices, end users, or other data sources, the size of the dataset grows exponentially. This results in a backlog of data to be processed and the data is no longer processed in near real time.

Redis Streams enables you to collect both binary and text data in a time series format. The consumer groups of Redis Stream help you match the data processing rate of your continuous application with the rate of data arrival from various sources. Apache Spark’s Structured Streaming API enables real-time decision making for continuous applications.

Dave Nielsen demonstrates how to integrate open source Redis with Apache Spark’s Structured Streaming API using that Spark-Redis library. He walks you through the code and runs a live continuous application.

Prerequisite knowledge

  • A basic understanding of real-time data, streaming, and analytics
  • A basic understanding of streaming solution such as Redis Streams and an analytics platform like Apache Spark (useful but not required)

What you'll learn

  • Understand real-time streaming analytics and display for interactive data
Photo of Dave Nielsen

Dave Nielsen

Redis Labs

Dave Nielsen is the head of community and ecosystem programs at Redis Labs and the cofounder of CloudCamp, a series of unconferences about cloud computing. Over his 19-year career, he’s been a web developer, systems architect, technical trainer, developer evangelist, and startup entrepreneur. Dave resides in Mountain View with his wife, Erika, to whom he proposed in his coauthored book PayPal Hacks.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)

Contact us

For conference registration information and customer service

For more information on community discounts and trade opportunities with O’Reilly conferences

Become a sponsor

For information on exhibiting or sponsoring a conference

For media/analyst press inquires