Engineering the Future of Software
Feb 3–4, 2019: Training
Feb 4–6, 2019: Tutorials & Conference
New York, NY

Stream processing for the serverless generation

Ben Stopford (Confluent)
2:15pm–3:05pm Wednesday, February 6, 2019
Cloud native
Location: Trianon Ballroom
Secondary topics:  Best Practice
Average rating: ****.
(4.00, 1 rating)

Who is this presentation for?

  • Architects



Prerequisite knowledge

  • A basic understanding of databases and messaging systems

What you'll learn

  • Understand the relationship between stream processing and serverless functions, including the relative merits of each today
  • Explore patterns used to build event-driven applications using stream processors and distributed logs
  • See how these fields will merge in the future


People often think of stream processing as a toolset for running “big data” computations on high-velocity data streams: problems that are too fast to be computed in batches. But stream processing is really a far richer field that applies correctness to any arbitrary graph of intertwined functions—a problem that complex serverless applications are forced to contend with in one way or another. But serverless technologies optimize for a different goal: abstracting your programs away from the underlying infrastructure, executing when triggered, and scaling automatically. So what happens if we mix the best of these two worlds?

Ben Stopford explores this question by examining how stream processing systems have evolved away from being simple functions sewn together by a messaging system. Ben looks at the core building blocks (e.g., exactly once semantics and buffering and joining of event streams) as well as higher-level patterns used by contemporary event-driven applications (e.g., event sourcing, CQRS, and event streams as a source of truth). This inevitably leads to a model where stream processors play the role of a kind of database that derives rich, use case-specific event streams, keeping your functions simple, stateless, and scalable. Ben concludes by reflecting on what the future likely holds for these two fields as the approaches converge in serverless frameworks that have far richer capabilities than the state of the art today.

Photo of Ben Stopford

Ben Stopford


Ben Stopford is a technologist working in the Office of the CTO at Confluent, a company backing the popular Apache Kafka messaging system. He has two decades of experience in the field and has focused on distributed data infrastructure for the last half of it. He’s the author of Designing Event-Driven Systems from O’Reilly.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)