Engineering the Future of Software
Feb 3–4, 2019: Training
Feb 4–6, 2019: Tutorials & Conference
New York, NY

Stream processing for the serverless generation

Ben Stopford (Confluent)
2:15pm–3:05pm Wednesday, February 6, 2019
Cloud native
Location: Trianon Ballroom
Secondary topics:  Best Practice

Who is this presentation for?

  • Architects

Level

Intermediate

Prerequisite knowledge

  • A basic understanding of databases and messaging systems

What you'll learn

  • Understand the relationship between stream processing and serverless functions, including the relative merits of each today
  • Explore patterns used to build event-driven applications using stream processors and distributed logs
  • See how these fields will merge in the future

Description

People often think of stream processing as a toolset for running “big data” computations on high-velocity data streams: problems that are too fast to be computed in batches. But stream processing is really a far richer field that applies correctness to any arbitrary graph of intertwined functions—a problem that complex serverless applications are forced to contend with in one way or another. But serverless technologies optimize for a different goal: abstracting your programs away from the underlying infrastructure, executing when triggered, and scaling automatically. So what happens if we mix the best of these two worlds?

Ben Stopford explores this question by examining how stream processing systems have evolved away from being simple functions sewn together by a messaging system. Ben looks at the core building blocks (e.g., exactly once semantics and buffering and joining of event streams) as well as higher-level patterns used by contemporary event-driven applications (e.g., event sourcing, CQRS, and event streams as a source of truth). This inevitably leads to a model where stream processors play the role of a kind of database that derives rich, use case-specific event streams, keeping your functions simple, stateless, and scalable. Ben concludes by reflecting on what the future likely holds for these two fields as the approaches converge in serverless frameworks that have far richer capabilities than the state of the art today.

Photo of Ben Stopford

Ben Stopford

Confluent

Ben Stopford is a technologist working in the Office of the CTO at Confluent, the company behind Apache Kafka. He’s worked on a wide range of projects, from implementing the latest version of Kafka’s replication protocol to writing a book on event-driven systems. Previously, Ben led the design and build of a company-wide data service in a large financial services firm and worked on projects at Thoughtworks and UK-based enterprise companies. He is the author of the book Designing Event-Driven Systems from O’Reilly. You can learn more on his website.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)