Engineering the Future of Software
29–31 Oct 2018: Tutorials & Conference
31 Oct–1 Nov 2018: Training
London, UK

Event streaming as a source of truth

Benjamin Stopford (Confluent)
15:5016:40 Tuesday, 30 October 2018
Secondary topics:  Best Practice
Average rating: ****.
(4.64, 11 ratings)

Who is this presentation for?

  • Architects

Prerequisite knowledge

  • Familiarity with messaging, microservices, databases, and basic system architecture

What you'll learn

  • Understand how the event streaming as a source of truth pattern provides a powerful alternative to using RPCs, enterprise messaging, or a shared database to communicate and share data across different applications or microservices
  • Learn how event sourcing, event-driven architectures and stream processing relate to this pattern, how the event-driven mindset alters the way we think about system design, and how stream processing tools help us manage event streams directly, as they move around the organization

Description

One of the most interesting and provocative patterns to face the software architecture community is the use of event streaming as a source of truth—a pattern where replayable logs, like Apache Kafka, are used for both communication as well as event storage, incorporating the retentive properties of a database in a system designed to share data across many teams, clouds, and geographies.

Ben Stopford believes this concept to be transformative. Such a bold claim should of course be met with a healthy degree of skepticism, but the interesting thing about communication patterns is that their value comes from often subtle, systemic effects, particularly where humans are involved. You will be familiar with these already: email, Twitter, Slack, and Facebook are all conceptually similar forms of communication but display very different dynamics in practice, yet zeroing in on exactly why these tools operate and feel so different is rarely as simple as it may seem.

Join Ben to explore the event streaming pattern and discover the systemic effects it has on the architectures we build around it, helping you see where the value really lies. Ben examines the relationship between events, event sourcing, and stream processing, as well as the concept of a database unbundled or turned inside out. He also explores how the pattern encourages subtler systemic effects: easier evolution, a more ephemeral view on data, and systems that seamlessly span departments, cloud providers, and geographies.

Photo of Benjamin Stopford

Benjamin Stopford

Confluent

Ben Stopford is a technologist working in the Office of the CTO at Confluent (the company behind Apache Kafka), where he has worked on a wide range of projects, from implementing the latest version of Kafka’s replication protocol to developing strategies for streaming applications. Previously, Ben led the design and build of a company-wide data platform for a large financial institution and worked on a number of early service-oriented systems, both in finance and at ThoughtWorks. He is the author of the book Designing Event Driven Systems.