Engineering the Future of Software
Feb 25–26, 2018: Training
Feb 26–28, 2018: Tutorials & Conference
New York, NY

Event storage in a distributed system

Stephen Pember (Toast)
10:45am–12:15pm Wednesday, February 28, 2018
Secondary topics:  Best Practice, Overview
Average rating: ***..
(3.33, 3 ratings)

Who is this presentation for?

  • Architects, tech leaders, and senior developers

Prerequisite knowledge

  • Familiarity with distributed systems, reactive programming, DDD, and CQRS

What you'll learn

  • Understand the value of event storage in distributed systems
  • Learn how to handle, store, and read these events

Description

Developers building modern distributed systems are embracing techniques that rely on asynchronous event handling for service communication. Reactive systems demand it. While it’s possible to ignore these events once they’ve been acted upon, in the long run they have massive analytical value. A wise developer would persist every event that flows through their system by building an event store mechanism.

In this context an event store isn’t one framework, tool, or library. It’s a pattern or a mechanism designed to provide queryable access to your events in some fashion. Event storage offers many practical benefits to distributed systems providing complete state changes over time. This allows for interesting analytical insights (provided you can mine these events appropriately), easy bootstrapping of new services into the system, and the possibility of viewing the state of the system at any point in time, among other things. However, there are a number of challenges when building an event store mechanism: Where should these events physically persist? Do we create one centralized event store, or do we rely on each service to store their own events? And how do we go back in time and query for events?

Stephen Pember explores some of the problems you may encounter and shares real-world patterns for working with event storage, championing the use of asynchronous events as a primary means of communication and techniques for persisting and accessing those events. Stephen briefly explores reactive programming and CQRS in order to see how they result in efficient, decoupled services before demonstrating how event storage can be a powerful analytical tool that enables custom projections of your data. Stephen weighs the merits of a centralized event store against local event stores and explores practical code examples for event design and event handling flow, based on his real-world experience. Although you’ll hear the word “event” so many times you’ll probably be sick of hearing it, you’ll walk away ready to build an event storage mechanism in your own system.

Photo of Stephen Pember

Stephen Pember

Toast

Steve Pember is a principal engineer and team lead at Toast, a creator of systems and point of sale devices for managing restaurants. Previously, he was a director of engineering, a CTO, and a principal consultant, all the while pushing for and building reactive, event-driven, microservices-based platforms. Steve is obsessed with highly scalable distributed systems, software architecture, and alternative data storage techniques like event sourcing, and he loves telling the world about them.