Build Systems that Drive Business
Sep 30–Oct 1, 2018: Training
Oct 1–3, 2018: Tutorials & Conference
New York, NY

Managing multiple sources of truth in distributed applications

Adam Wolfe Gordon (DigitalOcean)
1:30pm–2:10pm Tuesday, October 2, 2018
Distributed Systems
Location: Nassau Level: Intermediate
Secondary topics:  Distributed State

Prerequisite knowledge

  • A basic understanding of distributed application architecture
  • Experience with applications that manage external state (i.e., state outside of the application's database)


Most applications manage some sort of persisted state, and managing state in a distributed application can be a challenge. Storage solutions such as database systems or distributed key-value stores can do most of the heavy lifting, but only if your application can keep all its state in one place.

Some applications inherently have multiple sources of truth. For example, an application that manages cloud storage may keep metadata about storage volumes in a database, but the actual state of the storage volumes themselves is maintained by the underlying storage system. Keeping these multiple sources of truth consistent is a hard problem.

Adam Wolfe Gordon shares strategies, learned from real-world experience building a cloud storage orchestration system, for keeping multiple sources of truth consistent and maintaining application usability. These strategies include:

  • Choosing and using a single source of truth for user-visible state
  • Ordering state creation, updates, and deletion appropriately
  • Using deferred, asynchronous work to keep state consistent

These sound like simple ideas, but the details are tricky to get right. Throughout the talk, Adam explores examples of how these strategies have been applied in building an storage orchestration system for a public cloud and discusses challenges encountered along the way.

Photo of Adam Wolfe Gordon

Adam Wolfe Gordon


Adam Wolfe Gordon is a software engineer on the storage team at DigitalOcean, working primarily on block storage orchestration. He likes building elegant microservices, continuous deployment, and occasional forays into low-level software such as ceph and qemu.