Managing multiple sources of truth in distributed applications

Description

Most applications manage some sort of persisted state, and managing state in a distributed application can be a challenge. Storage solutions such as database systems or distributed key-value stores can do most of the heavy lifting, but only if your application can keep all its state in one place.

Some applications inherently have multiple sources of truth. For example, an application that manages cloud storage may keep metadata about storage volumes in a database, but the actual state of the storage volumes themselves is maintained by the underlying storage system. Keeping these multiple sources of truth consistent is a hard problem.

Adam Wolfe Gordon shares strategies, learned from real-world experience building a cloud storage orchestration system, for keeping multiple sources of truth consistent and maintaining application usability. These strategies include:

• Choosing and using a single source of truth for user-visible state
• Ordering state creation, updates, and deletion appropriately
• Using deferred, asynchronous work to keep state consistent

These sound like simple ideas, but the details are tricky to get right. Throughout the talk, Adam explores examples of how these strategies have been applied in building an storage orchestration system for a public cloud and discusses challenges encountered along the way.