Reconciling Eventually-Consistent Data with CRDTs

Operations and Culture
Location: Palace Suite Level: Intermediate
Average rating: **...
(2.36, 14 ratings)
Slides:   1-PDF 

Eventually consistent data is common in modern web applications. We all know about eventually consistent data stores, popularised by Dynamo and Cassandra, but there are other sources of eventually consistent data. Mobile applications can go offline, but that doesn’t stop people from using them, requiring synchronising data when they come back online. For robustness and to reduce network latency, applications may be hosted in multiple data centres connected by relatively high latency connections. It’s infeasible to maintain strong consistency these connections, again requiring synchronisation at regular intervals.

How do we reconcile eventually consistent data? One way is to use data types that we are guaranteed to merge without conflict. Then we easily reconcile all the different replicas in our system with a simple merge operation that we know cannot fail. These data types are known as conflict-free (sometimes commutative or convergent) replicated data types (CRDTs).

It seems straightforward to create, say, CRDT counters, but can we create CRDTs for complex data types such as sets? It turns out we can. There are wide variety of CRDTs available. In this talk I will describe the foundations of CRDTs, give some examples of known data types, and discuss issues that arise in practice.

Photo of Noel Welsh

Noel Welsh

Underscore Consulting

Noel has over fifteen years experience in software architecture and development, and over a decade in machine learning and data mining. His current project is Myna, which makes bandit algorithms accessible to all. Previous projects he’s been involved with include one of the first commercial products to apply machine learning to the Internet (eventually acquired by Omniture), a BAFTA award winning website, and a custom CMS used daily by thousands of students.

Noel is an active writer, presenter, and open source contributor. Noel has a PhD in machine learning from the University of Birmingham.

Sponsors

Sponsorship Opportunities

For exhibition and sponsorship opportunities at Velocity conference, contact Gloria Lombardo at +1 (203) 381-9245 or glombardo@oreilly.com

Media Partner Opportunities

For information on trade opportunities with O'Reilly conferences contact Jaimey Walking Bear at mediapartners
@oreilly.com

Contact Us

View a complete list of Velocity Europe 2013 contacts