Engineer for the future of Cloud
June 10-13, 2019
San Jose, CA

Schema evolution patterns

Alex Rasmussen (Bits on Disk)
4:45pm5:25pm Wednesday, June 12, 2019
Distributed Data
Location: 230 A
Average rating: ****.
(4.00, 1 rating)

Level

Intermediate

Prerequisite knowledge

  • Experience building services that communicate with one another or store data in a database

What you'll learn

  • Explore solutions to the problem of schema evolution

Description

Everybody’s talking about microservices, but nobody seems to agree on how to make them talk to each other. How should you version your APIs, and how does API version deprecation actually work in practice? Do you use plain old JavaScript object notation (JSON), Thrift, protocol buffers, GraphQL? How do teams communicate changes in their services’ interfaces, and how do consumer services respond? Separately, nobody seems to agree on how to handle migrating a service’s structured data without downtime. Do you write to shadow tables? Chain new tables off the old ones? Just run the migration live and hope nothing bad happens? Switch everything over to NoSQL?

Both these problems are instances of issues with schema evolution: what happens when the structure of your structured data changes. Rather than taking a prescriptive approach, Alex Rasmussen distills a lot of institutional knowledge and computer science history into a set of patterns and examines the trade-offs between them.

Photo of Alex Rasmussen

Alex Rasmussen

Bits on Disk

Alex Rasmussen is a data engineering consultant. Previously, he was VP of engineering at Freenome, an AI genomics company, and an early employee at Trifacta, a pioneer in the data wrangling space. He holds a PhD from the University of California San Diego (UCSD), where his dissertation focused on highly efficient large-scale data processing systems. While at UCSD, he led the TritonSort project, which set several world records in large-scale sorting.