Database migrations don't have to be painful, but the road will be bumpy

Description

Change is the only constant; upgrades are inevitable. Believe it or not, this also applies to your database. There are many drivers for such changes, such as technology stack updates and new data center or event cloud migrations. A database migration isn’t an easy task. Preparation and execution takes time and leaves no room for mistakes. And once you start, there’s no going back. But is this the only way?

Adrian Lungu and Serban Teodorescu explain how—inspired by the green-blue deployment technique—the Adobe Audience Manager team developed an active-passive database migration procedure that allows them to test database clusters in production, minimizing the risks without compromising the innovation. It was successfully applied twice to upgrade the entire database technology stack—but it’s never a smooth move when your databases are 200-node Cassandra clusters with hundreds of terabytes of data and downtime is not an option. The first migration was focused only on software upgrades. For the second upgrade, the team’s confidence was so high that they added a twist: a couple of more changes besides the Cassandra version—AWS instance type, operating system, disk settings, memory settings, JVM, and a few more. What could go wrong? Well. . .everything.

Adrian and Serban describe the migration technique and present an extensible database client that makes all the active-passive management possible with just a configuration change. Yes, you heard it right. No code changes, just configurations. They then share a series of tales and lessons learned during the migration of over 500 Cassandra nodes. Most of the lessons aren’t Cassandra related but instead apply to hardware, drivers, operating system, or the JVM—debugging for days and searching for that metric anomaly or that log line that would give us a hint on what went wrong. Join in to see how you can avoid some of these pitfalls in your own projects.