Change is the only constant; upgrades are inevitable. Believe it or not, this also applies to your database. There are many drivers for such changes, such as technology stack updates and new data center or event cloud migrations. A database migration isn’t an easy task. Preparation and execution takes time and leaves no room for mistakes. And once you start, there’s no going back. But is this the only way?
Adrian Lungu and Serban Teodorescu explain how—inspired by the green-blue deployment technique—the Adobe Audience Manager team developed an active-passive database migration procedure that allows them to test database clusters in production, minimizing the risks without compromising the innovation. It was successfully applied twice to upgrade the entire database technology stack—but it’s never a smooth move when your databases are 200-node Cassandra clusters with hundreds of terabytes of data and downtime is not an option. The first migration was focused only on software upgrades. For the second upgrade, the team’s confidence was so high that they added a twist: a couple of more changes besides the Cassandra version—AWS instance type, operating system, disk settings, memory settings, JVM, and a few more. What could go wrong? Well. . .everything.
Adrian and Serban describe the migration technique and present an extensible database client that makes all the active-passive management possible with just a configuration change. Yes, you heard it right. No code changes, just configurations. They then share a series of tales and lessons learned during the migration of over 500 Cassandra nodes. Most of the lessons aren’t Cassandra related but instead apply to hardware, drivers, operating system, or the JVM—debugging for days and searching for that metric anomaly or that log line that would give us a hint on what went wrong. Join in to see how you can avoid some of these pitfalls in your own projects.
Adrian Lungu is a computer scientist at Adobe working with Audience Manager, a leading solution in the DMP market. He’s been focused on the company’s Cassandra clusters ever since he joined the team, trying to build a scalable architecture that would keep up with the exponential growth of the product. Adrian holds a degree in computer science and engineering from Politehnica University of Bucharest as well as a DataStax Certified Apache Cassandra Professional certification.
Serban Teodorescu is an SRE at Adobe, where he’s part of a small team that manages 20+ Cassandra clusters for Adobe Audience Manager. Previously, he was a Python programmer, and he’s still trying to find out how a developer who preferred SQL databases ended up as an SRE for a Cassandra team. Apart from Cassandra and Python, he’s interested in automating infrastructure provisioning with Terraform.
©2019, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org