4–7 Nov 2019

The golden idol swap: Pragmatic database migration

13:2514:05 Wednesday, 6 November 2019
Location: Hall A4
Average rating: **...
(2.00, 1 rating)

Who is this presentation for?

  • DevOps engineers, backend engineers, and system architects

Level

Intermediate

Description

Whether it’s to scale the throughput of your system, the amount of data you can store, or addressing technical or business issues, companies sometimes have to switch to a new database. There’s a significant amount of work that goes into planning and implementing this switch, including designing schemas, coming up with a failover plan, and figuring out how to distribute data across data centers. On top of all that, migrating legacy services and data to the new database is a large challenge. When NS1 set out to switch from a document store database to a relational database, it decided to abstract the databases through a proxy to minimize the amount of code changes required for its legacy services. The proxy provided wire protocol layer compatibility to minimize changes at legacy call sites.

If NS1 didn’t go down this route, it would’ve had to swap out the database driver for several services, which were implemented in multiple languages. Additionally, the company decided to not use a protocol like HTTP or gRPC to accomplish this, as this would require modifying various function calls throughout its legacy codebase still.

With those options not well suited to the company, it decided to design a proxy that would use the document store’s wire protocol in the frontend. Christian Grabowski describes how NS1 implemented this proxy, what hurdles it faced, and what tough design decisions it had to make along the way, as well as how it reduced the amount of development effort required to successfully make the switch. As for requirements for a backend of the proxy, the company needed to be able to still write to the legacy database until its data was entirely moved over to the new database, so the proxy was either required to write to both backends or just one based on configuration. NS1 also needed the proxy to route to this correct databases based on collection and table as it planned to run this proxy both in its core and edge facilities.

But it wasn’t always smooth sailing, considering the significant differences between the two databases. NS1 had to make some sacrifices with regard to the features between the databases.

Prerequisite knowledge

  • General knowledge of databases
  • Familiarity with application-level protocols

What you'll learn

  • Gain an understanding of a use case for abstracting databases through a wire protocol with the purpose of migrating from one to another
Photo of Christian Grabowski

Christian Grabowski

NS1

Christian Grabowski is a backend software engineer at NS1, a next-generation DNS and traffic management company. Christian has worn many engineering hats over the course of his career and has worked on a variety of software but loves getting into the nitty-gritty, low-level code the most. When he’s not developing fast, intelligent DNS services, he’s rather active in the open source community, contributing to projects such as gobpf, BCC, and Kubernetes.

Contact us

confreg@oreilly.com

For conference registration information and customer service

partners@oreilly.com

For more information on community discounts and trade opportunities with O’Reilly conferences

velocity@oreilly.com

For information on exhibiting or sponsoring a conference

pr@oreilly.com

For media/analyst press inquires