Massively Sharded MySQL at Tumblr

Operations and Culture
Location: Hall 1 B/C Level:
Average rating: ***..
(3.62, 26 ratings)

When correctly architected, horizontal partitioning of data can lead to dramatically better fault tolerance and isolation, while significantly improving the manageability and flexibility of a web site. A well-sharded MySQL architecture can perform comparably to newer NoSQL solutions, while still offering the expressiveness of SQL and the time-tested robustness of InnoDB.

The road to complete sharding of Tumblr’s primary data has been a long one, and I’ll discuss how, when, and why we transitioned from a single database, to several functionally-partitioned datasets, and finally to a massively horizontally-sharded architecture.

Along the way, we’ve developed a set of tools that has made the sharding process significantly easier to execute and monitor. The bulk of this session will focus on the practical aspects of sharding, digging into the nitty-gritty of the tools we use to:

  • Transition from a single MySQL database to many without downtime
  • Automate the split of a large shard into several smaller ones extremely efficiently, without having to lock any rows or fail a single query, through clever use of MySQL replication
  • Determine the location of data in the application
  • Update the application’s database configuration across hundreds of web servers, without hitting problems from inconsistent state while the deploy is in progress

I’ll also cover some of the mistakes we’ve made, and the tools and tricks we used to gracefully back out of them, such as choosing the wrong shard sizes, and fixing uneven distribution of queries and data.

The session will broadly appeal to engineers at rapidly-growing, MySQL-backed sites. The information presented is all drawn from first-hand experience, and mostly cannot be found in existing books or web sites. Some level of audience familiarity with MySQL is assumed, but previous experience with sharding, InnoDB internals, or MySQL replication is not required.

Photo of Evan Elias

Evan Elias

Tumblr

Evan is a senior engineer at Tumblr, where he develops database solutions that scale to tens of billions of rows spanning over 100 MySQL installations. He has previously worked at Six Apart and Harvard University, and his software is used by major publications such as The New Yorker and Consumer Reports. He is intimately familiar with the internals of four major blogging platforms, and has been developing online software since the days of dial-up bulletin board systems.

Bulletin

Bulletin

  • ip-label
  • Compuware Corporation
  • dynaTrace
  • Keynote Systems
  • New Relic
  • Citrix Systems
  • Google
  • Apica
  • AppDynamics
  • CDNetworks
  • Cotendo
  • Dyn Inc.
  • ImmobilienScout24
  • Spil Games
  • Dyn Inc.

For information on exhibition and sponsorship opportunities at the conference, contact Gloria Lombardo at glombardo@oreilly.com

View a list of Velocity Europe contacts