The Story of Sharding at Box

Operations and Culture
Location: Buckingham Room Level: Intermediate
Presentation: external link
Average rating: *****
(5.00, 1 rating)

The database is typically the most difficult web application component to scale horizontally. This is especially true if the application was originally built with a single relational database in mind. In this talk we will present a step-by-step description of how Box’s web application stack was transitioned from one monolithic MySQL database to a fully sharded MySQL architecture. The emphasis will be on the incremental steps and best practices that enabled the successful execution of this fundamental change, all the while continuously serving 2 billion queries per day. The techniques presented will be relevant to web application architects, DevOps and DBAs trying to make the initial transition to a sharded solution.

We will begin with an overview of our web application architecture both before and after sharding. We will then walk through the modifications we made to our ORM layer to support this new architecture, including support for cross-shard queries and online moving of data between shards. We will continue with a detailed description of the technique we developed for migrating live data to shards without downtime, which also supports table by table migration for added flexibility. We will then describe how we use a similar migration technique to move and split shards for rebalancing purposes. Finally, we will discuss some of the challenges of maintaining a sharded databases architecture, and present some tools and techniques we have found to be useful. Throughout the talk we will focus on how to make large-scale changes in an incremental fashion, without adversely affecting functionality, and most importantly without downtime.

Photo of Tamar Bercovici

Tamar Bercovici

Box

Tamar Bercovici is a Senior Software Engineer at Box where she specializes in scaling Box’s database architecture and ORM layer. Prior to Box, Tamar was an early-stage employee at XMPie (now a Xerox company), where she drove the development of the award winning uImage product. Tamar holds a Ph.D. in Computer Science from the Technion – Israel Institute of Technology.

Sponsors

Sponsorship Opportunities

For information on exhibition and sponsorship opportunities at the conference, contact Gloria Lombardo at glombardo@oreilly.com

Media Partner Opportunities

For media partnerships, contact mediapartners@ oreilly.com

Press and Media

For media-related inquiries, contact Maureen Jennings at maureen@oreilly.com

Contact Us

View a complete list of Velocity contacts