Building and maintaining complex distributed systems
June 19–20, 2017: Training
June 20–22, 2017: Tutorials & Conference
San Jose, CA

How Shutterfly migrated 10+ billion photos to the cloud

Jack Chan (Shutterfly)
4:35pm–5:15pm Thursday, June 22, 2017
Level: Intermediate
Average rating: *****
(5.00, 1 rating)

Who is this presentation for?

  • Engineering leaders, architects, and technology consultants

Prerequisite knowledge

  • Basic knowledge of the cloud, high-availability systems, and web architecture

What you'll learn

  • Understand the complexities and challenges in implementing a migration strategy at a massive scale
  • Explore Shutterfly's migration strategy design, prerequisites, and emphasis on team culture and learn the importance of taking measurements every step of the way

Description

As part of Shutterfly’s multiyear Shutterfly 3.0 initiative, the company is modernizing its photo management solution. In 2016, the company took the bold step of migrating metadata from over 10 billion sets of photos from a private data center to AWS’s public cloud. This major feat was performed over the public internet across the North America continent, with a majority of those moved in less than 100 days—and within budget. Shutterfly Photos is now part of a hybrid cloud solution with image binaries hosted on-premises and metadata for photo management on AWS.

Jack Chan focuses on how Shutterfly evolved a multiclient photo management system integrated through a prior acquisition and scale it to handle 20x the amount of data, outlining how the company transferred the metadata from an on-premises facility over the wire to AWS Cloud, persists it in a database cluster, postprocesses the photos with machine-learning algorithms, precaches the metadata for performance, hosts a scalable API, integrated its mobile, desktop, and web clients, added tons of monitors, and set up KPI metrics, all in time to prepare for Shutterfly’s Q4 holiday season sales.

Jack relates the trials and tribulations of facing a huge unknown of taking on such a major initiative, the rally cry from technology leaders, and the mountains of scalability preparations that went on postmigration while running a parallel user experience within the tens of millions of paying customers.

Photo of Jack Chan

Jack Chan

Shutterfly

Jack Chan is a senior engineering manager in Shutterfly’s Photos group. He was recently heavily involved with helping the company with a hybrid cloud migration solution with photos-related API services on AWS paired with a set of core services in a private data center. Jack has been working in software engineering development for quite some time, helping startups scale up to millions of users with cloud solutions. Previously, he worked in IT organizations at Adobe, Apple, and 3Com.