Mar 15–18, 2020

Truth and reality of a cloud migration for large-scale data processing workflows

Harrison Wang (LiveRamp)
5:05pm5:45pm Tuesday, March 17, 2020
Location: LL21B

Who is this presentation for?

Data engineers, data architects, developers

Level

Intermediate

Description

LiveRamp faced critical decisions in 2018. After experiencing infrastructure failures in its private data center and facing the need for international operations, it decided to migrate its operations to the cloud. The company had to move hundreds of applications running on thousands of physical nodes, including tens of petabytes in databases and distributed file systems. And, obviously, the system as a whole needed to remain in production during the move.

Harrison Wang walks you through how one team managed that transition. He explains the team’s approach to the project, its specific role, and how that fit into the companywide effort. You’ll learn about the technical and logistical challenges such as data and code migration, networking bandwidth, operational practices, team dependencies, data security, timelines, and training.

It’s important to start early and remember that it’s not just a singular team that’s moving; it’s the whole company. You should expect to be surprised. Not every application runs the same on a different environment. A lot of work goes into understanding and managing team dependencies. Anything overlooked will definitely show up during the migration. And a lot more goes into the migration than just migrating the application itself. Consider the network, the data that needs to be migrated, and the infrastructure. Consider the timelines and dependencies that each team needs for migrating their own applications. Consider the data bandwidth needed during the migration. Using and moving data from A to B hampers productivity. Don’t underestimate the learning curve for everybody to get acquainted to the new technologies. Take the training and use the outside resources. There’s a delicate balance between “getting things done” and ensuring everbody is comfortable with “new technology.”

With a combination of tools built internally and supplied by the cloud provider, LiveRamp successfully moved its production to the cloud in 13 months. At no time were production services offline more than four hours at a time.

Prerequisite knowledge

  • A basic understanding of big data and distributed systems (useful but not required)

What you'll learn

  • Discover how LiveRamp moved its production to the cloud

Harrison Wang

LiveRamp

Harrison Wang is a backend software engineer for LiveRamp and was responsible for coordinating the cloud migration for the activations team.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)

Contact us

confreg@oreilly.com

For conference registration information and customer service

partners@oreilly.com

For more information on community discounts and trade opportunities with O’Reilly conferences

Become a sponsor

For information on exhibiting or sponsoring a conference

pr@oreilly.com

For media/analyst press inquires