Moving Day: Migrating Your Big Data from A to B

Data: Scaling
Location: C121/122
Average rating: **...
(2.75, 4 ratings)

In January we moved the Mozilla crash reporting system from old creaky hardware in San Jose to a new shiny datacenter in Phoenix. This system contains more than 30TB of data in HBase, the Hadoop database, and PostgreSQL. The data collecting app has a requirement for close to 100% uptime. On top of that we have data processing, an API, and a webapp. After many months of work, the migration went seamlessly.

In this session we’ll talk about:

  • The checklist manifesto, reprised, and understanding the critical path
  • How to move all that data in a reasonable timeframe
  • The importance of devops culture in success
  • Automating packaging and configuration and how it will save you
  • Understanding the difference between old and new platforms: correctness testing, load testing, and smoke testing

Attendees should walk away with an outline of everything they’ll need to do to achieve a successful migration.

Photo of Laura Thomson

Laura Thomson

Mozilla Corporation

Laura Thomson is Senior Director of Engineering, Firefox Engineering Operations at Mozilla, and is on the board of ISRG, the non-profit behind Let’s Encrypt. Laura has worked in various engineering roles at Mozilla for the last ten years, and previously worked in jobs including Principal/VP at OmniTI, engineer, consultant, and computer science academic. She has written several books on various open source technologies, including “PHP and MySQL Web Development” (with Luke Welling, her husband). Laura is from Melbourne, Australia, but now lives with her family on a rustic horse farm in Maryland, USA, where she relaxes by indulging in manual labor and other rural pursuits.

Josh Berkus

PostgreSQL Experts

Josh Berkus has been a member of the PostgreSQL Core Team since 2003 and has been working as a database consultant since 1998. His consulting specialties include:

  • Database design
  • Performance tuning and troubleshooting</li
  • Stored procedure debugging</li
  • Training</li
  • Reporting and BI
  • ETL and data cleaning

In addition to his PostgreSQL skills, Josh also:

  • Manages projects
  • Advises companies on working with open source communities
  • Speaks at 6 to 12 conferences a year, worldwide
  • Is an expert in SQL Server 2000
  • Co-founded the San Francisco PostgreSQL User Group

Josh’s work experience includes 8 years of consulting on database applications, primarily building applications for the legal and HR industries and performance tuning. He was also head of Sun Microsystem’s PosgtreSQL support staff for 2 years and helped launch BI startup Greenplum.

Corey Shields

Mozilla Corporation

Corey is a manager for Mozilla’s IT Systems team. He has long been an open source advocate, since first being introduced to Slackware on an Infomagick CD in the mid 90’s.

Previous to Mozilla, Corey has worked as a Unix supercomputing administrator for Indiana University and as a systems manager for the Open Source Lab at Oregon State University.

Photo of Justin Dow

Justin Dow

Mozilla Corporation

Justin, most people know him as “jabba” a long time nickname and is IRC nick, works as a Systems Administrator for Mozilla, managing the Infrastructure team. In his spare time he likes tinkering on classic cars and spending time with his family.