Build & maintain complex distributed systems
October 1–2, 2017: Training
October 2–4, 2017: Tutorials & Conference
New York, NY

Have you tried turning it off and turning it on again?

Tanya Reilly (Google)
1:30pm2:10pm Tuesday, October 3, 2017
Systems Engineering
Location: Gramercy Level: Intermediate

Who is this presentation for?

Systems engineers, architects

Prerequisite knowledge

Understand that distributed systems exist and comprise of various services communicating with each other. This talk is aimed at people designing and building systems, sites and disaster recovery plans.

What you'll learn

* Backups don't stand alone. Anything not tested probably won't work. Failover sites bitrot. * Restrict how systems can be coupled. Start managing your complexity before you think you need it. Be very aware of cycles, global systems that assume they'll always be running somewhere, and dependencies on external entities.


Most of us have a backup strategy and many of us have a restore strategy and several of us have a fully tested restore strategy. But backups are far from the whole story! This talk covers the parts of disaster recovery you might be less prepared for, and the dependencies that you might not think about until one day when you really do turn an entire service, entire site or (perish the thought!) an entire company off and on again.

We’ll look at why the best laid fallback plans tend to go wrong, and why you should start deliberately managing your dependencies long before you think you need to. And we’ll look at dependency cycles that make it difficult or impossible to restart groups of systems. Like, where do you store the documentation on how to recover the documentation server?

Photo of Tanya Reilly

Tanya Reilly


Tanya Reilly has been a Systems Administrator and Site Reliability Engineer at Google since 2005, working on low level infrastructure like distributed locking, load balancing and bootstrapping. Before Google, she was a Systems Administrator at, Ireland’s largest ISP, and before that she was the entire IT Department for a small software house.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)