How We Survived Hurricane Sandy: A Look At What Not To Do

Operations, Sutton North
Average rating: ***..
(3.38, 8 ratings)
Slides:   1-PDF 

In October of 2012, half of Manhattan went dark as Hurricane Sandy knocked out power throughout the tristate area. For those companies without adequate disaster response plans in place, it was a long week of trying to do something – anything – to get their site back up. This talk will focus on the first-hand experience of a social network whose servers were co-located in a lower Manhattan datacenter that lost power. By doing so, we hope that Velocity participants will have a good template of what they shouldn’t be doing.

Some highlights of our misery:

  • Office AND datacenter lost power
  • Long TTLs on DNS entries… in zone files on DNS servers that were offline due to the outage
  • IT ninjas going into dark buildings to pull servers out and carrying them up and down dozens of flights of stairs
  • Setting up a temporary “datacenter” in our DBA’s house
  • Moving our codebase into “the cloud” and re-architecting on-the-fly
  • Trying to coordinate updates and response internally and externally

In this talk, we plan on presenting what our infrastructure looked like prior to the storm, how we handled the outage, and what precautions and methods we put in place after our site returned to normalcy. We will also discuss the organizational challenges we have and continue to face with regards to resource allocation and defining the importance of emergency preparedness.

Hurricane Sandy taught us quite a bit about our own fallibility. Hopefully, by telling our story at Velocity, we can help other companies who have not thought about or dedicated time to disaster response.

Mike Zaic

Wild Sky Media

Mike Zaic is the Vice President of Engineering at CafeMom, the largest social network dedicated to moms. Prior to CafeMom, he worked for small to medium sized startups, helping them to grow from inception to millions of users.

Dan White

CafeMom

Dan White is the Senior Vice President of Technology at CafeMom.com. He has a special knack for building great databases and making things work.

Sponsors

Sponsorship Opportunities

For information on exhibition and sponsorship opportunities at the conference, contact Gloria Lombardo at (203) 381-9245 or glombardo@oreilly.com

Media Partner Opportunities

For media partnerships, contact mediapartners@ oreilly.com

Press and Media

For media-related inquiries, contact Maureen Jennings at maureen@oreilly.com

Contact Us

View a complete list of Velocity contacts