Disaster resilience the Waffle House way, from flattops to feature flags and more

Description

Waffle House’s hurricane disaster plan has everything you could want from an IT disaster plan, including contact trees, failover states, and runbooks on partial operation. Although Waffle House is not likely the first place you would look for inspiration about IT disaster resilience, it proves that considering the way other industries handle outages actually helps you construct better state machines to deal with your specific situations and conditions. You could even use status instrumentation to trigger feature flags that route people to different versions of your products, depending on status.

Heidi Waterhouse shares lessons about state drawn from the world outside computers and explains how to quantify them using a finite state machine and implement them automatically while you are in a less-than-perfect condition. You’ll leave with a new way to think about resilience in the face of downtime and the shades of down that can still be operational. You may also depart with a deep need for smothered hash browns.