Chaos engineering and war gaming with a flamethrower

Description

Companies operating in the critical path of internet traffic are constantly exposed to DDoS attacks of all types and scales. Ideally, in most at-scale systems, the smaller and more mundane attacks are mitigated automatically. But because scale can vary, and attacks can progress dynamically as attackers get creative, operations teams need to be ready to respond. Certainly, ops teams need the tools and visibility required to mitigate attacks available at their fingertips. But they also need the institutional knowledge and the muscle memory required to dive into the fray and quickly mitigate new attack patterns under pressure.

Shannon Weyrick takes you through how to keep engineering from becoming complacent between major attacks, how you can introduce DDoS identification and mitigation skills to new engineers and build the reflex-level familiarity necessary to succeed under pressure, and how you can continually test your platform and your engineers to ensure resiliency and readiness. The answer’s simple: Consistently attack your own platform.

War game DDoS mitigation exercises helped NS1’s engineers keep the rust off, exercised its tools, uncovered and forced it to address architectural constraints, and built trust and camaraderie among its team. Shannon outlines what NS1 learned along the way using production data and realistic scenarios, tools used to generate attack traffic including the in-house Flamethrower, best practices for operating live fire drills and getting the most out of them, and some of the benefits of pursuing this approach.