Build resilient systems at scale
October 12–14, 2015 • New York, NY

Incident management for DevOps

Chris Hawley (Blackrock 3 Partners), Ron Vidal (Blackrock 3 Partners), Rob Schnepp (Blackrock 3 Partners)
3:30pm–5:00pm Monday, 10/12/2015
Location: Regent Parlor
Average rating: ****.
(4.58, 12 ratings)

Prerequisite Knowledge

Bring an open mind and desire to learn about a best practice, forged by the American fire service for managing a "bad day." This is not at all about using technology to solve an emergency situation; it's about people making good decisions and working together. IMS is an optimization tool for people!


Without question, the future of computing promises more scale, more complexity, and certainly more change, all at greater velocity. However, scale, complexity, and change, especially when occurring at ever increasing velocity, are the natural enemies of stability, performance, availability, and reliability.

Many companies have experienced the fear, pain, and embarrassment of handling a technology failure so significant it shook the core of the business both at the time and into the future. Without a standardized way to organize the people responding to incidents and solving technology problems, the time to restore services gets longer and longer.

This session dives into the nuts and bolts of the Incident Management System, which is in use by a number of site reliability teams. Additionally, we describe “how to not let a good crisis go to waste” by learning from each response in productive after action reviews (AAR).

The main points include:

  • Description of the Incident Management System as the best framework to organize the people responding to an incident
  • Explanation of the need and value of a trained incident commander (IC) to lead the response
  • Discussion of the soft skills of how an IC works with subject matter experts (SME) to solve problems
  • The process for conducting after action reviews, using an honest, blameless, and thorough format
  • The case for implementing AAR findings into production to prevent future incidents
Photo of Chris Hawley

Chris Hawley

Blackrock 3 Partners

Chris Hawley is deputy program manager on contract managing the International Counterproliferation Program (ICP) of the Defense Threat Reduction Agency (DTRA), the US Department of Defense’s official combat support agency for countering the entire spectrum of chemical, biological, radiological, nuclear, and high-yield explosive threats globally.

Photo of Ron Vidal

Ron Vidal

Blackrock 3 Partners

Ron Vidal is a partner at Blackrock 3 Partners, a leading incident management firm. Ron’s technology career spans 30 years as a senior executive in critical infrastructure including fiber optic and wireless telecommunications networks, data centers, electric power networks, and oil and gas facilities for Level 3 Communications, MFS Communications, UUNet Technologies, and Kiewit. Ron led teams on $19 billion of M&A transactions and $14 billion of public market financings. Ron managed Level 3’s executive response in New York City after the 9/11 World Trade Center terrorist attack and previously served on Mayor Dinkins’s NYC Task Force on Network Reliability. Ron is a technical peer reviewer for FEMA’s Assistance to Firefighters Grant program and has been a volunteer firefighter in four states. Ron is a member of two working groups on the California Cybersecurity Task Force.

Photo of Rob Schnepp

Rob Schnepp

Blackrock 3 Partners

Rob Schnepp is a 30-year veteran of the fire service and retired as the division chief of special operations for the Alameda County, CA, Fire Department. Rob has vast experience in emergency response and served as incident commander on numerous large-scale emergencies. Rob has written two hazardous materials response textbooks and numerous peer-reviewed fire-service-related articles on incident command. He is an instructor at the National Fire Academy and for the US Defense Threat Reduction Agency, providing hazmat/WMD training to an international audience. Rob is a principal in Blackrock 3 Partners, a firm specializing in consulting, training, and war-gaming in the areas of incident management and command.

Stay Connected

Follow Velocity on Twitter Facebook Group Google+ LinkedIn Group


More Videos »

O’Reilly Media

Tech insight, analysis, and research