All Software Architecture, All the Time
June 10-13, 2019
San Jose, CA
Please log in

How did things go right: Learning more from incidents (Velocity)

1:25pm–2:10pm Wednesday, June 12, 2019
Overcoming Obstacles: Lessons in Resilience
Location: Expo Hall Sessions
Average rating: ****.
(4.00, 1 rating)

Who is this presentation for?

  • SREs, DevOps, production engineering, and incident management



Prerequisite knowledge

  • Familiarity with an incident

What you'll learn

  • Learn what creates a useful and readable incident investigation
  • Ask the right questions to you dive deep to find contributing factors rather than stopping at a root cause
  • Explore the conditions that existed at the time that allowed an incident to occur
  • Discover which conversations to begin in your organizations to find out how normal work is successful
  • Ensure the pressure to learn outweighs the pressure to fix


Solely learning from failure isn’t a fundamental—it’s a limitation. A look into the new view of safety, human, and organizational performance and resilience engineering shows us that safety, great performance, and sources of resilience do not come from the absence of failure but rather the presence of adaptive capacity.

Navigating a perfect storm in a world where availability is made up and the nines don’t matter requires expertise. Ryan Kitchens details more rewarding ways to approach incident investigation without overly focusing on failure prevention by asking what’s going on when it seems like nothing is happening; exploring what’s going to keep failure from being worse when it does occur; examining how teams adapt successfully when preventative techniques fail; and diving into how we should prioritize the effort to develop systems that help us safely manage the consequences of failure. These can’t be resolved by trying to explain the causes of failure and fixing remediation items. We move the needle forward and increase our opportunity for learning from success with some fundamental and practical ways that get us from “Why did things go wrong?” to “How did things go right?”