All Software Architecture, All the Time
June 10-13, 2019
San Jose, CA

How I failed to build a runbook automation system and what I learned (Velocity)

1:15pm–2:00pm Thursday, June 13, 2019
Overcoming Obstacles: Lessons in Resilience
Location: Expo Hall Sessions
Average rating: ****.
(4.00, 1 rating)

Level

Non-technical

Prerequisite knowledge

  • Familiarity with DevOps or SRE concepts (useful but not required)

What you'll learn

  • Understand how a solution failed and learn from the experience

Description

Our intentions can be good, the technical ability and time may be there, and we’re going to build the thing to make work easier and more productive, allowing everyone to apply their labor to only the most valuable tasks—yet sometimes it’s still not enough. This is a postmortem of a solution that was designed to solve a common operational problem but failed.

Tim Bonci examines the scars and hopefully provides insights into finding and addressing the right problems in the right places that should be broadly useful in building and deploying your own transformational processes and tools. This is particularly relevant to brownfields teams looking for ways to modernize their processes and anyone who struggles with needing humans to change how they work. Tim explains why shifting human processes to computer automation does not always produce the expected results and how treating nonurgent alerts as a work queue is an anti-pattern.