Build & maintain complex distributed systems
October 1–2, 2017: Training
October 2–4, 2017: Tutorials & Conference
New York, NY

Debugging complex systems

Terran Melconian (Air Network Simulation and Analysis)
1:30pm2:10pm Wednesday, October 4, 2017
Average rating: ***..
(3.50, 4 ratings)

Who is this presentation for?

  • Engineers who work in operations

Prerequisite knowledge

  • Experience writing software and providing services on the web
  • Familiarity with diagnosing faults in complex systems with multiple interoperating components (useful but not required)

What you'll learn

  • For novice engineers, learn how to debug complex systems
  • For experienced engineers and managers, learn an explicit, conscious process to teach novice engineers how to debug complex systems


Skills for diagnosing failures in complex, interacting systems are critically important but rarely taught. Even those with experience and expertise can struggle to articulate how they do what they do in order to pass the knowledge on. Drawing on a combination of his own experience carrying the pager and time spent observing and teaching others, Terran Melconian has distilled an explicit, teachable process for efficiently isolating faults.

Starting with the observed symptom (for example, a page), Terran demonstrates how to draw a diagram of possible causes, which bifurcate the search space, and how to collect new observations to decide which path to follow at each fork. He also shares a very common anti-pattern—observe a symptom, hypothesize a cause for the fault, and write and deploy code to address this cause—explains why this process often fails to produce effective results, and outlines what to do instead.

Photo of Terran Melconian

Terran Melconian

Air Network Simulation and Analysis

Terran Melconian is a data science consultant and trainer at Air Network Simulation and Analysis. Terran has worked in the consumer web space for the last decade in roles that included software development, operations, data warehousing, and data science. Previously, he built up and managed teams at TripAdvisor and Jobcase, where he focused on hiring generalists and teaching them about the specifics. Terran is passionate about continuing education for high-performing professionals.