Engineering the Future of Software
Feb 3–4, 2019: Training
Feb 4–6, 2019: Tutorials & Conference
New York, NY
Please log in

Anatomy of testing in production: A Netflix original case study

Vasanth Asokan (Netflix)
4:50pm–5:40pm Tuesday, February 5, 2019
DevOps & Continuous Delivery
Location: Trianon Ballroom
Secondary topics:  Best Practice, Case Study
Average rating: ****.
(4.26, 23 ratings)

Who is this presentation for?

  • Software architects, software engineers, architects, senior software engineers, DevOps engineers, build engineers, and QA engineers



Prerequisite knowledge

  • Familiarity with microservices (including testing) and distributed systems

What you'll learn

  • Explore Netflix's patterns for testing large-scale distributed systems
  • Understand the benefits and risks of these patterns


So you want to test your complex application that involves large-scale distributed systems. But how do you feel about testing it effectively just using your test environment? Today, automated testing of Netflix client and server applications runs at scale in production. Within a few years, the company’s testing has gone from a low-volume manual mode to one where it is continuous, voluminous, and fully automated. Collectively, Netflix teams create hundreds of thousands of tester accounts every day, each being used in thousands of test scenarios, to the point where service providers are more wary of getting paged for causing instability to internal testers than for causing an external outage.

Vasanth Asokan offers a study of the evolution and anatomy of production testing at scale at Netflix, explaining why there was a desire to test in production, what Netflix did to try to keep testing out of production, and where testing belongs, anyway. Along the way, Vasanth shares a few case studies to demonstrate both the benefits and the less tangible diffused impacts of concentrated, uncoordinated testing against customer-facing infrastructure. Vasanth also looks at other forms of testing, such as load, failure, and simulation testing, and explains the role they play in ensuring a fully functioning customer experience.

Join in to learn whether the benefits outweigh the risks of executing untested code in production or whether it’s better to focus on creating a production mirror. If you run large-scale distributed systems, this talk will better inform your overall testing strategy, illustrate specific techniques that work at scale, and provide trade-offs to consider.

Photo of Vasanth Asokan

Vasanth Asokan


Vasanth Asokan is an Engineering Leader at Netflix, where he heads a developer productivity team for large-scale microservices. Service oriented architectures, continuous integration and delivery, automation, testing, resiliency, serverless trends, developer experience, and education are favorite topics. In a former phase of his career, he focussed on embedded SoC development and EDA tools, compilers, Embedded RTOS-es, and Eclipse plug-in development. He likes exploring vague opportunities, building bridges between ideas and solving problems (both human and technical). Curious by nature and people oriented, he has a high regard for products, processes, and engineering that actually reach people in meaningful ways.