Skip to main content

Mitigating User Experience from 'Breaking Bad': The Twitter Approach

Arun Kejariwal (Independent), Piyush Kumar (Twitter Inc.)
Average rating: ***..
(3.25, 8 ratings)

Frequent deployments, large set of in-flight A/B tests, new product launches etc. directly impact the profile of application metrics as well as system metrics. Specifically, the above can induce sudden breakouts – which manifest themselves as a mean-shift or a rampup (these are different from an anomaly) – in the time series of a given metric. Further, the profile on the incoming traffic may also experience a breakout due to a variety of reasons such as, but not limited to, roll out of a new feature or roll out for a new platform; this in turn results in breakouts in application and/or system metrics.

Breakouts can potentially impact performance of the corresponding service and consequently impact the end user experience. To alleviate the impact of breakouts – in other words, preventing user experience from ‘Breaking Bad’ – we developed statistically rigorous techniques to automatically detect breakouts in a timely fashion. The breakouts detected are used to guide capacity planning. In particular, there are two scenarios:

  • Positive breakout: Depending on the magnitude, deploy additionally capacity
  • Negative breakout: Depending on the magnitude, scale down the current capacity

We shall walk the audience through how the techniques are being at Twitter using REAL data.

Photo of Arun Kejariwal

Arun Kejariwal


Arun is currently a Staff Capacity Engineer at Twitter where he works on research and development of novel techniques to improve the accuracy of capacity models and demand forecasts. Prior to joining Twitter, @arun_kejariwal worked on research and development of practical and statistically rigorous methodologies to deliver high performance, availability and scalability in large scale distributed clusters. Some of the techniques developed have been published in peer-reviewed international conferences/journals.

@arun_kejariwal received his Bachelor’s degree in EE from IIT Delhi and doctorate in CS from UCI.

Photo of Piyush Kumar

Piyush Kumar

Twitter Inc.

Piyush is a part of the capacity planning team at Twitter. His interest lies in applying machine learning and statistical techniques to solve technical problems. He is a graduate of School of Computer Science, Carnegie Mellon University, Pittsburgh, PA where he emphasized on machine learning. In his free time he enjoys hiking, playing music, traveling and cooking.

Comments on this page are now closed.


Picture of Piyush Kumar
Piyush Kumar
09/17/2014 1:03pm EDT

Slides are here:

Picture of Arun Kejariwal
Arun Kejariwal
09/16/2014 8:13am EDT

Hi Brian,

I am glad that you likes the content. Thanks for attending the talk. We shall post the slides in a couple of days. Please check Slideshare or our LinkenIn Profiles for the slides.

Brian Peterson
09/16/2014 7:47am EDT

Will the slides for this session be available afterwards? The information and visuals are great, but the speaker was moving so fast I was unable to capture notes before the next slide was up.