A Systematic Approach to Capacity Planning in the Real World

Bryce Yan (Twitter), Arun Kejariwal (MZ)
Operations, Mission City Ballroom B4
Presentation: external link
Average rating: ***..
(3.51, 67 ratings)

Scalability is one of the most critical aspects of business at any growing Internet company today. In this regard, capacity planning plays a key role. At Twitter, we employ a pro-active approach rather than reactive. Specifically, we use predictive models to estimate various application and systems metrics going forward (in contrast, a reactive approach may handicap scaling multiple services at the same time and hence can be adversarial to end-user experience). Achieving good prediction is non-trivial owing to, but not limited to, time varying nature of user growth. To this end, we employ multiple techniques (listed below) to drive capacity planning at Twitter.

  • Correlation Analysis
  • Rolling Correlation Analysis
  • Use of Exponential Smoothing
  • Trending Analysis
  • Dynamic Conditional Correlation

In this talk, we shall highlight the pros and cons of the above and walk the audience through the use of the above with use cases at Twitter.

Presentation

Photo of Bryce Yan

Bryce Yan

Twitter

@bryce_yan is currently a Staff Capacity Engineer at Twitter where he works on various techniques to improve the accuracy of our capacity models and demand forecasts. Prior to joining Twitter, @bryce_yan managed or worked in development, performance, QA, databases, and ops in many companies large and small including Marketo, Salesforce.com, and Siebel Systems.

@bryce_yan received his Bachelor’s degree in physics at Berkeley and did his PhD work in quantum field theories at UCLA.

Photo of Arun Kejariwal

Arun Kejariwal

MZ

Arun Kejariwal is a statistical learning principal at Machine Zone (MZ), where he leads a team of top-tier researchers and works on research and development of novel techniques for install and click fraud detection and assessing the efficacy of TV campaigns and optimization of marketing campaigns. In addition, his team is building novel methods for bot detection, intrusion detection, and real-time anomaly detection. Previously, Arun worked at Twitter, where he developed and open-sourced techniques for anomaly detection and breakout detection. His research includes the development of practical and statistically rigorous techniques and methodologies to deliver high-performance, availability, and scalability in large-scale distributed clusters. Some of the techniques he helped develop have been presented at international conferences and published in peer-reviewed journals.

Comments on this page are now closed.

Comments

Samuel Trim
07/01/2013 6:13am PDT

Thanks Arun!

Picture of Arun Kejariwal
Arun Kejariwal
06/27/2013 10:52am PDT

Hi Samuel,

Thanks for your interest. The slides of our talk are available at the following link:

http://www.slideshare.net/arunkejariwal/a-systematic-approach-to-capacity-planning-in-the-real-world

Best, Arun

Samuel Trim
06/27/2013 9:46am PDT

Hi, I was checking in to see if you two would be publishing slides for this presentation. This was very intuitive and I would like to add some more in depth documentation to my notes.

Thanks!

Sponsors

Sponsorship Opportunities

For information on exhibition and sponsorship opportunities at the conference, contact Gloria Lombardo at (203) 381-9245 or glombardo@oreilly.com

Media Partner Opportunities

For media partnerships, contact mediapartners@ oreilly.com

Press and Media

For media-related inquiries, contact Maureen Jennings at maureen@oreilly.com

Contact Us

View a complete list of Velocity contacts