Build & maintain complex distributed systems
October 1–2, 2017: Training
October 2–4, 2017: Tutorials & Conference
New York, NY

A hands-on data science crash course for modeling and predicting the behavior of (large) distributed systems

Bart De Vylder (CoScale), Pieter Buteneers (CoScale)
9:00am12:30pm Monday, October 2, 2017
Location: Regent Level: Intermediate

Who is this presentation for?

  • DevOps engineers, researchers, and anyone who wants to learn how to get started with data science and apply it to distributed systems

Prerequisite knowledge

  • A working knowledge of Python (equivalent to first five chapters of the Python tutorial)

Materials or downloads needed in advance

  • A WiFi-enabled laptop with the browser of your choice installed

What you'll learn

  • Gain a solid starting point to further explore and apply data science techniques to your infrastructure or domain of interest

Description

Data science is a hot topic. However, the high number of available software libraries, languages, and platforms is often overwhelming for those who want to get started in the field. Bart De Vylder and Pieter Buteneers offer a practical introduction that goes beyond the hype, exploring data analysis and modeling techniques applied to the behavior of distributed systems.

Using hosted IPython notebooks and a real-world dataset of monitoring data originating from a nontrivial distributed application, consisting of both stateful and stateless services communicating over a message bus, Bart and Pieter walk you through the Python scientific ecosystem (NumPy, SciPy, and scikit-learn) as they demonstrate different data visualization techniques that help the interpretation of the data and the models built from it. Bart and Pieter then discuss data clustering techniques, such as those to automatically discover which servers or containers are running in a load-balanced fashion, and show you how to apply correlation analysis and dimensionality reduction techniques. Modern monitoring systems easily capture tens of thousands of metrics, but many of these metrics are highly correlated and don’t convey much extra information. Applying dimensionality reduction techniques to automatically discover these correlations helps in understanding and visualizing the data and is a step in the process of preparing and modeling the data.

Bart and Pieter also outline supervised machine learning techniques to model data and touch on the important concepts of overfitting and cross-validation, considering the advantages and disadvantages of both simple linear techniques and more advanced ones. They then explain how to put these models in action and make predictions, discussing techniques for performing what-if analysis related to capacity planning (e.g., which resource will be the next bottleneck if the number of web requests keeps increasing?) and robustness (e.g., what is the impact on a service’s SLA if a node falls out?).

Photo of Bart De Vylder

Bart De Vylder

CoScale

Bart De Vylder is a data scientist at CoScale. Previously, Bart was active in software engineering and architecture, with a focus on distributed systems. His interests lie in machine learning and building reliable, scalable data processing systems. Bart holds a PhD in artificial intelligence from the Free University of Brussels.

Photo of Pieter Buteneers

Pieter Buteneers

CoScale

Pieter Buteneers is a data strategist and machine learning consultant at CoScale. Pieter is relatively new to the DevOps field. Previously, he was a postdoc at Ghent University, where he did research on AI, machine learning, and deep learning. Pieter has also given a TEDx talk (on a somewhat unrelated subject).

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)