Skip to main content

Some Simple Math to get Some Signal out of Your Ops Data Noise

Toufic Boubez (Metafor Software)
Mission City Ballroom B4
Average rating: ****.
(4.00, 20 ratings)

You’ve instrumented your system and application to the hilt. You can now “measure all the things”. Your team has set up thousands of metrics collecting millions of data points a day. Now what?

Analyzing this mountain of data and extracting signal from the noise is not easy, so most IT ops teams only keep an eye on a small fraction of the metrics they collect, or they run some simplistic analytics that don’t generate any useful information. The choice of what analytic method to use ranges from simple statistical analysis to sophisticated machine learning techniques. And one algorithm doesn’t fit all data.

While the more advanced algorithms require math nerds with PhD’s to develop, there are some basic statistical methods that anyone can implement and they can provide surprisingly valuable insights. In this talk, Toufic will show you how to determine the distribution of your data and explain why this is important. We will then explore a few statistical techniques that are appropriate for the most common distributions and discuss their pros and cons. Yes, there will be math in this talk but you will be able to follow along. The goal is for you to walk away with at least one technique that you can apply right away in your monitoring environment to improve the signal to noise ratio and get information out of your data.

Photo of Toufic Boubez

Toufic Boubez

Metafor Software

Toufic has been passionate about machine learning for over 20 years. Prior to Metafor, he was the founder and CTO of Layer 7 Technologies, a leader in API security and management and recently acquired by CA. Prior to Layer 7, Toufic was the founding CTO of Saffron Technology, a big data analytics company specializing in associative memory technology. Toufic is also a well-known SOA and Web Services pioneer and was Chief Architect for Web Services at IBM’s Software Group, where he founded the first SOA team and drove IBM’s early XML and Web Services strategies. He was the co-editor of the W3C WS-Policy specification, and the co-author of the OASIS WS-Trust, WS-SecureConversation, and WS-Federation submissions. He has also participated on the OASIS WS-Security, SAML and UDDI Technical Committees. He is the author of many publications, articles and several books, including “Building Web Services with Java”, and is one of the co-authors of the SOA Manifesto. Toufic holds a Master of Electrical Engineering degree from McGill University and a Ph.D. in Biomedical Engineering from Rutgers University.