Presented By O'Reilly and Cloudera
Make Data Work
March 13–14, 2017: Training
March 14–16, 2017: Tutorials & Conference
San Jose, CA

Deep learning for IT operations intelligence using open source tools

Shivnath Babu (Duke University | Unravel Data Systems)
4:20pm5:00pm Thursday, March 16, 2017
Secondary topics:  Deep learning
Average rating: *****
(5.00, 1 rating)

Deep learning has enabled massive breakthroughs in speech recognition, visual object detection, and natural language processing. Beyond image, text, and voice processing, companies like Apple have applied deep learning to detect fraud in the Apple store and to extend battery life on all personal communication devices.

Shivnath Babu offers an introduction to using deep learning to solve complex problems in IT operations analytics. Shivnath focuses on how deep learning can derive operations insights automatically for the complex big data application stack composed of systems such as Hadoop, Spark, Cassandra, Elasticsearch, and Impala, using examples of open source tools for deep learning. These insights enable IT staff to ensure that the full big data application stack works reliably and with efficient resource usage.

IT monitoring technology today collects more than a hundred types of monitoring data from all levels of the big data application stack. Deep learning techniques can discover intricate structure in such large and complex datasets to facilitate data correlation, classification, organization, and prediction. Shivnath highlights how these features allow personnel in different roles within an enterprise to function much more efficiently and intelligently than before so that:

  • Data analyst/scientists can automatically speed up their applications running on the big data application stack without having to worry about the hundreds of tuning knobs in the stack. Deep learning models can quickly pinpoint optimal configurations for these tuning knobs from billions of possible choices.
  • IT ops/managers can identify performance problems and their root causes more easily and can identify potential bottlenecks, balance load, and tune factors that influence cluster performance. By using recommendations from deep learning models, reliance on trial-and-error methods can be reduced, saving a tremendous amount of time and lowering system operations costs.
  • CIOs can compare cost against performance of running applications specific to their organization on various platforms for big data. Deep learning can generate what-if models to analyze and simulate the impact of hypothetical changes on application behavior on platforms such as on-premises and the cloud, helping CIOs make intelligent decisions on resource acquisition, allocation, and capacity planning.
Photo of Shivnath Babu

Shivnath Babu

Duke University | Unravel Data Systems

Shivnath Babu is an associate professor of computer science at Duke University, where his research focuses on ease of use and manageability of data-intensive systems, automated problem diagnosis, and cluster sizing for applications running on cloud platforms. He is also the chief scientist at Unravel Data Systems, the company he cofounded to solve the application management challenges that companies face when they adopt systems like Hadoop and Spark. Unravel originated from the Starfish platform built at Duke, which has been downloaded by over 100 companies. Shivnath has received a US National Science Foundation CAREER Award, three IBM Faculty Awards, and an HP Labs Innovation Research Award. He has given talks and distinguished lectures at many research conferences and universities worldwide. Shivnath has also spoken at industry conferences, such as the Hadoop Summit.