Presented By O'Reilly and Cloudera
Make Data Work
31 May–1 June 2016: Training
1 June–3 June 2016: Conference
London, UK

The future is now: Leveraging Hadoop for real-time, predictive insights

Steven Noels (NGDATA)
11:15–11:55 Friday, 3/06/2016
Hadoop use cases
Location: Capital Suite 10/11 Level: Intermediate
Average rating: ***..
(3.50, 10 ratings)

Prerequisite knowledge

Attendees should have a working knowledge of the Hadoop ecosystem.


Hadoop has been shown to be a great platform for deriving insights from large quantities of data. With Hadoop, organizations can gather and analyze massive amounts of customer data to optimize processes to improve the customer experience, but there is a caveat. Too often, this optimization takes place as part of a series of manual processes—with analytical pipelines running in batches.

Looking at the business context in which these insights are being applied, you quickly find that the ability to act on real-time data is often equally relevant for making a real impact to organizations trying to be truly data driven. Steven Noels explains how to prime the Hadoop ecosystem for real-time data analysis and actionability, examining ways to evolve from batch processing to real-time stream-based processing.

Topics include:

  • The distinction between collecting and analyzing data at rest versus data in motion in the Hadoop ecosystem and how that impacts the relevancy of results
  • How analysis of data in motion can enable organizations to transition from pure analytics (understanding the past) to prediction (scoring the probability of a certain outcome) to optimization (actively guide and proactively influence the future)
  • How to effectively architect applications that can bridge the gap between data at rest and data in motion using a Lambda architectural style
  • The most important traits for real-time big data applications, including a clearly defined and isolated process logic, an embedded framework for actionability, and a permanent memory of historical insights
  • Tips and lessons learned from real customer deployments.
  • Referenced technologies Apache HBase, Kafka, and Spark, as well as industry cases from media and financial services
Photo of Steven Noels

Steven Noels


Steven Noels is the SVP of product at NGDATA, where he is responsible for NGDATA’s overall product strategy and roadmap. Steven cofounded Outerthought—now known as NGDATA—and is the original designer of the Lily platform, which sits at the core of the NGDATA software portfolio. Outerthought was nominated Cool Vendor in Enterprise Content Management by Gartner in 2010. Steven has 15 years of product management experience and is extensively networked with the open source community in and around the Apache Hadoop big data ecosystem. Prior to NGDATA, Steven held senior roles in technology consulting and product management with Alcatel and Wolters-Kluwer, specializing in complex and large-scale data management problems and content publishing. He’s a member of the Apache Software Foundation and holds a board position in the GentBC innovation platform.