Presented By O'Reilly and Cloudera
Make Data Work
March 28–29, 2016: Training
March 29–31, 2016: Conference
San Jose, CA

Real-time Hadoop: What an ideal messaging system should bring to Hadoop

Ted Dunning (MapR, now part of HPE)
11:50am–12:30pm Wednesday, 03/30/2016
IoT and Real-time

Location: 210 C/G
Tags: real-time
Average rating: ***..
(3.78, 9 ratings)

Prerequisite knowledge

Attendees should have a basic understanding of software architecture and components.


Application developers and architects today are interested in making their applications as real-time as possible. To make an application respond to events as they happen, developers need a reliable way to move data as it is generated across different systems, one event at a time. In other words, these applications need messaging.

Messaging solutions have existed for a long time. However, when compared to legacy systems, newer solutions like Apache Kafka offer higher performance, more scalability, and better integration with the Hadoop ecosystem. Kafka and similar systems are based on drastically different assumptions than legacy systems and have vastly different architectures. But do these benefits outweigh any tradeoffs in functionality? Ted Dunning dives into the architectural details and tradeoffs of both legacy and new messaging solutions to find the ideal messaging system for Hadoop.

Topics include:

  • Queues versus logs
  • Security issues like authentication, authorization, and encryption
  • Scalability and performance
  • Handling applications that span multiple data centers
  • Multitenancy considerations
  • APIs, integration points, and more
Photo of Ted Dunning

Ted Dunning

MapR, now part of HPE

Ted Dunning is the chief technology officer at MapR, an HPE company. He’s also a board member for the Apache Software Foundation, a PMC member, and committer on a number of projects. Ted has years of experience with machine learning and other big data solutions across a range of sectors. He’s contributed to clustering, classification, and matrix decomposition algorithms in Mahout and to the new Mahout Math library and designed the t-digest algorithm used in several open source projects and by a variety of companies. Previously, Ted was chief architect behind the MusicMatch (now Yahoo Music) and Veoh recommendation systems and built fraud-detection systems for ID Analytics (LifeLock). Ted has coauthored a number of books on big data topics, including several published by O’Reilly related to machine learning, and has 24 issued patents to date plus a dozen pending. He holds a PhD in computing science from the University of Sheffield. When he’s not doing data science, he plays guitar and mandolin. He also bought the beer at the first Hadoop user group meeting.

Comments on this page are now closed.


Picture of Ted Dunning
Ted Dunning
04/06/2016 6:04am PDT

for the slides.

Arthur Yeo
04/04/2016 4:17am PDT

Where are your slides located?