Presented By O'Reilly and Cloudera
Make Data Work
March 28–29, 2016: Training
March 29–31, 2016: Conference
San Jose, CA

Real-time Hadoop: What an ideal messaging system should bring to Hadoop

Ted Dunning (MapR Technologies)
11:50am–12:30pm Wednesday, 03/30/2016
IoT and Real-time

Location: 210 C/G
Tags: real-time
Average rating: ***..
(3.78, 9 ratings)

Prerequisite knowledge

Attendees should have a basic understanding of software architecture and components.


Application developers and architects today are interested in making their applications as real-time as possible. To make an application respond to events as they happen, developers need a reliable way to move data as it is generated across different systems, one event at a time. In other words, these applications need messaging.

Messaging solutions have existed for a long time. However, when compared to legacy systems, newer solutions like Apache Kafka offer higher performance, more scalability, and better integration with the Hadoop ecosystem. Kafka and similar systems are based on drastically different assumptions than legacy systems and have vastly different architectures. But do these benefits outweigh any tradeoffs in functionality? Ted Dunning dives into the architectural details and tradeoffs of both legacy and new messaging solutions to find the ideal messaging system for Hadoop.

Topics include:

  • Queues versus logs
  • Security issues like authentication, authorization, and encryption
  • Scalability and performance
  • Handling applications that span multiple data centers
  • Multitenancy considerations
  • APIs, integration points, and more
Photo of Ted Dunning

Ted Dunning

MapR Technologies

Ted Dunning has been involved with a number of startups—the latest is MapR Technologies, where he is chief application architect working on advanced Hadoop-related technologies. Ted is also a PMC member for the Apache Zookeeper and Mahout projects and contributed to the Mahout clustering, classification, and matrix decomposition algorithms. He was the chief architect behind the MusicMatch (now Yahoo Music) and Veoh recommendation systems and built fraud-detection systems for ID Analytics. Opinionated about software and data-mining and passionate about open source, he is an active participant of Hadoop and related communities and loves helping projects get going with new technologies.

Comments on this page are now closed.


Picture of Ted Dunning
Ted Dunning
04/06/2016 6:04am PDT

for the slides.

Arthur Yeo
04/04/2016 4:17am PDT

Where are your slides located?