Presented By O'Reilly and Cloudera
Make Data Work
March 28–29, 2016: Training
March 29–31, 2016: Conference
San Jose, CA

Real-time Hadoop: What an ideal messaging system should bring to Hadoop

Ted Dunning (MapR Technologies)
11:50am–12:30pm Wednesday, 03/30/2016
IoT and Real-time

Location: 210 C/G
Tags: real-time
Average rating: ***..
(3.78, 9 ratings)

Prerequisite knowledge

Attendees should have a basic understanding of software architecture and components.

Description

Application developers and architects today are interested in making their applications as real-time as possible. To make an application respond to events as they happen, developers need a reliable way to move data as it is generated across different systems, one event at a time. In other words, these applications need messaging.

Messaging solutions have existed for a long time. However, when compared to legacy systems, newer solutions like Apache Kafka offer higher performance, more scalability, and better integration with the Hadoop ecosystem. Kafka and similar systems are based on drastically different assumptions than legacy systems and have vastly different architectures. But do these benefits outweigh any tradeoffs in functionality? Ted Dunning dives into the architectural details and tradeoffs of both legacy and new messaging solutions to find the ideal messaging system for Hadoop.

Topics include:

  • Queues versus logs
  • Security issues like authentication, authorization, and encryption
  • Scalability and performance
  • Handling applications that span multiple data centers
  • Multitenancy considerations
  • APIs, integration points, and more
Photo of Ted Dunning

Ted Dunning

MapR Technologies

Ted Dunning is chief applications architect at MapR Technologies. He’s also a board member for the Apache Software Foundation, a PMC member and committer of the Apache Mahout, Apache Zookeeper, and Apache Drill projects, and a mentor for various incubator projects. Ted contributed to clustering, classification, and matrix decomposition algorithms in Mahout and to the new Mahout Math library. He also designed the t-digest algorithm used in several open source projects and by a variety of companies. Previously, Ted was chief architect behind the MusicMatch (now Yahoo Music) and Veoh recommendation systems and built fraud-detection systems for ID Analytics (LifeLock). Ted has coauthored a number of books on big data topics, including several published by O’Reilly related to machine learning, and has 24 issued patents to date. He holds a PhD in computing science from the University of Sheffield. He is on Twitter as @ted_dunning.

Comments on this page are now closed.

Comments

Picture of Ted Dunning
Ted Dunning
04/06/2016 6:04am PDT

See http://www.slideshare.net/tdunning/real-timehadoop
for the slides.

Arthur Yeo
04/04/2016 4:17am PDT

Ted,
Where are your slides located?