Presented By O'Reilly and Cloudera
Make Data Work
Feb 17–20, 2015 • San Jose, CA

YARN vs. MESOS: Can’t We All Just Get Along?

Ted Dunning (MapR, now part of HPE)
2:20pm–3:00pm Friday, 02/20/2015
Hadoop & Beyond
Location: 230 C
Average rating: ***..
(3.50, 2 ratings)
Slides:   1-PPTX 

In the battle for datacenter resource management, there are two heavyweights duking it out for the world championship. In the red corner is YARN, a big data contender and the successor to MapReduce 1. In the blue corner is MESOS with it’s UC Berkeley pedigree and it’s proven performance at Twitter, Airbnb and Netflix. This is a battle that Don King would be ecstatic to promote. But maybe we could build a more powerful fighter by combining the best of both. What if you didn’t have to choose? What if you could use both MESOS and YARN in concert, each for what it is especially good at, rather than choosing? In this talk we will cover:

  • The differences between YARN and Mesos
  • How typical datacenters deploy both of these technologies in isolation
  • Why they are seen as competitors
  • How they can, instead, be used together
  • A demonstration of YARN and MESOS collaboratively sharing cluster resources
  • Case studies of actual production implementations
Photo of Ted Dunning

Ted Dunning

MapR, now part of HPE

Ted Dunning is the chief technology officer at MapR, an HPE company. He’s also a board member for the Apache Software Foundation; a PMC member; and committer on a number of projects. Ted has years of experience with machine learning and other big data solutions across a range of sectors. He’s contributed to clustering, classification, and matrix decomposition algorithms in Mahout and to the new Mahout Math library and designed the t-digest algorithm used in several open source projects and by a variety of companies. Previously, Ted was chief architect behind the MusicMatch (now Yahoo Music) and Veoh recommendation systems and built fraud-detection systems for ID Analytics (LifeLock). Ted has coauthored a number of books on big data topics, including several published by O’Reilly related to machine learning, and has 24 issued patents to date plus a dozen pending. He holds a PhD in computing science from the University of Sheffield. When he’s not doing data science, he plays guitar and mandolin. He also bought the beer at the first Hadoop user group meeting.