Skip to main content
Make Data Work
Oct 15–17, 2014 • New York, NY

Hadoop Operations - Best Practices from the Field

Chris Nauroth (Hortonworks), Suresh Srinivas (Hortonworks)
1:45pm–2:25pm Friday, 10/17/2014
Hadoop Platform
Location: Hall A 23/24
Average rating: ****.
(4.25, 4 ratings)
Slides:   1-PPTX 

You’ve successfully deployed Hadoop, but are you taking advantage of all of Hadoop’s features to operate a stable and effective cluster? In the first part of the talk, we will cover issues that have been seen over the last two years on hundreds of production clusters with detailed breakdown covering the number of occurrences, severity, and root cause. We will cover best practices and many new tools and features in Hadoop added over the last year to help system administrators monitor, diagnose and address such incidents.

The second part of our talk discusses new features for making daily operations easier. This includes features such as ACLs for simplified permission control, snapshots for data protection and more. We will also cover tuning configuration and features that improve cluster utilization, such as short-circuit reads and datanode caching.

Photo of Chris Nauroth

Chris Nauroth

Hortonworks

Chris Nauroth is a software engineer on the HDFS team at Hortonworks. He is an active contributor across the lowest layers of the Hadoop ecosystem: Hadoop Common, HDFS, YARN, and MapReduce. His most recent work includes porting Hadoop to Windows, several features to improve HDFS stability and HDFS ACLs. Prior to joining Hortonworks, Chris deployed and maintained Disney’s Hadoop infrastructure and developed web services and MapReduce jobs to make integration easier for Disney’s business units.

Photo of Suresh Srinivas

Suresh Srinivas

Hortonworks

Suresh is an Apache Hadoop committer and member of Apache Hadoop Project Management Committee (PMC). He is a long term active contributor to the Apache Hadoop project and has designed and developed many significant features for Hadoop. Prior to co-founding Hortonworks, he served as a software architect at Yahoo! working on Apache Hadoop, where he developed features and supported some of the largest installations of Hadoop clusters. .

Comments on this page are now closed.

Comments

Picture of Chris Nauroth
Chris Nauroth
10/13/2014 9:35am EDT

Hello, Shankar. Thank you for the comment.

Our presentation draws on our operations experience supporting Hortonworks Data Platform, but most of the advice is distro-agnostic. We focus on tools that are made available upstream in Apache Hadoop, so even if you’re running a different distro, I expect you’ll still find something valuable in our content.

Picture of Nara Neel
Nara Neel
10/13/2014 6:31am EDT

is this best practices based on hortonworks haddop cluster? or is it generic.