Presented By O'Reilly and Cloudera
Make Data Work
Sept 29–Oct 1, 2015 • New York, NY

HDFS operations made easy: Guide to the improved, full service HDFS File Browser

Ravi Prakash (Altiscale)
2:05pm–2:45pm Thursday, 10/01/2015
Hadoop Internals & Development
Location: 1 E16 / 1 E17 Level: Intermediate
Average rating: ***..
(3.83, 6 ratings)
Slides:   1-PDF 

In the 2.4.0 release, a new and improved HTML5-based native Web UI was introduced in Hadoop. The new WebUI used the WebHDFS API to talk to HDFS. However, it lacked write capabilities. Many customers requested this slick new UI to be improved to a full-service interface, including:

1. Ability to create directories: mkdir
2. Ability to change permissions: chmod
3. Ability to change owner and group: chown, chgrp
4. Ability to upload files
5. Ability to delete files and directories
6. Sort based on columns

This presentation will cover the challenges to achieving these features and how these challenges were solved, such as how the WebHDFS API was modified. If there is time, we will also cover how HttpFS was used to act as a single gateway for channeling all traffic.

Although there are several frameworks that also provide this functionality, we believe that such basic features should be included natively in Hadoop, without requiring additional components. We also show how this may be more efficient than an additional server proxying all traffic.

Future upgrades will also be covered, such as functionality for multiple file/directory uploads and other features to further improve the experience of HDFS users.

Photo of Ravi Prakash

Ravi Prakash

Altiscale

Ravi Prakash is a Hadoop committer and a senior software engineer at Altiscale. Previously, he was a senior software developer at Yahoo!, where he worked on Hadoop Core development (HDFS, MapReduce, and YARN). Ravi has also worked in software development at Tavare Research Labs and Motorola. Ravi has a BS in computer science from GGS Indraprastha University and an MS in computer science from the University of Southern California.