Presented By O'Reilly and Cloudera
Make Data Work
Sept 29–Oct 1, 2015 • New York, NY

Apache Hadoop operations for production systems

Kathleen Ting (Cloudera), Miklos Christine (Databricks), Philip Zeyliger (Cloudera), Philip Langdale (Cloudera)
9:00am–9:15am Tuesday, 09/29/2015
Production Ready Hadoop
Location: 1 E14/1 E15 Level: Intermediate
Average rating: ***..
(3.73, 15 ratings)

Materials or downloads needed in advance

Please bring a laptop with internet access, web browser, and a ssh client.

Description

(Full Day Tutorial)

Hadoop is emerging as the standard for big data processing and analytics. However, as usage of the Hadoop clusters grow, so do the demands of managing and monitoring these systems.

In this full-day tutorial, attendees will get an overview of all phases for successfully managing Hadoop clusters, with an emphasis on production systems — from installation, to configuration management, service monitoring, troubleshooting and support integration.

We will review tooling capabilities and highlight the ones that have been most helpful to users, and share some of the lessons learned and best practices from users who depend on Hadoop as a business-critical system.

Proposed Agenda Topics

  • Installation (hardware considerations, OS prerequisites, sanity testing, security considerations)
  • Configuration (mechanics, key configurations, resource management)
  • Troubleshooting (managing, troubleshooting, and debugging Hadoop clusters and applications)
    * Enterprise Considerations (scaling, logs, failure testing)
Photo of Kathleen Ting

Kathleen Ting

Cloudera

Kathleen Ting is currently a technical account manager at Cloudera, where she helps strategic customers deploy and use the Hadoop ecosystem in production. Kathleen has spoken on Hadoop, ZooKeeper, and Sqoop at many big data conferences, including Hadoop World, ApacheCon, and OSCON. She’s contributed to several projects in the open source community, is a committer and PMC Member on Sqoop, and is a coauthor of the Apache Sqoop Cookbook.

Photo of Miklos Christine

Miklos Christine

Databricks

Miklos Christine is a solutions engineer for Databricks. Miklos was previously a system engineer at Cloudera where he helped strategic customers deploy and use the Apache Hadoop ecosystem in production. He has contributed to several projects in the open source community, previously worked on the design and implementation of the system infrastructure for the OS that runs on Cisco’s routers and switches, and holds a BS in electrical engineering and computer sciences from the University of California-Berkeley.

Photo of Philip Zeyliger

Philip Zeyliger

Cloudera

Philip Zeyliger is a software engineer at Cloudera. He came to Cloudera from Google, where he worked on scalable storage for user-facing applications. Before that, he worked in finance at D.E. Shaw. Philip holds a bachelor’s degree in mathematics from Harvard University. His interests include systems and databases. He’s a committer on the Apache Avro project.

Photo of Philip Langdale

Philip Langdale

Cloudera

Philip Langdale is the engineering lead for cloud at Cloudera. He joined the company as one of the first engineers building Cloudera Manager and served as an engineering lead for that project until moving to working on cloud products. Previously, Philip worked at VMware, developing various desktop virtualization technologies. Philip holds a bachelor’s degree with honors in electrical engineering from the University of Texas at Austin.

Comments on this page are now closed.

Comments

Picture of Kathleen Ting
Kathleen Ting
10/30/2015 3:14pm EDT

@Abdul – you can download the slides here: http://tiny.cloudera.com/hadoop-ops-nyc-slides

Abdul Khan
10/29/2015 3:58pm EDT

Can I have access to slides or reading materials?

Picture of Kathleen Ting
Kathleen Ting
10/28/2015 2:41pm EDT

@Ravi – the best way to prevent latency issues is to colocate your CM server and relational database in the same datacenter (which some consider to be inside the cluster).

@Craig – in all three case studies, the problems were comparatively easy to diagnose: slow jobs. The hard part was pinpointing which part of the stack – Hadoop daemons, JVM, Linux, or Disk/CPU/Memory – caused the slow jobs. The 2012 leap second bug case study was probably the fastest one to pinpoint as we had 30 customers all experiencing high CPU usage and a responsive but sluggish cluster at the exact same time. From there we checked the kernel message buffer (by running dmesg) and looked for output confirming the leap second injection.

Craig McDonough
09/29/2015 8:03am EDT

For the Troubleshooting case studies, how long or how much effort did it take to identify the problem?

Ravi Sivakumaran
09/29/2015 5:31am EDT

Does the utility node have to be part of the cluster for hosting relational database (or) Can it exist outside the cluster? If it is outside, does the proximity and network bandwidth matter, if so, how?

Picture of Kathleen Ting
Kathleen Ting
09/28/2015 5:57pm EDT

@Greg – best to pre-register but there should be room. Looking forward to seeing you there.

@Ravi – as it is, there is nothing to download (besides the ssh key, for which we will provide a shortened link). All lab work will be done via specially configured Cloudera Live instances (URLs to be passed out during the tutorial).

Greg Huber
09/28/2015 5:09pm EDT

Kate:
Do we need to pre-register for this or just show up?

Ravi Sivakumaran
09/28/2015 12:40pm EDT

Is it possible to download the tutorial material ahead of time (before the start of event)?

Thanks, Ravi

Picture of Kathleen Ting
Kathleen Ting
09/18/2015 3:05pm EDT

Hi Tyleach, thanks for your interest and there’s no specific SSH client suggested. If you plan to participate in the hands-on lab, best to bring a laptop as there will be a fair amount of typing (of Hadoop commands, etc).

Tyleach Watkins
09/18/2015 9:02am EDT

Is a specific ssh client required/suggested? Is it at all possible to run the workshop on a tablet/ipad?

Picture of Kathleen Ting
Kathleen Ting
09/08/2015 3:09pm EDT

Hi Vijay – yes, this will be a hands-on workshop and thanks for asking. Please bring a laptop with internet access, a web browser, and a ssh client to participate in the lab portion of this tutorial.

Vijay Maddipoti
05/28/2015 12:18pm EDT

Is it a hands-on workshop?