Presented By O'Reilly and Cloudera
Make Data Work
March 13–14, 2017: Training
March 14–16, 2017: Tutorials & Conference
San Jose, CA

Guerrilla guide to Python and Apache Hadoop

Juliet Hougland (Cloudera)
1:30pm5:00pm Tuesday, March 14, 2017
Data science & advanced analytics
Location: LL21 B Level: Intermediate
Secondary topics:  Pydata
Average rating: ****.
(4.00, 2 ratings)

Who is this presentation for?

  • Python developers and Python-savvy data scientists

Prerequisite knowledge

  • Python development skills

Materials or downloads needed in advance

  • A WiFi-enabled laptop

What you'll learn

  • Understand how to do full Python development on the Hadoop stack, at Hadoop scale


Using an interactive demo format with accompanying online materials and data, data scientist Juliet Hougland offers a practical overview of the basics of using Python data tools with a Hadoop cluster. Juliet covers HDFS connectivity and dealing with raw data files and running SQL queries with a SQL-on-Hadoop system like Apache Hive or Apache Impala (incubating). She also explores the basics of accessing data with Spark, creating new Spark DataFrames, and implementing the two most common modeling workflows: fitting a model on a single node using scikit, saving the model, and performing embarrassingly parallel model application and fitting a model to distributed data using Spark MLlib.

Topics include:

  • Connecting to HDFS and reading and writing raw data files
  • Connecting to Impala and querying new datasets in HDFS using Ibis or raw SQL
  • Creating partitioned Impala or Hive tables in the Hive metastore
  • Using Python’s data visualization tools as part of an exploratory data analysis
  • Building complex analytic models
Photo of Juliet Hougland

Juliet Hougland


Juliet Hougland is a data scientist at Cloudera and contributor/committer/maintainer for the Sparkling Pandas project. Her commercial applications of data science include developing predictive maintenance models for oil and gas pipelines at Deep Signal and designing and building a platform for real-time model application, data storage, and model building at WibiData. Juliet was the technical editor for Learning Spark by Karau et al. and Advanced Analytics with Spark by Ryza et al. She holds an MS in applied mathematics from the University of Colorado, Boulder and graduated Phi Beta Kappa from Reed College with a BA in math-physics.