Using Hadoop for Big Data Analysis

“Big data” problems are increasingly common. Storage and compute services are inexpensive and easy to get in the cloud. New data sources — sensor readings, video and still imagery, audio, telemetry from software systems and devices, web logs, and data from the biological and physical sciences are exploding, and it is now possible to store them on-line cheaply.

Apache Hadoop is a powerful tool for analyzing these new, very diverse, repostories. Hadoop can scan and analyze petabytes of data using a collection of commodity servers working in parallel.

Hadoop enforces a different programming paradigm from relational databases, large-scale data warehouses and special-purpose high-performance computing systems. Hadoop uses shared-nothing parallel execution to break large data processing tasks into small pieces that get distributed among many servers. Those tasks can include code written by the user that operates on complex data in its native format. Hadoop relies on a high-performance distributed file system, HDFS, for data storage and replication.

In this talk, I’ll explain what Hadoop and related open source projects do, how they operate, and how they are used in real-world workloads to answer questions that simply can’t be posed using other systems.