Presented By O’Reilly and Cloudera
Make Data Work
March 5–6, 2018: Training
March 6–8, 2018: Tutorials & Conference
San Jose, CA

Radically modular data ingestion APIs in Apache Beam

Eugene Kirpichov (Google)
11:50am12:30pm Wednesday, March 7, 2018
Secondary topics:  Data Integration and Data Pipelines
Average rating: ****.
(4.75, 4 ratings)

Who is this presentation for?

  • Senior engineers

Prerequisite knowledge

  • Experience with distributed data processing using any popular framework

What you'll learn

  • Understand the power and challenges of a unified batch/streaming programming model and the unique challenges of a batch/streaming agnostic API for data ingestion
  • Appreciate the modularity and composability advantages offered by treating data ingestion as just another data processing task
  • Learn to build highly modular data ingestion APIs using the new Beam programming model primitive Splittable DoFn


Apache Beam equips users with a novel programming model in which the classic batch/streaming data processing dichotomy is erased. Beam also offers a rich set of I/O connectors to popular storage systems. Beam adopts the philosophy that interacting with a storage system is just another parallel data processing task, so the I/O connectors are packaged as simple Beam transforms. However, the batch/streaming dichotomy still existed for authors of new I/O connectors: it is common knowledge that efficiently ingesting batch and streaming data is fundamentally different and, at a low level, requires fundamentally different (and rather heavyweight) APIs.

Over the years Google has identified a number of issues with these APIs, and its attempts to improve them have yielded an unexpected result. It generalized the most fundamental data processing primitive: the Map operation (DoFn in Beam). The generalization is called Splittable DoFn, and it not only allows developing data ingestion APIs in a way agnostic to batch versus streaming but is lightweight and transparently blends with the rest of the Beam programming model, enabling and popularizing new, highly modular design patterns for data ingestion APIs. Eugene Kirpichov details the modularity and composability advantages created by treating data ingestion as just another data processing task and walks you through building highly modular data ingestion APIs using Splittable DoFn.

Photo of Eugene Kirpichov

Eugene Kirpichov


Eugene Kirpichov is a staff software engineer on the Cloud Dataflow team at Google, where he works on the Apache Beam programming model and APIs. Previously, Eugene worked on Cloud Dataflow’s autoscaling and straggler elimination techniques. He is interested in programming language theory, data visualization, and machine learning.