Apache Beam equips users with a novel programming model in which the classic batch-streaming data processing dichotomy is erased. Beam also offers a rich set of I/O connectors to popular storage systems. Beam adopts the philosophy that interacting with a storage system is just another parallel data-processing task, so the I/O connectors are packaged as simple Beam transforms, which offers surprisingly large benefits in both modularity and scalability.
However, the batch-streaming dichotomy used to still exist for authors of new I/O connectors: it was common knowledge that efficiently ingesting batch and streaming data is fundamentally different and, at a low level, requires fundamentally different (and rather heavyweight) APIs.
Over the years, a number of issues with these APIs have been identified, and attempts to improve them have yielded an unexpected result—the most fundamental data processing primitive, the Map operation (DoFn in Beam), was generalized. Eugene Kirpichov offers an overview of this generalization—called Splittable DoFn in Beam—which not only allows developing data ingestion APIs in a way agnostic to batch versus streaming but is lightweight and transparently blends with the rest of the Beam programming model, enabling and popularizing new, highly modular design patterns for data ingestion APIs.
Eugene Kirpichov is a staff software engineer on the Cloud Dataflow team at Google, where he works on the Apache Beam programming model and APIs. Previously, Eugene worked on Cloud Dataflow’s autoscaling and straggler elimination techniques. He is interested in programming language theory, data visualization, and machine learning.
©2018, O’Reilly UK Ltd • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com