July 20–24, 2015
Portland, OR

Beyond messaging: Enterprise dataflow with Apache NiFi

Joe Witt (Onyara Inc.)
10:00am–10:40am Friday, 07/24/2015
Data Portland 252
Average rating: ****.
(4.77, 13 ratings)
Slides:   1-PDF 

Prerequisite Knowledge

Familiarity with messaging systems (JMS, Kafka) and the sorts of use cases they are designed to address is recommended.


If your organization is addressing the challenges and opportunities that big data presents, then you’re also an organization wrestling with dataflow. Dataflow is all about connecting systems. But in large organizations that are highly distributed, these systems almost certainly speak different protocols and understand different data formats – and that nature of connectivity changes all the time. Messaging-based solutions are a popular answer these days, but they don’t address many of the fundamental challenges of enterprise dataflow.

This talk will outline the broad range of capabilities needed in an end-to-end data flow strategy, and will put the value of messaging-based solutions in context. Systems throughout an enterprise evolve at different rates. Architectures are in a constant state of transformation. Ensuring compliance of distributed systems at scale requires new lineage tracking techniques. The idea of ‘real-time’ is more than just the delivery latency of a message. It is critical that organizations understand and have total control of their information supply chain.

In late 2014 the National Security Agency (NSA) donated to the Apache Software Foundation the project now known as Apache NiFi (incubating). NiFi resulted from more than eight years of lessons learned and development designed to tackle the complex and broad challenges of dataflow at scale. This talk details those lessons learned and the key features of NiFi.

Photo of Joe Witt

Joe Witt

Onyara Inc.

In 2006, Joe Witt created a dataflow framework that grew into a community, and evolved over eight years into what became Apache NiFi (incubating). Following NiFi’s open source release by the NSA in 2014, Joe has become an active committer and member of the Apache NiFi PPMC. Joe remains focused on solving global scale data flow challenges, and is a regular speaker on NiFi and data provenance.