Presented By O'Reilly and Cloudera
Make Data Work
Dec 4–5, 2017: Training
Dec 5–7, 2017: Tutorials & Conference

Executive Briefing: The five dysfunctions of a data engineering team

Jesse Anderson (Big Data Institute)
12:05pm12:45pm Wednesday, December 6, 2017
Average rating: ****.
(4.00, 3 ratings)

When a team fails with big data, it’s easy to blame the technology—and that’s exactly what usually happens. It’s far more difficult to look inward to determine the real reasons it failed, which are more often than not an artifact of the team itself.

Early and mid-level failures start well before the first line of code is written. They start with the formation of the team. A data engineering team should be multidisciplinary. It should have the appropriate and required skills before starting the project.

Jesse Anderson outlines five of the most common nontechnology reasons why data engineering teams fail:

  1. All DBAs: The entire data engineering team is made up of DBAs or a similar SQL-only skillset.
  2. Set up for failure: The team was never given the skills and tools to succeed. They’re expected to learn by osmosis and without any guidance.
  3. No one understands schema: No member of the team understands schema and how it leads to success with a mature data pipeline.
  4. No veterans: The team lacks a big data veteran who can keep the team from making terrible mistakes and decisions.
  5. Too ambitious: The team has no previous big data experience and sets out to create the holy grail of data pipelines. They lack an understanding of the level of complexity big data demands.
Photo of Jesse Anderson

Jesse Anderson

Big Data Institute

Jesse Anderson is a data engineer, creative engineer, and managing director of the Big Data Institute. Jesse trains employees on big data—including cutting-edge technology like Apache Kafka, Apache Hadoop, and Apache Spark. He’s taught thousands of students at companies ranging from startups to Fortune 100 companies the skills to become data engineers. He’s widely regarded as an expert in the field and recognized for his novel teaching practices. Jesse is published by O’Reilly and Pragmatic Programmers and has been covered in such prestigious media outlets as the Wall Street Journal, CNN, BBC, NPR, Engadget, and Wired. You can learn more about Jesse at