Presented By O'Reilly and Cloudera
Make Data Work
22–23 May 2017: Training
23–25 May 2017: Tutorials & Conference
London, UK

The five dysfunctions of a data engineering team

Jesse Anderson (Big Data Institute)
16:3517:15 Wednesday, 24 May 2017
Data-driven business management
Location: Capital Suite 2/3
Level: Non-technical
Average rating: *****
(5.00, 6 ratings)

Who is this presentation for?

  • Managers, VPs, directors, and team leads

Prerequisite knowledge

  • Management experience

What you'll learn

  • Explore the five most common non-technology reasons why data engineering teams fail


When a team fails with big data, it’s easy to blame the technology—and that’s exactly what usually happens. It’s far more difficult to look inward to determine the real reasons it failed, which are more often than not an artifact of the team itself.

Early and mid-level failures start well before the first line of code is written. They start with the formation of the team. A data engineering team should be multidisciplinary. It should have the appropriate and required skills before starting the project.

Jesse Anderson outlines five of the most common non-technology reasons why data engineering teams fail:

  1. All DBAs: The entire data engineering team is made up of DBAs or a similar SQL-only skillset.
  2. Set up for failure: The team was never given the skills and tools to succeed. They’re expected to learn by osmosis and without any guidance.
  3. No one understands schema: No member of the team understands schema and how it leads to success with a mature data pipeline.
  4. No veterans: The team lacks a big data veteran who can keep the team from making terrible mistakes and decisions.
  5. Too ambitious: The team has no previous big data experience and sets out to create the holy grail of data pipelines. They lack an understanding of the level of complexity big data demands.
Photo of Jesse Anderson

Jesse Anderson

Big Data Institute

Jesse Anderson is a data engineer, creative engineer, and managing director of the Big Data Institute. Jesse trains employees on big data—including cutting-edge technology like Apache Kafka, Apache Hadoop, and Apache Spark. He has taught thousands of students at companies ranging from startups to Fortune 100 companies the skills to become data engineers. He is widely regarded as an expert in the field and recognized for his novel teaching practices. Jesse is published by O’Reilly and Pragmatic Programmers and has been covered in such prestigious media outlets as the Wall Street Journal, CNN, BBC, NPR, Engadget, and Wired. You can learn more about Jesse at

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)


Picture of Jesse Anderson
19/07/2017 17:29 BST

The slides are uploaded here and you can read my Data Engineering Teams to go even deeper.

18/07/2017 13:53 BST

Hi Jesse,

Interesting session! Can you please post the slides used.