Creating a data engineering culture at USAA

Description

Data science is all the rage, but what happens when there isn’t a data engineering organization? USAA had a large and established data science organization, but it wasn’t able to fulfill all of the promises of big data. The data scientists were spending the majority of their time dealing with data quality issues or trying to create their own data pipelines—not the core competency of the data scientist. And when the complexity of data pipelines exceeded data scientists’ abilities, they had to stop.

Jesse Anderson and Thomas Goolsby explain what happened at USAA without data engineering, how they fixed it, and the results since. Creating data engineering culture started at the top of USAA’s management and included showing management how data engineering is different from data science and software engineering. Jesse and Thomas then worked with human resources to create the actual data engineer title and started creating the teams and hiring data engineers.

Since companies around the world quickly encounter a shortage of data engineers, they decided to look internally for software engineers with right makings for a data engineer. This involved giving the teams the right training, tools, and mentoring to become data engineers.

Topics include:

• What it looks like when you don’t have a data engineering culture
• Why it’s crucial to have a data engineering culture
• How to educate management about the need for data engineering
• How to find internal resources and make them data engineers