Data science is all the rage, but what happens when there isn’t a data engineering organization? USAA had a large and established data science organization, but it wasn’t able to fulfill all of the promises of big data. The data scientists were spending the majority of their time dealing with data quality issues or trying to create their own data pipelines—not the core competency of the data scientist. And when the complexity of data pipelines exceeded data scientists’ abilities, they had to stop.
Jesse Anderson and Thomas Goolsby explain what happened at USAA without data engineering, how they fixed it, and the results since. Creating data engineering culture started at the top of USAA’s management and included showing management how data engineering is different from data science and software engineering. Jesse and Thomas then worked with human resources to create the actual data engineer title and started creating the teams and hiring data engineers.
Since companies around the world quickly encounter a shortage of data engineers, they decided to look internally for software engineers with right makings for a data engineer. This involved giving the teams the right training, tools, and mentoring to become data engineers.
Jesse Anderson is a data engineer, creative engineer, and managing director of the Big Data Institute. Jesse trains employees on big data—including cutting-edge technology like Apache Kafka, Apache Hadoop, and Apache Spark. He’s taught thousands of students at companies ranging from startups to Fortune 100 companies the skills to become data engineers. He’s widely regarded as an expert in the field and recognized for his novel teaching practices. Jesse is published by O’Reilly and Pragmatic Programmers and has been covered in such prestigious media outlets as the Wall Street Journal, CNN, BBC, NPR, Engadget, and Wired. You can learn more about Jesse at Jesse-Anderson.com.
Thomas Goolsby is data and analytics director at USAA, where he focuses on member interaction data and research with universities to help provide new insights and build pipelines of talent.
©2019, O'Reilly Media, Inc. • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • email@example.com