There is a dirty little secret in all data work that everyone wants to ignore. It’s the 80 percent that has been called out in recent articles, under titles as diverse as “janitorial work” and “data engineering.” It’s real, it’s costly, and it hurts. It hurts business, and it hurts the people doing it. But some of us love it. It presents real challenges in data and makes everything work.
Data is all sales and marketing. The reality of data work is pain. Most data projects fail and are horrible experiences to work on. Phil Harvey explains that data is just too hard—the world needs to talk about real challenges so that we can start tackling them to deliver data projects that work. This is DataOps; there will be tears before bedtime.
When you break through the marketing and glamour of emerging data technologies, you fall into a pile of lies and difficult truths. You very quickly find battle lines drawn in a war of paradigms and fashion. The reality is that getting data is hard; talking to people about data is hard; robustness is hard; and the DataOps part of any “success” (few and far between as these are) is hardly every recognized. But there are things we can all do. Phil explores tools for understanding and communicating about the beautiful DataOps mess.
Phil Harvey is a Senior CSA for data and AI at Microsoft. Passionate about data and people, he believes empathy is the key data skill. He’s also a big, beardy geek.
©2016, O’Reilly UK Ltd • (800) 889-8969 or (707) 827-7019 • Monday-Friday 7:30am-5pm PT • All trademarks and registered trademarks appearing on oreilly.com are the property of their respective owners. • firstname.lastname@example.org
Apache Hadoop, Hadoop, Apache Spark, Spark, and Apache are either registered trademarks or trademarks of the Apache Software Foundation in the United States and/or other countries, and are used with permission. The Apache Software Foundation has no affiliation with and does not endorse, or review the materials provided at this event, which is managed by O'Reilly Media and/or Cloudera.