Meta your data; drain the big data swamp

Data lakes have enabled organizations to consolidate data across functional boundaries to drive insights from analytics. However, a lack of focus on an enterprise-wide strategy and architecture have restricted the otherwise capable data lake to solve multiple enterprise-wide problems that provide horizontal or vertical views of the data that can then create actionable strategies for the firm.

When BP wanted to implement an operational data lake, it was driven by the desire to reduce operational risk, improve predictability and operational awareness, prepare for and invest in the future, and gain a competitive advantage. To achieve this, BP developed a comprehensive strategy and approach for a sustainable and operational data lake that included data and information governance in a bill of data, a lightweight, portable, and scalable process design, metadata management technology for big data ecosystems, and an Agile approach to delivering business results.

Madhav Madaboosi and Meenakshisundaram Thandavarayan offer an overview of BP’s self-service operational data lake, which improved operational efficiency, boosting productivity through fully identifiable data and reducing risk of a data swamp. They cover the path and big data technologies that BP chose, lessons learned, and pitfalls encountered along the way.