Architecting an open source enterprise data lake

Description

There are so many business intelligence tools in the Hadoop ecosystem, and no common measure to identify the efficiency of each. Oftentimes, BI tools provide meaningful access to data for certain use cases while failing to fulfill other architectural needs, leaving you stuck with high subscription fees (usually paid upfront or on a yearly basis). So where do you begin to build or modify your enterprise data lake strategy?

Sagar Kewalramani shares real-world BI problems and how they were resolved with Hadoop tools and demonstrates how to build an effective data lake strategy with open source tools and components, focusing on business requirements, deriving business value, and creating a productive and Agile environment for data discovery and business analytics to achieve an organization’s goals.

Sagar offers an overview of an open source enterprise data lake that was designed for scalability, processing resource needs, storage requirements, and cost efficiency. This ecosystem is built using Kafka for reliable message transport, Spark for real-time data processing, Hive for analytical queries, HBase for tactical queries, and Python for machine learning requirements. Along the way, Sagar also illustrates how batch and real-time data processing and analysis can be implemented with open source tools.