Building a production-ready data lake in the cloud

Hadoop’s ability to handle large amounts of varied data has been a critical enabler of the big data revolution. Many organizations’ ambitions to become more data-driven, however, are held back by a shortage of resources as well as the time and expense needed to purchase and set up Hadoop’s hardware and software underpinnings. The cloud offers a natural alternative that overcomes these barriers and also promises superior economics with elastic scaling.

In this session, attendees will learn the key steps and best practices for implementing Hadoop in the cloud, to support a production-ready data lake use case. The process will be broken down into clear steps, including:

• Planning: understanding which workloads are a good match for Hadoop, selecting a distribution, performance benchmarking, scoping memory, and compute resource requirements
• Provisioning: setting up public cloud resources and installing Hadoop
• Moving data: connecting to data sources, encrypting data in movement, optimizing WAN transfer
• Connecting and using analytical tools: ensuring a seamless transition for end-users for all workloads, monitoring and guaranteeing workload SLAs are met
• Managing infrastructure: monitoring health, ensuring availability, disaster recovery
• Ensuring security: data encryption, managing users/roles/groups, key management, VPN

The session will conclude by sharing additional lessons learned, best practices and business outcomes.