Securing your cloud data lake with a "defense in depth" approach

Description

Cloud providers have made it extremely easy and inexpensive to store data in the cloud with object storage services such as S3 and ADLS. These services offer high availability, disaster recovery, and infinite scalability at an extremely low price point of about $20 per month per TB. As a result, companies often store many TBs or PBs of data on these systems.

One of the key challenges faced by most companies is how to secure this data and make sure that only the users and applications that need the data have access to it. Tomer Shiran and Jacques Nadeau explore a defense-in-depth approach to cloud data lake security which includes: authentication (leveraging authentication protocols such as OAuth to authenticate users and applications and taking advantage of various capabilities that enable applications to authenticate without any user intervention), access control (applying role-based access control, ACLs, and other restrictions (e.g., time) to control which users and applications can read and write data), encryption (leveraging encryption at rest (e.g., Azure storage service encryption [SSE]) to encrypt the data stored in the lake, and encryption in motion (e.g., transport layer security [TLS]) to encrypt data being sent between an application and the data lake store, as well as utilizing the cloud’s key management service to store and rotate keys), auditing (monitoring access to the data in order to catch violators before they can cause much harm), and network security (limiting the virtual networks and IP addresses from which the data lake store can be accessed).

In addition to demonstrating how to set up these critical security layers on AWS and Azure, Tomer and Jacques show how data can be secured at a more granular level with column- and row-based access control as well as masking of personally identifiable information (PII) data. By the time you leave, you’ll have a deep understanding of the range of data lake security capabilities available in the main public clouds, as well as best practices for integrating these capabilities with the processing and analysis layers of the cloud data lake.