Presented By O'Reilly and Cloudera
Make Data Work
September 26–27, 2016: Training
September 27–29, 2016: Tutorials & Conference
New York, NY

Securing Apache Kafka

Jun Rao (Confluent)
2:05pm–2:45pm Thursday, 09/29/2016
Security
Location: River Pavilion Level: Intermediate
Average rating: *****
(5.00, 1 rating)

Prerequisite knowledge

  • A high-level understanding of Kafka and its use cases
  • What you'll learn

  • Learn how to secure Apache Kafka
  • Description

    Kafka, developed at LinkedIn in 2010, was originally an open system to encourage adoption; developers could easily create new data streams, add data to the pipeline, and read data as it was created. It succeeded brilliantly at encouraging developers to build new data applications, improved the reliability of systems and applications, and helped LinkedIn scale its logging and data infrastructure.

    Unfortunately, as Kafka usage grew at LinkedIn (and at other sites), the problems with a totally open system became apparent. Developers might inadvertently cause production problems when creating new Kafka streams, engineers might change the configuration of critical systems, and employees might get access to sensitive data. As Kafka has been adopted by larger enterprises with more complex security requirements, the Kafka community has had to rethink its architecture.

    With Apache Kakfa 0.9, the community has introduced a number of features to make data streams secure. Jun Rao explains the motivation for making these changes and the threats that Kafka Security mitigates, discusses the design of Kafka security, and demonstrates how to secure a Kafka cluster. Jun also covers common pitfalls in securing Kafka and talks about ongoing security work.

    Topics include:

    • New security features in Kafka 0.9
    • The common usage pattern of the security feature
    • The access control model for Kafka
    • Configuring authentication, access control, and encryption
    • Using a secure Kafka cluster with other secure (and insecure) systems
    • Testing, monitoring, and tuning a secure Kafka cluster
    • Future work in Kafka security
    Photo of Jun Rao

    Jun Rao

    Confluent

    Jun Rao is the cofounder of Confluent, a company that provides a streaming data platform on top of Apache Kafka. Previously, Jun was a senior staff engineer at LinkedIn, where he led the development of Kafka, and a researcher at IBM’s Almaden research data center, where he conducted research on database and distributed systems. Jun is the PMC chair of Apache Kafka and a committer of Apache Cassandra.