October 30–31, 2016: Training
October 31–November 2, 2016: Tutorials & Conference
New York, NY

Securing application deployments in a multitenant CI/CD environment

3:50pm–4:30pm Wednesday, 11/02/2016
Tools and processes
Location: Grand Ballroom West Level: Intermediate
Average rating: ***..
(3.67, 3 ratings)

Prerequisite knowledge

  • Familiarity or experience with Jenkins, GitHub, and related hosted build and deploy tools

What you'll learn

  • Understand why security is so critical for a multitenant CI/CD platform
  • Learn why existing security controls are not adequate in the event of an attack
  • Explore real-world experience securing application deployments in CI/CD environment at Yahoo

Description

Continuous integration and continuous delivery (CI/CD) automate the way we build, distribute, and deploy applications to production environments. For instance, GitHub, Jenkins, Docker, and Mesos-Marathon are all integrated seamlessly to achieve continuous delivery of applications to production. In order to function, these systems often have permissions to access a wide range of systems and services outside of their environment, making them attractive targets for attackers. For example, your build job running on a Jenkins slave will have the permissions to push newly built packages to package repositories. Other examples include accessing a protected source repository like GitHub, testing web applications using SauceLabs, deploying an application using Mesos-Marathon, Chef, or SSH to cloud environments, and provisioning TLS private keys and secrets after deploying your application. Similarly, your workflow job may access a protected web service or a database server as part of functional test phase.

These requirements are often addressed by granting coarse permissions to slave systems. One common practice is to share a headless credential by all workflow jobs to access protected services. In another, hosted systems (e.g., Travis CI and Heroku) collect and store user OAuth tokens. These tokens are used behalf of the user to build, distribute and deploy applications. However, these approaches have a few problems. The current model treats the CI/CD system components as trust anchors and stores nonephemeral credentials and user OAuth tokens locally in those systems. By compromising one of these systems, the attacker gains access to all credentials. These credentials provide attacker, the ability to touch and reach all production systems. At the same time, the lack of a verifiable chain of trust from commit to deploy affects the integrity of deployed applications.

Binu Ramakrishnan highlights the dangers associated with a centralized multitenant CI/CD platform and shares his experience protecting a large-scale shared CI/CD platform at Yahoo. Binu presents a novel approach, auth events, to delegate user privileges to CI/CD workflow jobs. The auth events form the foundation of the chain of trust from the commit-to-deploy path and extend it to the deployed application.

Photo of Binu Ramakrishnan

Binu Ramakrishnan

Yahoo

Binu Ramakrishnan is a principal security engineer at Yahoo with over a decade of experience in building Internet-scale systems, anti-abuse systems, and application security. He currently leads security engagements in Yahoo mail, working closely with product engineers and leaders to help define and implement strategic security programs. Binu is an active participant in the industry-wide initiative to secure mail delivery infrastructure and contributed to the recent SMTP STS efforts. He is also the author of a few open source tools.