Build & maintain complex distributed systems
October 1–2, 2017: Training
October 2–4, 2017: Tutorials & Conference
New York, NY

Managing server secrets at scale with a vaultless password manager

Ignat Korchagin (Cloudflare)
3:50pm4:30pm Wednesday, October 4, 2017

Who is this presentation for?

  • SREs and data center administrators

Prerequisite knowledge

  • Familiarity with configuration management systems
  • A basic understanding of cryptography

What you'll learn

  • Understand the potential issues with managing secrets on real hardware
  • Explore current basic approaches for storing server secrets on local persistent storage and in configuration management systems
  • Learn an alternative method of managing secrets, which is much easier and supports many edge cases


“There is no cloud. It’s just someone else’s computer.”

Operating a large cluster, a data center, or a distributed network involves handling a lot of secrets. In almost all cases, you have to deal with at least four types of secrets for each piece of hardware: an SSH server key (or shell access key), a key to bootstrap your configuration management system, a disk encryption key, and some per-server credentials to access other services. And most of the time, these keys have to be set up before your configuration management kicks in, making the automation of this process more difficult.

Security-wise, it’s important to control where and when those secrets are generated. Often, keys are generated by startup scripts. However, during initial boot (especially on diskless systems), the system may have a low entropy level in its internal random number generator, resulting in statistically weak generated keys.

And once you have your keys, you need to store them securely. This presents a chicken-and-egg problem. An encrypted disk is a great solution, but guess what? You need a key to access an encrypted disk, and where do you store that? Also, where do you store keys for diskless systems?

Ignat Korchagin outlines a simple approach that combines hardware support and a little cryptography to help operationalize the management of all the secrets in your cloud and unify and simplify secret management for your hardware fleet.

Topics include:

  • Simple and automatic server key/password generation with good entropy
  • Easy key rollover integrated into your configuration management system
  • Provisioning keys on a diskless server
  • Providing an architecture where all disks on the server are encrypted without any external dependencies
Photo of Ignat Korchagin

Ignat Korchagin


Ignat Korchagin is a systems engineer at Cloudflare, where he works mostly on platform and hardware security. Ignat’s interests are cryptography, hacking, and low-level programming. Previously, he was senior security engineer for Samsung Electronics’s Mobile Communications Division, and his solutions can be found in many older Samsung smartphones and tablets. Ignat started his career as a security researcher in the Ukrainian government’s communications services.