Build & maintain complex distributed systems
October 1–2, 2017: Training
October 2–4, 2017: Tutorials & Conference
New York, NY

The Phone Book is On Fire: Lessons from the Dyn DNS DDoS

Lex Neva (Heroku)
3:50pm4:30pm Wednesday, October 4, 2017
Resilience Engineering
Location: Gramercy Level: Intermediate

Who is this presentation for?

System Admins, Systems Engineers, SREs, Networking Engineers, and Technical Managers

Prerequisite knowledge

DNS knowledge including recursive queries would be useful, though I will briefly go over the crucial parts for those unfamiliar.

What you'll learn

How to set up redundant nameservers correctly. All sorts of interesting and cool details about how DNS works. How to find the correct answer to tricky technical questions when no one else knows.


I thought I knew a lot about DNS, but the Dyn DDoS last October showed me that I had much more to learn. I spent the following month deep-diving to figure out how to implement resilient DNS properly: reading RFCs, Googling, asking questions of pros, and performing real-world experiments when no one knew the answers. In this talk, I’ll share what I learned and how I learned it.

Opinions on how to react to a DDoS against your nameservers vastly differ. Some companies lowered the TTL for their in-zone NS records to make it easier to add a new nameserver in a crisis. I’ll show you my empirical proof that this doesn’t work.

Adding a redundant DNS server in advance is a good solution, but it can be much harder than it sounds. What can you do if your DNS records are heavily dynamic and your provider doesn’t offer AXFR support? I’ll go through the options.

Even if you’re a seasoned DNS pro, I aim to surprise you with my discoveries.

Photo of Lex Neva

Lex Neva


Lex has 7 years of experience keeping large services running, including Linden Lab’s Second Life,, and Heroku. While originally trained in computer science, he found that he most enjoyed applying his software engineering skills to operations. A veteran of many large incidents, he has strong opinions on incident response, retrospectives, on-call sustainability, and good development and release processes.

Leave a Comment or Question

Help us make this conference the best it can be for you. Have questions you'd like this speaker to address? Suggestions for issues that deserve extra attention? Feedback that you'd like to share with the speaker and other attendees?

Join the conversation here (requires login)


Picture of Lex Neva
Lex Neva | SRE
06/28/2017 6:53am EDT

I intend to allow recording of this session.

Picture of André Morrow
06/09/2017 8:01am EDT

Sessions are included in the conference’s video compilation if the speaker allows. We will not be able to confirm that this talk will be included until just after Velocity NY.

zeroBS GmbH Germaine Adelt |
06/08/2017 11:05pm EDT

is there / willl there be a recording of that event?