Build Systems that Drive Business
Sep 30–Oct 1, 2018: Training
Oct 1–3, 2018: Tutorials & Conference
New York, NY

Rebuilding the airplane in flight. . .safely

3:50pm–4:30pm Wednesday, October 3, 2018
Systems Engineering and Architecture
Location: Murray Hill Level: Intermediate
Secondary topics:  Systems Architecture & Infrastructure
Average rating: ***..
(3.50, 2 ratings)

Prerequisite knowledge

  • General knowledge of internet infrastructure

What you'll learn

  • Learn how NS1 completed and rolled out a DNS server rewrite across its globally distributed network with no downtime—and how you can do the same


In 2017, NS1 embarked on a ground-up rewrite of its advanced DNS server software. This required significant research, planning, and execution time—over a year in total. Shannon Weyrick and James Royalty detail the various challenges NS1 encountered and key points the company had to consider to successfully engineer and deploy across its global managed DNS network with no negative impact or downtime to its customers.

Shannon and James share background on the decision to move forward with a rewrite (including DNSSEC and scaling requirements), research of appropriate technologies to balance performance, functionality, and engineering velocity, phased milestones with a hybrid release approach to better facilitate product iteration and to gain operational experience, a system for tee-testing traffic for verification of correctness during deploy, and the utilization of an anycast network for fault isolation during roll out. Along the way, they discuss the many minor successes, failures, setbacks, and delays that you may face day to day and offer tips and advice to support you in your own quests to rebuild your airplanes in flight. You’ll leave with an appreciation for the challenges involved in planning and executing a large-scale rewrite and deployment of critical path software across a widely distributed network.

Photo of Shannon Weyrick

Shannon Weyrick


Shannon Weyrick is vice president of architecture at NS1. A 20-year veteran of internet infrastructure, Shannon is an accomplished technical architect, developer, and leader whose experience encompasses both development and operations of globally distributed platforms. Previously, Shannon worked at INAP and F5. A regular open source contributor, he has led and worked on a wide range of infrastructure projects from high-performance servers to novel programming languages and runtimes, and he enjoys writing and speaking at industry conferences.

James Royalty


James Royalty is a lead engineer at NS1.