October 30–31, 2016: Training
October 31–November 2, 2016: Tutorials & Conference
New York, NY

Saving time: How a few committed people helped hold up the Internet. . .again

Susan Sons (Center for Applied Cybersecurity Research, Indiana University)
2:10pm–2:50pm Wednesday, 11/02/2016
The human element
Location: Rendezvous Trianon Level: Beginner
Average rating: *****
(5.00, 3 ratings)

Prerequisite knowledge

  • Familiarity with general technical concepts like what an Internet protocol is and why long delays in patching vulnerabilities are bad

What you'll learn

  • Understand the dangers lurking in our neglected software infrastructure but also that these are concrete challenges that can be overcome, just as they were for NTP


Susan Sons tells the story of the ongoing intervention to save the troubled but ubiquitous Network Time Protocol’s reference implementation, explaining how social, technical, and resourcing challenges came together to threaten a core piece of Internet infrastructure and how these challenges were overcome.

In February, 2015, NTP—the reference implementation of the Network Time Protocol, which tells nearly every device in the world what time it is—was deeply troubled. Vulnerabilities were going unpatched for months or years, and everyone from script kiddies to APTs were having a field day with this essential service. The code base was not yet C99 compliant—that is, by 2015, it had not caught up to the coding standard of 1999—and documentation was years out of date. The build system was brittle, and the code, while open source, was locked up in a proprietary repository that drive-by contributors could not access. The sole maintainer was a solitary, aging coder operating on a rotting infrastructure.

The security implications of this mess couldn’t be more dire: accurate time is crucial to the stock exchanges and banking and finance generally, as well as cryptography, GPS navigation, scientific experiments around the world, and countless other applications. However, this crucial service is best known in some circles as the ideal jumping-off point for amplification of DDoS attacks on other systems and a great entry point for taking down a system running the service itself.

Plenty of people talk about how to secure shiny, new software at a company with a clear goal and leadership structure. Unfortunately, the money you pay your ISP goes mostly to hardware and their own operations or profits. The software that supports the core protocols of the Internet is, for the most part, maintained ad hoc by open source volunteers. When something goes horribly wrong, it generally isn’t clear who is in charge of fixing it or where the resources to do so should come from. Susan discusses how her team took over this aging, yet critical, behemoth and:

  • Organized stopgap resources for a software project that no one owned, no one makes money from, but everyone depends on;
  • Built a functional team for securing and improving a delicate piece of software that includes esoteric algorithms almost no one had deep knowledge of in 2015;
  • Navigated social and political barriers to getting the job done;
  • Overcame more than four decades of technical debt;
  • Staged a refactor that would make many developers hide under their beds;
  • Built the build, test, and support infrastructure necessary to ensure stability for a user base we have limited information on but which includes critical-but-esoteric applications such as real-time operating systems used in scientific research facilities for high-precision timing of physics, astronomy, geology, and other experiments; and
  • Managed to still like one another a year later.
Photo of Susan Sons

Susan Sons

Center for Applied Cybersecurity Research, Indiana University

Susan Sons is a hacker, author, and miscreant based in Bloomington, Indiana. In her working life, she aids NSF- and DHS-funded projects in establishing and maintaining sound information security practices. In her off hours, Susan codes, writes, and leads ICEI, the Internet Civil Engineering Institute, a nonprofit that supports the open source software infrastructure upon which the internet and computing in general depend. When not rescuing software projects, Susan lifts weights, practices martial arts, and gives her time as a volunteer search and rescue worker.