Skip to main content

Deploying on the Edge

Rob Peters (EdgeCast Networks)
Operations
Beekman
Average rating: ****.
(4.00, 2 ratings)
Slides:   1-PDF    external link

We operate a global edge network that delivers many types of modern web traffic, including dynamic applications, websites, mobile apps, live and on-demand streams, and large-file downloads. We strive to maintain reliability, performance, and functionality as we develop and deploy the http server software that handles this traffic. In this talk we’ll cover the evolution of our deployment best practices as we have learned from the community and from our own experiences, including the following:

  • Go fast — the deployment cycle should be as short as possible in order to minimize batch size (so as to constrain the scope of the unexpected, because there is always something unexpected) and reduce risk and mean-time-to-recovery.
  • But not too fast — the deployment cycle should be long enough to be very confident that the latest release has no new issues before moving on to the next release. The time to “long enough” may vary significantly depending on the layer of the software stack.
  • Monitor everything — you can’t fix a problem until you can visualize it.
  • But really monitor just a few things — find the smallest set of vital signs that can reliably indicate “is everything running smoothly?”
  • Be able to roll forward/backward almost instantly, keeping in mind that the links between command/control and edge systems may be slow and/or lossy.
  • Be “lazy" — a programmer’s “lazy” can mean spending days building something that turns a 10-second task into a 2-second task. This is exactly the right approach when it comes to deployment, where it means spending extra time to ensure that new code and configurations are built in such a way that they can be deployed painlessly. This often amounts to strict compatibility between the default behaviors of adjacent versions, configurability to easily turn new functionality on/off, and comprehensive hooks for testing and monitoring.
  • Minimize risk in the deployment process itself — while each new software update may have different and unique changes, the procedure for deploying the update can be the same every time.
  • Don’t be too portable/configurable — if the application should never run in production without package XYZ, then it shouldn’t pretend to be portable to an environment without XYZ.
Photo of Rob Peters

Rob Peters

EdgeCast Networks

Rob Peters is Chief Architect at Verizon EdgeCast, where he was one of the early members of the engineering group. Peters currently leads teams that work with EdgeCast’s core http cache/proxy engine and its surrounding infrastructure ecosystem of software and hardware. He enjoys thinking about deployment methodologies and looking at graphs and grids full of colorful numbers, and is often mistaken for a subversion and perl fanboy.

Prior to joining EdgeCast in 2008, Rob completed a Ph.D. in Computation & Neural Systems from the California Institute of Technology and spent several years in post-doctoral work at the University of Southern California building biologically-inspired computer vision software. He will be happy to show you a live demo of these algorithms that still manage to run on his laptop. You can find Rob online at https://twitter.com/rjpcal.