Deploying Large Payloads At Scale

Operations and Culture
Location: Hall 1 B/C Level:
Average rating: ****.
(4.16, 38 ratings)

Deploying large payloads at scale

In 2010 we started experimenting with hiphop as a means to deliver our site to our users. One of the things that became pretty clear from the start was that we needed a different way of deploying our hiphop compiled binary to our servers due to the size of the binary.

We created a bittorrent based system using a custom tracker that exploits knowledge about our datacenter setup and the location of the servers. This way we are able to deploy 500-750MB of payload to hundreds of servers in well under 3 minutes. The chosen method is generic enough to be useful to others.

The main objective of the custom tracker is to limit the use of bandwidth by the bittorrent system, and prevent flooding of rack uplinks by bittorrent clients.

The talk will discuss general architecture, implementation issues, setup, lessons learned and differences with murder by twitter…

The tracker may be released as open source, but no definitive commitment has been made to open source the tracker.

Photo of Ramon van Alteren

Ramon van Alteren

Hyves

Large scale system administrator / infrastructure developer working at Hyves.nl the primier dutch social network.

Bulletin

Bulletin

  • ip-label
  • Compuware Corporation
  • dynaTrace
  • Keynote Systems
  • New Relic
  • Citrix Systems
  • Google
  • Apica
  • AppDynamics
  • CDNetworks
  • Cotendo
  • Dyn Inc.
  • ImmobilienScout24
  • Spil Games
  • Dyn Inc.

For information on exhibition and sponsorship opportunities at the conference, contact Gloria Lombardo at glombardo@oreilly.com

View a list of Velocity Europe contacts