Deploying Large Payloads At Scale

Deploying large payloads at scale

In 2010 we started experimenting with hiphop as a means to deliver our site to our users. One of the things that became pretty clear from the start was that we needed a different way of deploying our hiphop compiled binary to our servers due to the size of the binary.

We created a bittorrent based system using a custom tracker that exploits knowledge about our datacenter setup and the location of the servers. This way we are able to deploy 500-750MB of payload to hundreds of servers in well under 3 minutes. The chosen method is generic enough to be useful to others.

The main objective of the custom tracker is to limit the use of bandwidth by the bittorrent system, and prevent flooding of rack uplinks by bittorrent clients.

The talk will discuss general architecture, implementation issues, setup, lessons learned and differences with murder by twitter…

The tracker may be released as open source, but no definitive commitment has been made to open source the tracker.