Deploying large payloads at scale
In 2010 we started experimenting with hiphop as a means to deliver our site to our users. One of the things that became pretty clear from the start was that we needed a different way of deploying our hiphop compiled binary to our servers due to the size of the binary.
We created a bittorrent based system using a custom tracker that exploits knowledge about our datacenter setup and the location of the servers. This way we are able to deploy 500-750MB of payload to hundreds of servers in well under 3 minutes. The chosen method is generic enough to be useful to others.
The main objective of the custom tracker is to limit the use of bandwidth by the bittorrent system, and prevent flooding of rack uplinks by bittorrent clients.
The talk will discuss general architecture, implementation issues, setup, lessons learned and differences with murder by twitter…
The tracker may be released as open source, but no definitive commitment has been made to open source the tracker.
Large scale system administrator / infrastructure developer working at Hyves.nl the primier dutch social network.
For information on exhibition and sponsorship opportunities at the conference, contact Gloria Lombardo at email@example.com
View a list of Velocity Europe contacts