Winning the Metrics Battle (finally)

The Guardian didn’t set out to have a full time metrics and monitoring role, but when we interviewed someone who fitted that role we realised that was a missing piece in our strategy. The best thing we’ve done since is listen to him and become obsessed with metrics and monitoring.

• gathering metrics on everything
• gathering metrics frequently
• alerting on the same metrics that we graph
• that everything included copious quantities of developer provided application metrics
• to provide visibility to everyone

The problem was how to make that a reality.

In this talk we are going to tell the story of our mistakes and successes in trying to make this a reality. We’ll talk about the products we evaluated, those we tried and failed to use and those we have currently settled on (ganglia, graphite and some custom software). We’ll talk about the friction and roadblocks that operations and developers have come across and steps we’ve taken to reduce and eliminate them (including self service dashboards and a standard (automated) way of collecting application metrics).

The result of the work we’ve done is that we have massively improved our visibility across both our data-centres and in the cloud; we can troubleshoot and fix problems more quickly; developers are able to provide application metrics and they have an interest in watching them. All of which has brought developers and operations closer together.