Common Sense Performance Indicators in the Cloud

Every month at SEOmoz we serve nearly a hundred million requests from a dataset of web pages and links that grows by over 50TB every six weeks. In this session I’ll explore the performance indicators we watch to monitor our Amazon Web Services-based web and link data services API.

Latency and throughput are mission critical to us. We’ve had a significant challenge in understanding the performance of our AWS components (EC2, S3) and our traditional components (e.g., memcache, lighttpd). When we began our project over two years ago, many of these components were brand new. In that time we’ve gained expertise about measuring and testing the performance of a cloud-based application, using low-cost or free software tools, and common sense.

• performance indicators we look at (and everyone else should too):
  • load average
  • CPU usage by process
  • disk activity
  • memory usage by process
  • network usage
  • throughput
  • response time
• how those indicators apply to cloud infrastructure
• what those indicators tell us about end-to-end performance
• performance characteristics we see from S3 and EC2
  • what response times can you expect from S3 inside EC2 and outside?
  • what throughput will S3 deliver?
  • disk and network throughput on EC2
• the simple, free software tools we use to measure these performance indicators

In this session I’ll show our live performance dashboard illustrating several dimensions of any production system’s performance. I’ll briefly discuss how these dimensions interact in general and in an AWS-based architecture. I’ll point out what these indicators say about capacity planning.

Throughout this discussion of performance indicators, I’ll present out-of-the-box, free software tools that you can use to collect, analyze, and present this kind of data. And I’ll show how they work together to provide a manageable monitoring solution for a cloud-based application.