Hadoop Operations: Managing Big Data Clusters

Hadoop and MapReduce are enabling consumers of big data to store and process ever increasing volumes of data. Logs, sensor data, user generated content, feeds – the list goes on.

At Cloudera, we’ve witnessed the same process occur at organizations across the world: what starts as a research project on a small Hadoop cluster ends up in the production pipeline, thus creating a new burden for the operations team. A number of tools for managing a reliable and scalable Hadoop installation have been produced by Hadoop’s vibrant user community, including major contributions from Yahoo!, Facebook, and IBM Research. We’ll demonstrate how to put these tools to use in your Hadoop cluster.

During this session, we’ll share war stories from clusters we’ve managed, as well as specific tips and tricks for scaling Hadoop from tens to thousands of nodes. We’ll cover the following in detail:

• Cluster Setup: How do you get started? Where can you find stable binaries? How do you image your systems? How do you keep track of masters, slaves, and other deviants?
• Monitoring and Alerting: How do you know if your cluster is feeling well? What monitoring tools work best with Hadoop? What tricks can help your monitoring systems scale with your cluster?
• Upgrades: Hadoop has a fast release cycle. When should you upgrade? Will it be smooth? How will this impact your users?
• Optimization: Are you 100 servers performing more like 60 or 160? How do you tell? What about those 50 tuning parameters?