Apps Behaving Badly

What to do when something goes horribly wrong in production? Well of course we hope that it never happens, but there are occasions when mistakes occur or soething unexpected comes up and your servers start chewing memory and not completing connection, everything is going to hell. At the guardian our CMS has a number of architecture decisions made that allow us to recover from almost all forms of failure, and we’ll detail how some of these work, and why we made them work they way we chose to. Once you’ve managed to patch the system into such a state that it can recover, the next vital task is to reason out why it happened and how we can fix it. There is a method that we use when addressing serious site failures, and a number of tools and approaches that you can use after the fact to try to reinterpret what happened and trace back in time.