I’m interested in knowing how people perform analysis when an incident happens that requires intervention on a given system, such as VM/Server reboots, process restarts etc.
At the moment, we have an issue whereby if a system becomes busy and load spikes or a system locks up, crashes etc we don’t really have a way of precisely knowing what happened. We run Munin on most systems for graph data, but, to use a recent example, we had a load spike on a web server (running Apache) at the weekend. Munin isn’t reporting anything unusual in the graphs apart from a blank space but something definitely caused it to overload.
When systems lock up or crash, we don’t have a precise way of knowing why, usually, logs don’t print anything out of what was happening and then we’re left in limbo because investigations are inconclusive. Does anyone have any precise way of knowing what happened when things like this go on? What processes, possibly vhosts, etc are causing problems?
Would be great to get some ideas of how others manage and deal with this.