Incident Response 101: From First Alert to Clean Post-Mortem

Incident Response 101: From First Alert to Clean Post-Mortem

Every incident, no matter how chaotic it feels in the moment, follows the same five-stage lifecycle. Teams that name the stages and track them get faster, calmer, and better at preventing repeats. Teams that do not just keep firefighting the same fires.

Stage 1: Detection

The clock starts the moment something breaks. The goal of detection is to make that gap—break to alert—as small as possible without crying wolf.

  • Use multi-region checks so a regional blip does not page you.

  • Require N consecutive failures (PingHarbor defaults to 2) before alerting.

  • Track Mean Time To Detect (MTTD) as a metric. If it is creeping up, your monitoring is drifting.

Stage 2: Triage

Page the right person, not everyone. Triage answers three questions: Is it real? How bad is it? Who owns it?

A status page update at this stage costs you nothing and buys massive goodwill. "We are aware of an issue affecting logins and are investigating" beats silence every time.

Stage 3: Mitigation

Mitigation is not the same as fix. Roll back the deploy, fail traffic over to the secondary region, disable the misbehaving feature flag. Stop the bleeding first, root-cause later.

Common mitigations worth pre-building runbooks for:

  • Deploy rollback (one command, documented).

  • Feature flag kill switches.

  • Read-only mode for the database.

  • CDN cache purge or origin failover.

Stage 4: Resolution & Communication

Verify the fix from the same external monitors that caught the failure. Update the status page, notify customers, and only then mark the incident closed. Closing too early is the fastest way to lose trust.

Stage 5: Post-Mortem

Blameless post-mortems are the only kind worth writing. The output is not "who screwed up" but "what about our system let a single mistake escalate this far". Track action items in your normal work tracker so they do not vanish.

A useful post-mortem template:

  • Timeline (UTC), with detection, triage, mitigation, resolution timestamps.

  • User impact (numbers, not adjectives).

  • Root cause and contributing factors.

  • What went well, what went badly.

  • Action items with owners and due dates.

How PingHarbor supports the lifecycle

PingHarbor records every check, every state transition, and every alert delivery. The Incidents page gives you a chronological timeline you can paste straight into a post-mortem doc, with detection time, recovery time, and per-region first-failure data already filled in.

You cannot improve what you do not measure. MTTD, MTTR, and incident count per service are the three numbers worth tracking.