Incident response runbooks that get used
A runbook that opens with system architecture is documentation for onboarding, not operations for 3 AM. Incident runbooks get used when they start with symptoms, branch on observable signals, and end with verification steps short enough to execute under stress.
The pager fired at 2:47 AM. On-call opened the wiki. The runbook began with a microservices diagram from 2022. Three services had been renamed. One was deleted. The diagram had no link to logs, no "if error rate on X," no rollback command. Twenty minutes passed reading prose about system philosophy. Incident response runbooks that get used share one property: they start where the incident starts — a symptom, an alert, a customer report — and branch on checks that take seconds, not chapters that take attention.
Runbooks are not documentation. Documentation teaches the system. Runbooks guide action when the system is failing and cognition is impaired. The difference is structure: decision trees versus essays, commands versus concepts, verification steps versus historical context.
Runbooks fail when written for calm readers
| Runbook anti-pattern | Why it fails at 3 AM |
|---|---|
| Architecture overview first | Reader needs action, not context |
| Prose paragraphs | Scanning fails under stress |
| Stale service names | Trust collapses on first wrong reference |
| No owner or last-verified date | Unclear if safe to follow |
| Links to links | Navigation breaks on mobile |
| "Contact the team" | No phone, no escalation path |
Good runbooks assume: reader may be new, tired, alone, and unfamiliar with recent deploys. Every step must be executable without institutional memory.
A runbook is a program for humans. If it does not branch on conditions, it is a script that only runs on one machine — yours, six months ago.
Anatomy of a runbook that gets opened
Header (five lines max)
- Service / symptom name
- Severity context (when to use this runbook)
- Last verified date + owner
- Related alerts (PagerDuty, Grafana names)
- Escalation contact
Symptom entry points
Multiple entry paths — same root cause may present differently:
Entry: Checkout error rate > 2%
Entry: Payment provider webhook failures
Entry: Customer report "card charged, order missing"
Decision tree (core)
1. Check payment provider status page → outage?
YES → Post status, enable maintenance message, DONE (monitor)
NO → step 2
2. Recent deploy < 30 min?
YES → Execute rollback runbook §1 → verify → DONE or escalate
NO → step 3
3. DB connection pool exhausted?
YES → Scale pool / kill long queries (link command) → verify
NO → escalate to payments on-call
Commands copy-paste ready
Exact CLI, dashboard URLs, SQL with placeholders marked {{service}}. No "run the usual cleanup."
Verification
How to know the incident is resolved — metric thresholds, smoke test, sample transaction.
Communication template
Status page text, internal Slack message — fill-in-the-blank.
Link to rollback strategy decision trees inside deploy-related branches — not duplicate, reference.
Organize by symptom, not by service org chart
Service-based runbooks ("Payments Runbook") grow stale when ownership shifts. Symptom-based runbooks ("Checkout failures") align with how alerts and customers arrive.
Cross-link related runbooks:
- High error rate → deploy rollback path
- Elevated latency → database / cache runbook
- Auth failures → identity provider runbook
Index page: alert name → runbook URL. On-call should not search Confluence.
Keep runbooks alive
Runbooks rot faster than code.
Last verified date — quarterly game day or post-incident update required.
Post-incident mandate — every sev-1/sev-2 updates or creates runbook within 48 hours. Blameless postmortem action item: "runbook gap."
Automate steps where possible — runbook triggers script for safe operations (restart worker, clear stuck queue). Human steps for judgment calls.
Length limit — one screen per decision node. Long runbooks split into linked sub-runbooks.
SLO burn alerts tell you that you are in incident territory. Runbooks tell you what to do — pair every burn alert with runbook link in alert payload.
Roles during incidents: what runbooks assume
Even small teams benefit from explicit roles in runbook header:
| Role | Responsibility |
|---|---|
| Incident commander | Coordinates, communicates, decides escalation |
| Investigator | Follows technical tree |
| Communicator | Status page, customer support sync |
| Scribe | Timeline for postmortem |
Solo on-call may wear all hats — runbook still notes which hat each section serves so nothing is forgotten.
How should teams write incident runbooks?
These standards separate operational runbooks from wiki filler.
How long should a runbook be?
As short as possible for the decision path. Target: first action within 60 seconds of opening. If architecture context is needed, link to separate doc — do not prepend.
What belongs in the runbook vs postmortem?
Runbook: how to fix or mitigate now. Postmortem: why it happened, systemic fixes. Runbooks may note known false positives from past incidents — one line, not full postmortem.
Should runbooks include rollback commands?
Yes, when deploy correlation is common failure mode. Reference centralized rollback decision tree; include service-specific commands (redeploy tag, toggle flag name).
A common argument runs the other way
The opposing view holds that good observability and automated remediation make runbooks obsolete — that dashboards and self-healing systems replace written procedures.
Automation handles known knowns. Incidents are often novel combinations — deploy plus provider outage plus traffic spike. Runbooks capture organizational memory for check sequences automation should not run unsupervised (rollback, data repair). Observability without runbooks produces informed paralysis — lots of graphs, no next step.
Key takeaways
- Runbooks start with symptoms and alerts — not architecture diagrams.
- Decision trees with yes/no branches beat prose; first action within 60 seconds.
- Copy-paste commands, verification thresholds, communication templates mandatory.
- Alert name → runbook URL index; stale runbooks worse than none — verify quarterly.
- Post-incident updates within 48 hours for sev-1/sev-2; link rollback and SLO runbooks.
- Split long procedures into linked sub-runbooks; one screen per decision node.
Conclusion
Incident response runbooks that get used are written for the version of on-call who has been awake ninety minutes and does not remember last quarter's deploy. Symptoms first, branches on observable facts, verification before all-clear. Everything else is documentation with a runbook label.
The exercise: pick last three pages. For each, open the linked runbook. Time to first executable action. Over two minutes — rewrite before the next page.