signcodify
Start free

Incident runbook

A free incident runbook template: severity levels, detection, first five minutes, diagnosis, mitigation, communication, rollback and follow-up, with commands and a status update template.

Best for
On-call engineers and SRE teams responsible for production services
Length
One page per alert or failure mode
Sections
8

A runbook tells the person on call exactly what to do when a specific thing breaks. It is written for someone who was asleep two minutes ago and may not know this service well.

Use one runbook per alert or failure mode, and keep commands copy-paste ready. The example is for a fictional checkout API.

Tip. Link each runbook from the alert itself. A runbook nobody can find at 3 a.m. does not exist.

  1. 01

    Summary

    Which alert triggers this runbook, what it means for users in one sentence, and the service owner.

    text
    Alert:   checkout-api 5xx rate above 2% for 5 minutes
Impact:  customers cannot complete payment
Owner:   payments team, #payments-oncall

  2. 02

    Severity

    How to decide the severity level, with clear examples, so nobody debates it during an incident. Link to the company-wide definitions.

    • SEV1: checkout fully down or data loss; page the incident commander
    • SEV2: errors above 5% or one region down
    • SEV3: elevated errors under 5%, no customer reports yet

  3. 03

    First five minutes

    A short, ordered checklist: acknowledge, open an incident channel, check the dashboard, check recent deploys. The goal is to stop guessing quickly.

    text
    1. Acknowledge the page
2. Open #inc-<date>-checkout and post: "Investigating checkout errors"
3. Open the checkout dashboard (link)
4. Check deploys in the last 60 minutes:
   kubectl -n payments rollout history deploy/checkout-api

  4. 04

    Diagnosis

    The likely causes in order of how often they happen, each with the command or query that confirms or rules it out.

    • Recent deploy: compare error start time with the rollout time
    • Payment provider outage: check the provider status page and the provider error code in logs
    • Database connections exhausted: check the pool saturation panel
    • Expired certificate or secret: check the secrets expiry report

  5. 05

    Mitigation

    The fastest safe actions to reduce impact, even before the root cause is known. Restore service first, investigate second.

    bash
    # roll back the last deploy
kubectl -n payments rollout undo deploy/checkout-api

# shift traffic away from a failing region
./scripts/traffic shift --from us-east --to us-west --percent 100

  6. 06

    Communication

    Who to tell and when: internal updates every 30 minutes for SEV1 and SEV2, and when to post on the public status page. Include a template so updates are fast and consistent.

    text
    Status: Investigating | Identified | Monitoring | Resolved
Impact: Some customers see errors at checkout.
What we are doing: Rolling back a deploy from 14:05 UTC.
Next update: 14:45 UTC

  7. 07

    Rollback and recovery checks

    How to confirm the service is healthy again: which metrics to watch, for how long, and what must be true before the incident is marked resolved.

  8. 08

    Follow-up

    Open a postmortem for every SEV1 and SEV2 within two working days, blameless and focused on systems. Update this runbook with anything that was missing or wrong.

    • Timeline with UTC timestamps
    • Customer impact and duration
    • Contributing factors
    • Action items with owners and dates

Mistakes to avoid

  • Commands that need edits before they run, such as placeholder namespaces.
  • Diagnosis steps without a way to confirm or rule out each cause.
  • Investigating the root cause before mitigating customer impact.
  • Runbooks that are not linked from the alert, so on-call cannot find them.
  • Never updating the runbook after an incident proves it wrong.

Questions about this template

What is the difference between a runbook and a playbook?
A runbook covers one specific alert or failure with exact steps. A playbook covers a type of situation, such as a security incident or a full region outage, and coordinates roles across teams.
How do we keep runbooks current?
Review them after every incident they were used in, and give each one an owner. Signcodify marks a page as stale when the linked code or config changes and the page does not.
Should runbooks be private?
Yes, usually. They contain internal hostnames and commands. Keep them in a private space with on-call engineers as readers.

Last updated October 30, 2026

Suggest an edit

Use this template in Signcodify

It opens in the block editor with every section above in place. Sync it to your repository or publish it to your docs site.

Use this template