Incident Management for Small Teams: Runbooks, Postmortems, and Escalation That Actually Works

Every incident management guide assumes a dedicated SRE team, tiered on-call rotation, and a platform team maintaining your observability stack. If you have two engineers and one is on vacation when production breaks, you need different advice.

This is that advice.

The core problem with incident process at small scale

Small teams copy enterprise incident frameworks and then stop using them. The runbooks are too long. The severity matrix requires a committee to apply. The postmortem template demands eight sections before anyone writes the first sentence. Under stress, people skip the process entirely, which is exactly when you need it.

The goal is not to implement Google’s SRE handbook. The goal is a minimum viable process that engineers actually follow at 2 AM when they’ve been paged.

---

Severity classification that isn’t theater

Most severity frameworks use five levels. Most teams use two: “bad” and “very bad.” Build around reality.

A three-tier system works for teams under ten:

SEV-1: Production is down or data is at risk. Revenue or user data is actively impacted. Wake someone up now, regardless of time.

SEV-2: Significant degradation. Core features broken for a subset of users. Needs same-day resolution during waking hours.

SEV-3: Minor degradation or a bug with a workaround. Log it, fix it in the next sprint.

The most important rule: one person makes the severity call, immediately. If you find yourself in a Slack thread debating whether something is SEV-1 or SEV-2, you’ve already wasted fifteen minutes. The person who gets paged first makes the call. If they’re wrong, you adjust. Being wrong about severity costs less than deliberating.

Define your SEV-1 criteria concretely enough that classification is a one-question check:

// document this somewhere engineers can find it in < 10 seconds
const SEV1_CRITERIA = [
  "Checkout / payment flow is broken",
  "Authentication is down (users cannot log in)",
  "Data loss or corruption is occurring or suspected",
  "API error rate > 10% sustained for 5+ minutes",
  "Response times > 10x baseline sustained for 5+ minutes",
] as const;

// If your incident matches any item above, it's SEV-1.
// If you're uncertain, treat it as SEV-1 and downgrade later.

That last rule matters. Downgrading an over-escalated incident wastes thirty seconds. Under-escalating a real SEV-1 can cost hours.

---

Runbooks that people actually use under stress

The failure mode of runbooks: they’re written when calm, read when panicked, and abandoned when they don’t match the current situation.

Runbooks that get used share three properties. They’re short (under two pages per failure mode). They’re imperative (do X, then do Y) not explanatory (X happens because Y interacts with Z). And they include exact commands, not descriptions of commands.

Here’s a practical runbook structure:

# [Service Name]: High Error Rate

**When to use this runbook:** API error rate alert fires, or error rate
visible in Grafana dashboard > 5% for 5+ minutes.

## Step 1: Confirm the scope (2 minutes)
Check: https://grafana.internal/d/api-errors
- Is it one endpoint or all endpoints?
- Is it one region or all regions?
- When did it start? (Check deploy history)

## Step 2: Check recent deploys
git log --oneline -20
# Or check your CI/CD dashboard. A deploy in the last 30 minutes
# is the most likely cause.

## Step 3: If recent deploy is suspect, roll back
# For Cloudflare Workers:
npx wrangler deployments list
npx wrangler rollback [deployment-id]

# For a Node.js service on Railway/Fly.io:
# Use the dashboard rollback button. It's faster than the CLI.

## Step 4: If no recent deploy, check dependencies
curl -s https://status.stripe.com/api/v2/status.json | jq '.status.indicator'
curl -s https://status.aws.amazon.com/ | grep -i 'service disruption'

## Step 5: If dependency issue confirmed
- Set status page to "investigating" with description
- Notify on-call Slack channel with dependency name and your source
- Wait and monitor; no code changes needed

## Step 6: If cause still unknown
- Escalate to [name] at [contact]
- Increase log verbosity: [exact command or link to config]
- Pull last 100 error logs: [exact query]

## Resolution checklist
- [ ] Error rate back to baseline
- [ ] Status page updated to "resolved"
- [ ] Incident timeline posted to #incidents channel
- [ ] Postmortem scheduled (SEV-1) or ticket created (SEV-2)

Notice what’s not in this runbook: architecture diagrams, explanations of why the system works this way, or conditional branches beyond a single branch point. Those belong in your system documentation, not your runbook.

One runbook per named failure mode. “Database issues” is not a runbook. “Primary database connection exhaustion” is a runbook.

Store runbooks where engineers will actually find them during an incident. Notion or Confluence work. A runbooks/ directory in your repo works. A shared Google Doc works. What doesn’t work: a runbook that requires three clicks and a search to locate.

---

Communication during an incident

Silence is the worst thing you can do during an incident. Stakeholders filling the void with speculation is worse than a status update that says “we don’t know the cause yet.”

The incident channel pattern: When a SEV-1 starts, one person creates a dedicated Slack channel named #inc-YYYY-MM-DD-short-description. All incident communication goes there. This isolates noise from your main engineering channel and creates a log of the incident automatically.

Status page updates: Every five minutes during active investigation, even if the update is “still investigating, no new information.” The format:

[14:32] Investigating elevated API error rates affecting checkout flow.
Cause unknown, team engaged.

[14:37] Identified a dependency issue with our payment processor.
Working on a workaround.

[14:51] Deployed workaround. Error rates returning to normal.
Monitoring for 10 minutes before marking resolved.

[15:03] Resolved. Root cause: Stripe webhook processing queue backed up
due to high traffic. Workaround: increased worker concurrency.
Full postmortem to follow.

For status pages, you don’t need Atlassian Statuspage at $299/month. Alternatives worth using:

• Instatus: $20/month for most small teams. Good API, clean UI.
• Freshstatus: Free tier is generous for small teams.
• Upptime: Free, GitHub-based, open source. Works well if you’re comfortable with GitHub Actions managing your status page.
• Cachet: Self-hosted, free. Operational overhead is real but it’s zero subscription cost.

For stakeholder communication during SEV-1, one update per fifteen minutes is the right cadence. Assign this explicitly to one person. The person debugging the incident should not be writing stakeholder updates. On a two-person team, that’s impractical, so the update cadence drops to “when something changes.”

---

Escalation when you don’t have tiers

Enterprise escalation assumes you have L1, L2, L3 support and an SRE team. You have a Slack group with three engineers in it.

Escalation at small scale has two paths: technical escalation (this person knows the system better) and decision escalation (this requires someone with authority to make a call).

Build an explicit escalation contact list. Not a rotation, just a list. For each critical system, one person is the technical owner. That person’s phone number and Slack handle is written down somewhere accessible to the whole team. When the on-call engineer hits a wall, they call the system owner directly. No tickets, no queue.

# Escalation Contacts

## Payment integration (Stripe)
Primary: @alice (knows the webhook system end to end)
Backup: @bob (wrote the original integration)
External: Stripe support +1-888-926-2289 (have account credentials ready)

## Auth system
Primary: @carol
Backup: @alice
External: Auth0 support (plan includes incident response: [link to ticket portal])

## Infrastructure (AWS)
Primary: @bob
External: AWS Support (Premium: [link to case portal], case PIN: XXXXX)

## Business decisions (take prod offline, roll back data)
Escalate to: [CTO/founder name], [phone number], [Signal/WhatsApp handle]

The business decisions row is the one teams forget. There are incidents where the correct technical action is clear but requires authorization: rolling back a migration that affects customer data, taking a service offline to prevent data corruption, or notifying customers about a security event. Know ahead of time who makes those calls and how to reach them outside business hours.

---

Postmortems that produce real improvements

Most postmortems are blame laundering operations. Someone writes “human error” as the root cause, the team nods, and nothing changes. The next incident repeats.

A blameless postmortem starts from one premise: every engineer made the best decision they could with the information available at the time. The question is never “who made a mistake” but “what made the mistake easy to make and hard to catch.”

When to write a postmortem: Every SEV-1. SEV-2 incidents that recur or reveal a systemic issue. Nothing else. A team of five cannot sustain weekly postmortems; they’ll stop writing them.

Postmortem structure that works:

# Incident Postmortem: [Short description]
Date: YYYY-MM-DD
Duration: X hours Y minutes
Severity: SEV-1
Author: [Name]
Reviewers: [Names]

## What happened (2-3 sentences, plain English)
At 14:23 UTC, our payment API began returning 500 errors for all
checkout requests. The incident lasted 40 minutes and affected
approximately 200 transactions. No payments were lost; all failed
requests were retried successfully.

## Timeline
| Time  | Event |
|-------|-------|
| 14:23 | Alert fired: API error rate > 10% |
| 14:25 | On-call acknowledged, started investigation |
| 14:31 | Identified Stripe webhook backlog as cause |
| 14:47 | Deployed concurrency fix |
| 15:03 | Error rate normalized, incident closed |

## Root cause
[One specific technical statement. "The Stripe webhook processor
has a concurrency limit of 5 workers. A traffic spike at 14:20
created a queue of 800+ webhooks. The processor fell behind,
causing payment status checks to time out across all checkout flows."]

## Contributing factors
- [Specific, concrete items. Not "lack of monitoring" but
  "no alert existed for webhook queue depth"]
- [Not "inadequate capacity planning" but "worker concurrency
  was set to 5 in 2023 and never revisited as traffic grew 4x"]

## Action items
| Item | Owner | Due |
|------|-------|-----|
| Add alert: webhook queue depth > 100 | @alice | 2026-03-28 |
| Add dashboard panel: webhook processor lag | @alice | 2026-03-28 |
| Increase base concurrency to 20, document scaling formula | @bob | 2026-04-01 |
| Runbook: webhook backlog recovery steps | @carol | 2026-04-01 |

## What went well
- On-call response time was under 3 minutes
- Rollback procedure worked cleanly
- Communication to stakeholders was timely

## What we'll do differently
[Only if different from the action items above]

The action items section is where most postmortems fail. Vague items (“improve monitoring”) never get done. Every item needs an owner, a due date, and a specific deliverable. Review completion in sprint planning. If an item slips three sprints in a row, that’s a signal about priority.

---

Tooling on a budget

Alerting and on-call routing:

PagerDuty starts at $21/user/month. For a three-person team, that’s $63/month minimum, more once you need advanced routing. Alternatives:

• Better Uptime: $20/month flat for small teams, includes on-call rotation, status page, and incident timeline.
• OpsGenie: Has a free tier for up to 5 users with basic alerting. Atlassian’s pricing has increased, but the free tier remains usable.
• Grafana OnCall: Open source, self-hosted or Grafana Cloud. Free for small teams on Grafana Cloud. If you’re already using Grafana for observability (you should be), this is the obvious choice.
• ntfy.sh + custom alerts: For very small teams, a self-hosted ntfy instance + custom webhook alerts to your phone is free and surprisingly reliable. You lose on-call rotation management, but you gain zero subscription cost.

Runbook storage:

Linear, Notion, GitHub Wiki, or a docs/runbooks/ directory in your monorepo all work. The right choice is wherever your engineers already go when they need to look something up. Don’t add a new tool for runbooks.

Incident coordination:

Slack is sufficient. If you’re using Discord or Teams, those work too. The key is a naming convention for incident channels and a bot or workflow that creates the channel automatically when an alert fires. Slack’s workflow builder can do this for free.

---

Building incident response muscle memory

The problem with incident response is that you only practice it when something breaks. By then, stress is high and you’re learning in production.

Game days fix this. A game day is a scheduled exercise where you deliberately break something in staging and run your incident response process as if it’s real. One engineer plays the responder, everyone else watches and notes gaps, then you hold a short retrospective.

One hour, once a quarter, covers the basics. The goal is not to simulate every failure mode. The goal is to make sure everyone knows where the runbooks are, knows how to create an incident channel, knows how to update the status page, and has done it at least once before they need to do it under pressure.

A minimal game day scenario:

# Game Day: Simulated Database Connection Exhaustion

**Setup (done by facilitator before game day):**
- Set database max_connections to 5 in staging
- Run a connection-leaking script: [link to script]
- Watch for the alert to fire

**Responder's objective:**
- Acknowledge the alert within 5 minutes
- Create an incident channel
- Identify the cause using only the runbook and available tooling
- Post a status update to the staging status page
- Resolve the incident
- Produce an incident timeline within 30 minutes of resolution

**Facilitator notes:**
- Observe and note any steps where the responder got stuck
- Do not help. Let them use the runbook.
- Record what was missing from the runbook.

**Debrief questions:**
1. What was the first thing you looked at? Was it the right thing?
2. Where did the runbook help? Where did it fall short?
3. How long did it take to find the relevant alert/dashboard?
4. What would have made this faster?

The debrief surfaces gaps in documentation and tooling that you’ll never find any other way.

---

The minimum viable incident process

For a team under five engineers with no incident process today, start here and add nothing until each item is working:

1. Write SEV-1 criteria as a bulleted list. Post it in your engineering Slack channel.
2. Build one runbook for your most critical failure mode. Keep it under one page.
3. Set up a status page. Instatus or Upptime takes two hours.
4. Define your escalation contacts list. Two names per critical system.
5. Run one game day in the first month.
6. After your first real SEV-1, write a postmortem using the template above.

Don’t add a postmortem process before you’ve had an incident. Don’t buy on-call tooling before you’ve run a game day. Process should follow actual pain.

The teams that handle incidents well are not the ones with the most sophisticated tooling. They’re the ones where every engineer has done it before and knows where to find the information they need. Muscle memory and documentation quality beat process complexity every time.