Automated Incident Response for Small Teams: Runbook Automation, Auto-Remediation, and Escalation Workflows

A three-person team gets paged at 2am just as often as a thirty-person team. The difference is the thirty-person team has an on-call rotation that distributes the pain, a runbook that tells you what to do when your eyes are half open, and automation that handles the recoverable cases before anyone is woken up at all.

Most of the writing on incident response assumes you have a dedicated platform team, an enterprise Datadog contract, and engineers whose full-time job is reliability. This is not that. This is for the 5-person startup where the same engineer who wrote the feature is also on-call for it, and where your monitoring budget competes directly with AWS costs.

The good news is that the hard parts of incident response are not the tooling. They are the decisions: what gets routed where, what gets auto-remediated versus escalated, and what context you want captured for the post-mortem. Those decisions are cheap to encode once you understand the primitives.

Alert Routing and Deduplication

Raw alerts from Prometheus, CloudWatch, or Datadog are noisy. The same underlying problem frequently produces dozens of alerts: the database is slow, so your API latency alert fires, your error rate alert fires, and your synthetic check alert fires. If each of those pages someone separately, you have manufactured an incident out of a single root cause.

The first job of an incident response system is deduplication: group alerts that share a root cause and route a single notification. The second job is routing: send the right alert to the right person, not the same person for everything.

Here is a webhook handler that does both. It accepts alert payloads (Grafana AlertManager format is common across Prometheus, Loki, and Grafana OnCall) and groups them by a fingerprint before deciding whether to page:

interface AlertPayload {
  version: string;
  groupKey: string;
  status: "firing" | "resolved";
  alerts: Alert[];
}

interface Alert {
  labels: Record<string, string>;
  annotations: Record<string, string>;
  startsAt: string;
  endsAt?: string;
}

interface IncidentGroup {
  id: string;
  fingerprint: string;
  alerts: Alert[];
  firstSeenAt: Date;
  lastSeenAt: Date;
  paged: boolean;
}

const incidentGroups = new Map<string, IncidentGroup>();

function fingerprintAlerts(alerts: Alert[]): string {
  // Group by service + alertname, ignoring instance-level labels
  // that vary across replicas but share the same root cause.
  const key = alerts
    .map((a) => `${a.labels["service"] ?? "unknown"}:${a.labels["alertname"] ?? "unknown"}`)
    .sort()
    .join("|");
  return Buffer.from(key).toString("base64url");
}

async function handleAlertWebhook(payload: AlertPayload): Promise<void> {
  if (payload.status === "resolved") {
    await handleResolution(payload);
    return;
  }

  const fingerprint = fingerprintAlerts(payload.alerts);
  const existing = incidentGroups.get(fingerprint);

  if (existing) {
    // Dedup: update the group but do not re-page if already paged
    existing.alerts.push(...payload.alerts);
    existing.lastSeenAt = new Date();
    incidentGroups.set(fingerprint, existing);

    // Re-page only if the incident has been open >30 minutes
    // and is still firing. This catches stuck incidents without
    // flooding the on-call with duplicates.
    const ageMinutes = (Date.now() - existing.firstSeenAt.getTime()) / 60_000;
    if (ageMinutes > 30 && existing.paged) {
      await escalate(existing, "still-firing-after-30m");
    }
    return;
  }

  const group: IncidentGroup = {
    id: crypto.randomUUID(),
    fingerprint,
    alerts: payload.alerts,
    firstSeenAt: new Date(),
    lastSeenAt: new Date(),
    paged: false,
  };

  incidentGroups.set(fingerprint, group);
  await routeIncident(group);
}

async function routeIncident(group: IncidentGroup): Promise<void> {
  const service = group.alerts[0]?.labels["service"];
  const severity = group.alerts[0]?.labels["severity"] ?? "warning";

  // Route to the right on-call based on service ownership.
  // In a small team this might just be "whoever is on-call this week"
  // but encoding service ownership here pays off as the team grows.
  const owner = resolveOwner(service);

  if (severity === "critical") {
    group.paged = true;
    await sendPage(owner, group);
  } else {
    // Warning severity: write to Slack, do not page.
    await sendSlackAlert(owner.slackChannel, group);
  }

  incidentGroups.set(group.fingerprint, group);
}

The resolveOwner function can start as a simple lookup table and grow into a database-backed service ownership registry as the team scales. Keep it simple until the pain of a flat table forces you to generalize.

Runbook Automation

A runbook is a list of diagnostic steps. The problem is that a runbook living in Notion or Confluence requires a half-awake engineer to read it, decide which steps apply, and execute them manually while the incident is live.

Automated runbooks flip this: the system executes the diagnostic steps and attaches the results to the incident before it pages anyone. The on-call wakes up to a notification that already contains pod restart counts, recent deploy timestamps, memory utilization, and the last 50 error log lines. The human job becomes interpretation and decision-making, not data collection.

interface DiagnosticResult {
  step: string;
  output: string;
  durationMs: number;
  error?: string;
}

type DiagnosticStep = (incident: IncidentGroup) => Promise<DiagnosticResult>;

const runbookRegistry: Record<string, DiagnosticStep[]> = {
  "api-high-latency": [checkRecentDeploys, checkPodRestarts, checkDatabaseConnections, fetchRecentErrors],
  "database-slow-queries": [checkActiveConnections, checkLongRunningQueries, checkDiskUtilization],
  "worker-queue-depth": [checkQueueDepth, checkConsumerHealth, checkDeadLetterQueue],
};

async function runDiagnostics(
  incident: IncidentGroup,
  runbookKey: string
): Promise<DiagnosticResult[]> {
  const steps = runbookRegistry[runbookKey] ?? [checkRecentDeploys, fetchRecentErrors];
  const results: DiagnosticResult[] = [];

  // Run steps sequentially so later steps can depend on earlier context.
  // If a step fails, record the error and continue rather than aborting.
  for (const step of steps) {
    const start = Date.now();
    try {
      const result = await Promise.race([
        step(incident),
        timeout(10_000, `Step timed out after 10s: ${step.name}`),
      ]);
      results.push(result);
    } catch (err) {
      results.push({
        step: step.name,
        output: "",
        durationMs: Date.now() - start,
        error: err instanceof Error ? err.message : String(err),
      });
    }
  }

  return results;
}

async function checkRecentDeploys(incident: IncidentGroup): Promise<DiagnosticResult> {
  const start = Date.now();
  const service = incident.alerts[0]?.labels["service"] ?? "unknown";

  // Query your deployment system. This example uses a generic HTTP call;
  // swap in your actual deploy API (ArgoCD, Render, Railway, etc.)
  const response = await fetch(`${process.env.DEPLOY_API_URL}/services/${service}/deployments?limit=5`);
  const deploys = await response.json() as Array<{ sha: string; deployedAt: string; author: string }>;

  const recentDeploy = deploys[0];
  const ageMinutes = recentDeploy
    ? Math.round((Date.now() - new Date(recentDeploy.deployedAt).getTime()) / 60_000)
    : null;

  const output = ageMinutes !== null
    ? `Last deploy: ${recentDeploy.sha.slice(0, 7)} by ${recentDeploy.author}, ${ageMinutes}m ago`
    : "No recent deploys found";

  return { step: "checkRecentDeploys", output, durationMs: Date.now() - start };
}

async function checkPodRestarts(incident: IncidentGroup): Promise<DiagnosticResult> {
  const start = Date.now();
  const namespace = incident.alerts[0]?.labels["namespace"] ?? "default";
  const service = incident.alerts[0]?.labels["service"] ?? "";

  const { execSync } = await import("child_process");
  const raw = execSync(
    `kubectl get pods -n ${namespace} -l app=${service} -o json`,
    { timeout: 8_000 }
  ).toString();

  const podList = JSON.parse(raw);
  const restartSummary = podList.items
    .map((pod: Record<string, unknown>) => {
      const status = pod.status as Record<string, unknown>;
      const containerStatuses = status.containerStatuses as Array<Record<string, unknown>> ?? [];
      const restarts = containerStatuses.reduce((sum: number, cs) => sum + ((cs.restartCount as number) ?? 0), 0);
      return `${pod.metadata?.name}: ${restarts} restarts`;
    })
    .join(", ");

  return {
    step: "checkPodRestarts",
    output: restartSummary || "No pods found",
    durationMs: Date.now() - start,
  };
}

The timeout wrapper prevents a slow Kubernetes API or a hung database query from blocking the diagnostic pipeline. Partial results attached to an incident are better than no results.

Auto-Remediation Patterns

Some incidents have a known fix that is safe to apply automatically. The classic cases are:

• A pod is crash-looping due to a transient external dependency: restart it.
• Traffic spiked and a service has no headroom: scale up the replica count.
• A bad code path is causing errors for a subset of users: toggle a feature flag to disable it.

The risk with auto-remediation is false confidence. If your auto-remediation hides the symptom without fixing the root cause, you have made the system harder to diagnose. Two guardrails help: always attach a remediation record to the incident timeline, and always set a maximum retry count so that auto-remediation cannot loop forever.

interface RemediationAction {
  type: "restart-pod" | "scale-up" | "toggle-feature-flag" | "noop";
  parameters: Record<string, unknown>;
}

interface RemediationResult {
  action: RemediationAction;
  success: boolean;
  output: string;
  appliedAt: Date;
  attemptNumber: number;
}

const MAX_AUTO_REMEDIATION_ATTEMPTS = 2;

async function attemptAutoRemediation(
  incident: IncidentGroup,
  diagnostics: DiagnosticResult[]
): Promise<RemediationResult | null> {
  const action = selectRemediationAction(incident, diagnostics);
  if (action.type === "noop") return null;

  const existingAttempts = await getRemediationAttempts(incident.id);
  if (existingAttempts.length >= MAX_AUTO_REMEDIATION_ATTEMPTS) {
    // Already tried twice. Stop and let the human decide.
    return null;
  }

  const result = await executeRemediation(action, existingAttempts.length + 1);
  await recordRemediationAttempt(incident.id, result);
  return result;
}

function selectRemediationAction(
  incident: IncidentGroup,
  diagnostics: DiagnosticResult[]
): RemediationAction {
  const alertName = incident.alerts[0]?.labels["alertname"] ?? "";
  const service = incident.alerts[0]?.labels["service"] ?? "";
  const namespace = incident.alerts[0]?.labels["namespace"] ?? "default";

  // CrashLoopBackOff with a recent restart count > 3: restart the deployment
  const restartStep = diagnostics.find((d) => d.step === "checkPodRestarts");
  if (alertName === "PodCrashLooping" && restartStep && !restartStep.error) {
    return {
      type: "restart-pod",
      parameters: { namespace, service },
    };
  }

  // High error rate on a known risky feature: toggle the flag off
  if (alertName === "HighErrorRate" && isKnownRiskyFeature(service)) {
    return {
      type: "toggle-feature-flag",
      parameters: { flagKey: `${service}-enabled`, value: false },
    };
  }

  // Queue depth above threshold with healthy consumers: scale up
  if (alertName === "WorkerQueueDepthHigh") {
    return {
      type: "scale-up",
      parameters: { namespace, service, targetReplicas: 6 },
    };
  }

  return { type: "noop", parameters: {} };
}

async function executeRemediation(
  action: RemediationAction,
  attemptNumber: number
): Promise<RemediationResult> {
  const appliedAt = new Date();

  if (action.type === "restart-pod") {
    const { namespace, service } = action.parameters as { namespace: string; service: string };
    const { execSync } = await import("child_process");
    execSync(`kubectl rollout restart deployment/${service} -n ${namespace}`, { timeout: 15_000 });
    return {
      action,
      success: true,
      output: `Restarted deployment/${service} in ${namespace}`,
      appliedAt,
      attemptNumber,
    };
  }

  if (action.type === "scale-up") {
    const { namespace, service, targetReplicas } = action.parameters as {
      namespace: string;
      service: string;
      targetReplicas: number;
    };
    const { execSync } = await import("child_process");
    execSync(
      `kubectl scale deployment/${service} --replicas=${targetReplicas} -n ${namespace}`,
      { timeout: 10_000 }
    );
    return {
      action,
      success: true,
      output: `Scaled ${service} to ${targetReplicas} replicas`,
      appliedAt,
      attemptNumber,
    };
  }

  if (action.type === "toggle-feature-flag") {
    const { flagKey, value } = action.parameters as { flagKey: string; value: boolean };
    await fetch(`${process.env.FEATURE_FLAG_API_URL}/flags/${flagKey}`, {
      method: "PATCH",
      headers: { "Content-Type": "application/json", Authorization: `Bearer ${process.env.FEATURE_FLAG_API_KEY}` },
      body: JSON.stringify({ enabled: value }),
    });
    return {
      action,
      success: true,
      output: `Set flag ${flagKey} to ${value}`,
      appliedAt,
      attemptNumber,
    };
  }

  return {
    action,
    success: false,
    output: "Unknown action type",
    appliedAt,
    attemptNumber,
  };
}

Escalation Workflows

Escalation is not just about paging a second person when the first does not respond. It is about encoding your team’s decision logic: who owns this service, who is the backup, and when does an incident become a severity-1 that wakes up everyone with production access.

interface EscalationPolicy {
  serviceOwner: OnCallContact;
  backup: OnCallContact;
  sev1Threshold: {
    durationMinutes: number;
    orAlertNames: string[];
  };
  broadcastChannel: string; // Slack channel for sev-1 broadcasts
}

interface OnCallContact {
  name: string;
  email: string;
  phoneNumber?: string;
  pagerdutyUserId?: string;
  opsgenieUserId?: string;
}

async function escalate(incident: IncidentGroup, reason: string): Promise<void> {
  const policy = resolveEscalationPolicy(incident.alerts[0]?.labels["service"] ?? "");
  const ageMinutes = (Date.now() - incident.firstSeenAt.getTime()) / 60_000;

  const isSev1 =
    ageMinutes >= policy.sev1Threshold.durationMinutes ||
    incident.alerts.some((a) => policy.sev1Threshold.orAlertNames.includes(a.labels["alertname"] ?? ""));

  if (isSev1) {
    await broadcastSev1(incident, policy, reason);
  } else {
    await pageBackup(incident, policy, reason);
  }
}

async function pageBackup(
  incident: IncidentGroup,
  policy: EscalationPolicy,
  reason: string
): Promise<void> {
  const payload = buildPagePayload(incident, policy.backup, reason);

  if (policy.backup.pagerdutyUserId) {
    await sendPagerDutyAlert(payload, policy.backup.pagerdutyUserId);
  } else if (policy.backup.opsgenieUserId) {
    await sendOpsgenieAlert(payload, policy.backup.opsgenieUserId);
  }
}

async function sendPagerDutyAlert(
  payload: { summary: string; details: Record<string, unknown> },
  userId: string
): Promise<void> {
  await fetch("https://events.pagerduty.com/v2/enqueue", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      Authorization: `Token token=${process.env.PAGERDUTY_ROUTING_KEY}`,
    },
    body: JSON.stringify({
      routing_key: process.env.PAGERDUTY_ROUTING_KEY,
      event_action: "trigger",
      payload: {
        summary: payload.summary,
        source: "incident-automation",
        severity: "critical",
        custom_details: payload.details,
      },
      // Assign to the specific responder when PD Advanced is available.
      // Falls back to on-call schedule when not.
      responders: [{ type: "user_reference", id: userId }],
    }),
  });
}

async function sendGrafanaOnCallAlert(
  alertTitle: string,
  message: string,
  integrationToken: string
): Promise<void> {
  // Grafana OnCall simple alert integration endpoint.
  // Works with the free tier of Grafana Cloud.
  await fetch(`https://oncall-prod-us-central-0.grafana.net/integrations/v1/formatted_webhook/${integrationToken}/`, {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      title: alertTitle,
      message,
    }),
  });
}

The integration with PagerDuty, Opsgenie, and Grafana OnCall all reduce to an HTTP POST with a structured payload. Grafana OnCall is worth calling out specifically: it is free up to a meaningful scale when self-hosted, integrates directly with Grafana dashboards and alerting, and has a functional mobile app. For a team that already runs Grafana for dashboards, it is the lowest-friction starting point.

Incident Timeline Construction

Post-mortems require a timeline. Without automated timeline construction, someone spends an hour correlating Slack messages, deployment logs, and alert timestamps after the fact. If your automation writes structured events during the incident, the timeline is already there when the incident closes.

interface TimelineEvent {
  incidentId: string;
  timestamp: Date;
  eventType:
    | "alert-fired"
    | "diagnostic-started"
    | "diagnostic-completed"
    | "remediation-applied"
    | "escalation-sent"
    | "resolved"
    | "comment";
  actor: "system" | string; // 'system' for automation, name for humans
  description: string;
  metadata?: Record<string, unknown>;
}

async function recordTimelineEvent(event: Omit<TimelineEvent, "timestamp">): Promise<void> {
  const entry: TimelineEvent = { ...event, timestamp: new Date() };

  // Write to your database. The simplest starting point is a single
  // append-only table with a jsonb metadata column.
  await db.query(
    `INSERT INTO incident_timeline (incident_id, timestamp, event_type, actor, description, metadata)
     VALUES ($1, $2, $3, $4, $5, $6)`,
    [entry.incidentId, entry.timestamp, entry.eventType, entry.actor, entry.description, entry.metadata ?? {}]
  );
}

async function buildIncidentSummary(incidentId: string): Promise<string> {
  const events = await db.query<TimelineEvent>(
    `SELECT * FROM incident_timeline WHERE incident_id = $1 ORDER BY timestamp ASC`,
    [incidentId]
  );

  const lines = events.rows.map((e) => {
    const ts = e.timestamp.toISOString();
    const prefix = `[${ts}] ${e.actor}`;
    return `${prefix}: ${e.description}`;
  });

  return lines.join("\n");
}

Storing metadata as JSONB gives you the flexibility to attach diagnostic output, remediation results, and Kubernetes events without defining a rigid schema upfront. Query it with metadata->>'key' when you need to filter post-mortems by remediation type or root cause.

Tradeoffs

Approach	Complexity	Coverage	When to use
Alert-to-Slack only	Minimal	Low	First 3 months, pre-on-call rotation
Alert routing + dedup	Low	Medium	Once you have >3 services in production
Runbook automation	Medium	High	Once you have post-mortems with “we gathered data manually”
Auto-remediation	Medium-High	High	Once you have recurring incidents with a known fix
Full timeline + post-mortem	High	Full	Once incidents are costing you more than the automation would

Production Considerations

Webhook idempotency. AlertManager and Grafana OnCall can deliver the same alert payload multiple times. Your handler needs to be idempotent: check whether the fingerprint already exists before creating a new incident group. A database unique constraint on the fingerprint column is the right backstop.

Runbook permissions. The process running your automation needs kubectl access to restart pods and scale deployments. Scope this carefully with a Kubernetes service account that has only the verbs it needs (get, list, update on Deployments, not delete on everything). In practice: create a dedicated incident-responder service account in each namespace and bind it to a role that allows rollout restarts.

Alert fatigue from auto-remediation noise. If you auto-remediate and then the alert resolves, every resolved alert should update the incident state in your system and send a “resolved” note to the Slack thread. Silence that never explains itself trains your team to ignore the noise.

Opsgenie vs PagerDuty vs Grafana OnCall. For a team of 3-15, the pricing differences are meaningful. Grafana OnCall self-hosted is free with a Grafana stack you are already running. Opsgenie free tier supports up to 5 users. PagerDuty starts at $21/user/month for the features that matter (schedules, escalations, mobile app). If you are already paying for Grafana Cloud, start there.

Dry-run mode. Ship every auto-remediation action with a DRY_RUN=true environment variable that logs the intended action without executing it. Run in dry-run mode for one week of incidents before enabling live remediation. This catches the cases where your selectRemediationAction logic would fire the wrong playbook.

Putting It Together

The integration point is a single HTTP server that receives alert webhooks and orchestrates the rest:

import { createServer, IncomingMessage, ServerResponse } from "http";

async function readBody(req: IncomingMessage): Promise<unknown> {
  return new Promise((resolve, reject) => {
    let data = "";
    req.on("data", (chunk) => { data += chunk; });
    req.on("end", () => { resolve(JSON.parse(data)); });
    req.on("error", reject);
  });
}

const server = createServer(async (req: IncomingMessage, res: ServerResponse) => {
  if (req.method !== "POST" || req.url !== "/webhook/alerts") {
    res.writeHead(404);
    res.end();
    return;
  }

  const payload = await readBody(req) as AlertPayload;
  res.writeHead(202);
  res.end();

  // Respond 202 immediately. AlertManager has a short timeout and will
  // retry if you do not ack quickly. All async work happens after the ack.
  setImmediate(async () => {
    await handleAlertWebhook(payload);
    if (payload.status === "firing") {
      const incidentId = /* resolve from payload */ "";
      const runbookKey = payload.alerts[0]?.labels["runbook"] ?? "default";
      const diagnostics = await runDiagnostics({ id: incidentId } as IncidentGroup, runbookKey);
      const remediation = await attemptAutoRemediation({ id: incidentId } as IncidentGroup, diagnostics);
      if (remediation) {
        await recordTimelineEvent({
          incidentId,
          eventType: "remediation-applied",
          actor: "system",
          description: remediation.output,
          metadata: { action: remediation.action, attempt: remediation.attemptNumber },
        });
      }
    }
  });
});

server.listen(3000, () => console.log("Incident automation listening on :3000"));

Start with the webhook receiver and alert deduplication. Add runbook diagnostics once you have your first post-mortem where the problem was data collection, not the fix. Add auto-remediation only for incidents you have seen at least three times with the same known resolution. The system grows with your understanding of your own failure modes, not in advance of it.

The on-call at 2am is still a person making a judgment call. The automation’s job is to make sure that person is making an informed decision in two minutes instead of spending twenty minutes gathering context.