Monitoring and Alerting Strategy for Startups: What to Alert On, How to Reduce Noise, and When to Page

Most small engineering teams set up monitoring the same way: add every metric that looks interesting to a dashboard, wire up thresholds for CPU and memory, and route everything to a Slack channel. Within two weeks, the Slack channel is muted by the entire team and no one looks at it.

The problem is not a lack of metrics. It is a lack of a decision framework for what different signals should actually trigger. An alert that wakes someone up at 2 AM carries a very different cost than a message in a Slack channel, which carries a very different cost than a log entry no one reads until an incident. Conflating these three categories is what produces alert fatigue.

This guide covers how to build an alerting strategy that your on-call engineers will trust rather than ignore.

---

The Alert Pyramid

The most useful mental model for alert classification is a three-tier pyramid. At the top: pages. In the middle: notifications. At the base: logs.

Pages (PagerDuty, OpsGenie, phone call) are reserved for conditions that require human action within minutes. If the condition can wait until morning, it is not a page. If there is nothing the on-call engineer can do about it right now, it is not a page. The bar is high: broken checkout flow, API returning 5xx for more than 60 seconds, payment processing down.

Notifications (Slack channel, email) are for conditions that degrade the system and need attention within hours, but not immediately. High error rates in background jobs, elevated latency on non-critical paths, disk usage above 70%, unusual traffic patterns. These show up in a low-interruption channel that the team checks during business hours.

Logs are for everything else. Failed individual requests, transient timeouts that self-healed, debug signals you need when investigating an incident but that do not represent actionable degradation on their own.

The common failure mode is treating notifications as pages and logs as notifications. This bloats both tiers and destroys team trust in the system.

A useful exercise: audit your existing alerts and ask, “If this fires at 3 AM, should someone actually wake up?” For most startups, the honest page list is 5-8 conditions.

---

SLI-Based Alerting vs. Threshold-Based Alerting

Threshold-based alerting is the default: CPU above 80%, memory above 90%, response time above 500ms. It is easy to set up and easy to understand, but it fires too often on conditions that do not affect users and misses conditions that do.

SLI-based alerting starts from a different question: what does “working correctly” mean from the user’s perspective? You define a Service Level Indicator (SLI) for each critical user journey, then alert when actual performance falls below the threshold that would breach your target.

For an API-heavy product, your SLIs are typically:

• Availability: percentage of requests that return non-5xx responses
• Latency: percentage of requests completing under some threshold (p95 or p99)
• Error rate: percentage of requests returning application-level errors

An SLO (Service Level Objective) sets a target: 99.5% availability, p95 latency under 400ms. An alert fires when your error budget consumption rate is burning faster than your SLO allows.

The non-obvious benefit of error budget alerting is that it naturally handles bursty traffic. If your API normally serves 10,000 requests per hour and 50 of them fail, that is a 0.5% error rate. Whether you alert on that depends on your SLO and how much budget has already been consumed that month, not on the raw count.

A simple SLI calculation for a 30-day rolling window:

interface ErrorBudgetStatus {
  sloTarget: number;        // e.g., 0.995
  windowDays: number;       // e.g., 30
  totalRequests: number;
  errorRequests: number;
  currentAvailability: number;
  errorBudgetMinutes: number;
  consumedBudgetMinutes: number;
  burnRate: number;
  shouldPage: boolean;
}

function calculateErrorBudget(
  sloTarget: number,
  windowDays: number,
  totalRequests: number,
  errorRequests: number
): ErrorBudgetStatus {
  const windowMinutes = windowDays * 24 * 60;
  const currentAvailability = (totalRequests - errorRequests) / totalRequests;

  // Total minutes of downtime allowed in the window
  const errorBudgetMinutes = (1 - sloTarget) * windowMinutes;

  // Minutes "consumed" by current error rate
  const consumedBudgetMinutes = (1 - currentAvailability) * windowMinutes;

  // How fast are we consuming the budget? 1.0 = exactly on pace, >1.0 = too fast
  const burnRate = consumedBudgetMinutes / errorBudgetMinutes;

  // Page if burning at 14x the sustainable rate (common 1-hour burn rate heuristic)
  const shouldPage = burnRate > 14;

  return {
    sloTarget,
    windowDays,
    totalRequests,
    errorRequests,
    currentAvailability,
    errorBudgetMinutes,
    consumedBudgetMinutes,
    burnRate,
    shouldPage,
  };
}

The burn rate multiplier matters here. A burn rate of 1x means you will exhaust the entire monthly budget in 30 days. A burn rate of 14x means you will exhaust it in roughly 50 hours. The 14x threshold is the Google SRE workbook’s recommendation for a 1-hour window page trigger. You adjust it based on your actual SLO window and how quickly your team can respond.

For most startups, starting with SLI-based alerting on your top 2-3 user journeys and keeping threshold-based alerts only for infrastructure limits (disk at 85%, database connections near pool max) is a practical split.

---

Custom Health Check Endpoints

Before any alerting tool can do its job, you need reliable health check endpoints that distinguish between “the server is running” and “the system is actually working.”

A minimal but production-useful health check pattern:

import { Request, Response } from "express";

interface HealthCheckResult {
  status: "healthy" | "degraded" | "unhealthy";
  checks: Record<string, ComponentHealth>;
  latencyMs: number;
}

interface ComponentHealth {
  status: "healthy" | "degraded" | "unhealthy";
  latencyMs: number;
  message?: string;
}

async function checkDatabase(pool: DatabasePool): Promise<ComponentHealth> {
  const start = Date.now();
  try {
    await pool.query("SELECT 1");
    return { status: "healthy", latencyMs: Date.now() - start };
  } catch (err) {
    return {
      status: "unhealthy",
      latencyMs: Date.now() - start,
      message: err instanceof Error ? err.message : "Unknown database error",
    };
  }
}

async function checkRedis(client: RedisClient): Promise<ComponentHealth> {
  const start = Date.now();
  try {
    const pong = await client.ping();
    if (pong !== "PONG") {
      return { status: "degraded", latencyMs: Date.now() - start, message: "Unexpected PING response" };
    }
    return { status: "healthy", latencyMs: Date.now() - start };
  } catch (err) {
    return {
      status: "unhealthy",
      latencyMs: Date.now() - start,
      message: err instanceof Error ? err.message : "Unknown Redis error",
    };
  }
}

export async function healthHandler(req: Request, res: Response): Promise<void> {
  const start = Date.now();

  const [db, cache] = await Promise.allSettled([
    checkDatabase(req.app.locals.db),
    checkRedis(req.app.locals.redis),
  ]);

  const checks: Record<string, ComponentHealth> = {
    database: db.status === "fulfilled" ? db.value : { status: "unhealthy", latencyMs: 0, message: "Check threw" },
    cache: cache.status === "fulfilled" ? cache.value : { status: "unhealthy", latencyMs: 0, message: "Check threw" },
  };

  const hasUnhealthy = Object.values(checks).some((c) => c.status === "unhealthy");
  const hasDegraded = Object.values(checks).some((c) => c.status === "degraded");

  const overallStatus: HealthCheckResult["status"] = hasUnhealthy
    ? "unhealthy"
    : hasDegraded
    ? "degraded"
    : "healthy";

  const statusCode = overallStatus === "unhealthy" ? 503 : 200;

  res.status(statusCode).json({
    status: overallStatus,
    checks,
    latencyMs: Date.now() - start,
  });
}

Two endpoints are worth having: a shallow /health that only checks if the process is running (used by your load balancer to route traffic) and a deep /health/ready that runs the above checks (used by your uptime monitor and alerting tools). Never route load-balancer health checks to the deep endpoint. If your Redis check flaps, you do not want the load balancer pulling the instance.

---

Alert Webhook Handler

When an alerting tool fires, you often want to route it to different destinations depending on severity. A webhook handler lets you normalize payloads from different tools and apply your routing logic in code rather than in the UI of six different SaaS products:

import { Request, Response } from "express";

type AlertSeverity = "critical" | "warning" | "info";

interface NormalizedAlert {
  id: string;
  severity: AlertSeverity;
  title: string;
  description: string;
  source: string;
  firedAt: Date;
  labels: Record<string, string>;
  runbookUrl?: string;
}

// Datadog monitor webhook payload shape
interface DatadogWebhookPayload {
  id: string;
  priority: string;
  title: string;
  text: string;
  alert_transition: string;
  tags: string[];
  alert_id: string;
}

function normalizeDatadog(payload: DatadogWebhookPayload): NormalizedAlert {
  const severity: AlertSeverity =
    payload.priority === "P1" ? "critical"
    : payload.priority === "P2" ? "warning"
    : "info";

  const runbookTag = payload.tags.find((t) => t.startsWith("runbook:"));

  return {
    id: payload.alert_id,
    severity,
    title: payload.title,
    description: payload.text,
    source: "datadog",
    firedAt: new Date(),
    labels: Object.fromEntries(payload.tags.map((t) => t.split(":") as [string, string])),
    runbookUrl: runbookTag ? runbookTag.replace("runbook:", "") : undefined,
  };
}

async function routeAlert(alert: NormalizedAlert): Promise<void> {
  if (alert.severity === "critical") {
    // Page on-call via PagerDuty
    await pagerduty.createIncident({
      title: alert.title,
      body: alert.description,
      severity: "critical",
      source: alert.source,
      details: { runbookUrl: alert.runbookUrl, labels: alert.labels },
    });
  }

  // All severity levels get a Slack message; the channel differs
  const channel = alert.severity === "critical" ? "#incidents" : "#alerts";
  await slack.postMessage({
    channel,
    text: `[${alert.severity.toUpperCase()}] ${alert.title}`,
    blocks: buildAlertBlocks(alert),
  });
}

export async function webhookHandler(req: Request, res: Response): Promise<void> {
  try {
    const source = req.headers["x-alert-source"] as string;
    let normalized: NormalizedAlert;

    if (source === "datadog") {
      normalized = normalizeDatadog(req.body as DatadogWebhookPayload);
    } else {
      res.status(400).json({ error: "Unknown alert source" });
      return;
    }

    await routeAlert(normalized);
    res.status(200).json({ ok: true });
  } catch (err) {
    console.error("Alert routing failed", err);
    res.status(500).json({ error: "Routing failed" });
  }
}

The key design decision here is normalization first, routing second. Different tools have wildly different payload shapes. Normalizing to a shared NormalizedAlert type means your routing logic stays clean and you can add new alert sources without rewriting routing conditions.

---

Fighting Alert Fatigue

Alert fatigue has a simple operational definition: the on-call engineer sees an alert and their first instinct is to acknowledge it and go back to sleep rather than investigate. Once you reach that state, your alerting is worse than useless. It trains people to ignore signals that may be real.

Three practices that meaningfully reduce noise:

Deduplication. If the same condition fires every 5 minutes, you want one incident, not 288 notifications per day. PagerDuty and OpsGenie handle this natively for pages. For Slack notifications, you need to implement it yourself: track the last notification time per alert fingerprint and suppress duplicates within a cooldown window (15-30 minutes is reasonable).

Correlation. A database slowdown often causes elevated API latency, which causes failed background jobs, which causes payment retry errors. Each of those can fire its own alert. If you do not correlate them into a single incident, the on-call engineer wakes up to five separate pages about the same root cause. Most incident management tools let you group alerts by a shared label (environment, service, component). Use it.

Runbooks, not summaries. An alert that says “API error rate elevated” is noise. An alert that says “API error rate elevated on /api/payments/charge. Last occurrence: [link]. On-call runbook: [link]” is signal. The runbook does not need to be long. It needs to answer three questions: What does this alert mean? What should I check first? What can I do without waking anyone else up? If writing a runbook for an alert takes more than 15 minutes, the alert is too vague.

A quarterly alert review is worth scheduling as a recurring calendar item. The agenda is simple: which alerts fired most in the last 90 days, which ones were false positives, and which ones were acknowledged but not investigated. Any alert with a false positive rate above 20% should be raised or turned into a log.

---

Tool Comparison

Dimension	Datadog	Grafana + Prometheus	Checkly + BetterStack
Setup time	30 min (agent install)	2-4 hours (self-hosted)	15 min
Cost at 5 engineers	$150-400/mo	Hosting cost only	$50-100/mo
SLO/error budget UI	Built-in, solid	Requires configuration	Basic in BetterStack
Custom metrics	Agent + API	PromQL, flexible	Webhook only
Log aggregation	Yes (expensive)	Yes (Loki)	No
Alerting routing	Integrated	AlertManager	Native + webhook
Best for	Full-stack visibility	Cost-sensitive, infra-heavy	Uptime and API monitoring

For most seed to Series A teams, the pragmatic starting point is Checkly or BetterStack for uptime and external health checks, combined with whatever APM your cloud provider includes (CloudWatch, Google Cloud Monitoring). Datadog is worth the cost when you have distributed services and need trace-level correlation between API calls. Grafana and Prometheus make sense when you are self-hosting infrastructure and have an engineer who will maintain the stack.

The mistake is over-investing in observability tooling before you have defined what you actually want to observe. Tool selection follows alert taxonomy, not the other way around.

---

On-Call Rotation and Escalation Policies

An on-call rotation without a written escalation policy is a socially negotiated system. That means the most responsive engineer becomes the de-facto on-call for every incident, regardless of rotation, because they respond before anyone else does.

Write the policy down:

1. Primary on-call responds within 15 minutes.
2. If no acknowledgment, secondary on-call gets paged at 15 minutes.
3. If no acknowledgment from secondary, the engineering manager is called at 30 minutes.
4. For any incident lasting more than 60 minutes, a second engineer joins.

The escalation is not a judgment about the on-call engineer’s capability. It is a forcing function that keeps incidents from dragging. People are more likely to escalate when it is the documented process rather than an admission that they need help.

Rotation cadence depends on team size. With 3-4 engineers, a weekly rotation with an explicit “shadow” week for new engineers is manageable. Below 3 engineers, a formal rotation is less useful than a shared understanding of who the backup person is on any given week.

---

Measuring Whether Your Alerting Is Working

You cannot improve what you do not measure. Four metrics that tell you whether your alerting strategy is healthy:

Signal-to-noise ratio: the percentage of fired alerts that required and received a human response within the expected window. If 40% of your critical alerts were acknowledged but closed without action, they were probably noise.

Time to acknowledge (TTA): how long between an alert firing and someone acknowledging it. A TTA consistently above 10-15 minutes on a critical alert suggests the on-call rotation is not functioning. A TTA of under 2 minutes on warning-level alerts suggests you are over-paging for non-critical conditions.

False positive rate: the percentage of alerts that, in post-mortem review, did not correspond to a real user-affecting problem. Target below 10% for pages, below 25% for notifications.

Mean time to resolve (MTTR) vs. time-to-detect (TTD): if TTD is consistently shorter than MTTR, your alerts are working. If MTTR is short but TTD is long (you are finding problems through user reports, not alerts), you are missing coverage on real failure modes.

Track these in a simple spreadsheet if you do not have tooling for it. The goal is not a perfect score. It is a trend line that tells you whether your quarterly alert reviews are having any effect.

---

Production Considerations

A few sharp edges that are worth naming explicitly:

On-call handoff documentation. At the start of each rotation, the outgoing on-call should write a brief note: what fired this week, what was investigated, what is still open. Without this, the incoming on-call has no context and spends the first hour of their rotation reconstructing history.

Alert testing. Alerts that have never fired in production may not fire when you need them to. Schedule a monthly synthetic test: deliberately trigger a condition (take a replica out of rotation, push a spike in synthetic error rate) and verify that the page actually lands.

Timezone hygiene. If your team is distributed across timezones, hard-coded on-call schedules in UTC are the safest approach. Tools that show local time in alert notifications are helpful but can introduce confusion during daylight saving transitions.

Post-mortem culture. The highest-value activity after a major incident is not fixing the immediate issue but writing down what the alerting system got right and wrong. Did the alert fire quickly? Did the runbook help? Did the on-call engineer spend 20 minutes figuring out who else to wake up? Those observations are the input for your next alert review.

The goal is not a monitoring system that never produces false positives. It is one where your engineers trust the pages enough to respond to them, the notifications enough to act on them, and the logs enough to rely on them when debugging. That trust is built incrementally, one alert review at a time.