Service Level Objectives in Practice: Defining SLOs, Burn-Rate Alerting, and Error Budget Policies for Production Systems

Two failure modes are common in teams that try to get serious about reliability. The first is alert fatigue: every metric gets a threshold, every threshold gets a page, and within a month the on-call rotation is silently ignoring half of them. The second is surprise incidents: the team has learned not to trust the alerting system, so they find out about real degradation from customer support tickets.

Service Level Objectives are the structural answer to both problems. They give you a principled way to decide what to alert on, how aggressively to alert, and what to do when the system is degrading. But SLOs are only useful when defined, calculated, and enforced with actual production data. This article works through every step: from choosing what to measure to writing Prometheus alerting rules that catch problems before users notice them.

---

Why Threshold Alerting Fails at Scale

Threshold alerting answers the question “is this metric above or below a number?” That sounds reasonable until your service handles real production traffic with genuine variance. A 500ms latency spike at 3 AM on a Wednesday that lasts 90 seconds is probably not worth waking anyone up. The same spike on a Friday afternoon that continues for 20 minutes is a serious incident.

Threshold alerting cannot distinguish between these two cases. It fires the moment the number crosses the line, regardless of duration, magnitude, or trajectory. You compensate by adding delays and moving thresholds, which introduces its own set of problems: real incidents slip through because you over-damped the alert, and false positives still fire because the threshold is arbitrary.

SLO-based alerting answers a different question: “given how quickly I am spending my error budget right now, when will it run out?” That question incorporates duration, magnitude, and trajectory automatically.

---

Choosing SLIs: What to Measure

A Service Level Indicator is a quantitative measure of some aspect of the service’s behavior. The goal is to pick metrics that directly reflect user experience, not internal system health.

The four standard SLI types cover most services:

Availability: the proportion of valid requests that received a successful response. “Successful” depends on context: for an HTTP API it usually means a non-5xx status code. For a data pipeline it might mean the job completed without error.

Latency: the proportion of requests that completed within a target duration. Note this is a ratio (requests under threshold / total requests), not a raw percentile. This framing makes it compatible with error budget arithmetic.

Error rate: the proportion of requests that resulted in an error. This overlaps with availability for synchronous APIs but becomes more distinct for async processing, queues, and batch systems.

Throughput: relevant when the system has a contractual minimum processing rate, such as a message queue that must consume at least N events per second. Less commonly used as a primary SLI but important for data infrastructure.

The key constraint: an SLI must be a ratio between 0 and 1. Your error budget arithmetic only works if the SLI expresses a proportion of “good events” out of “total events”. A raw latency number like p99=230ms is a statistic, not an SLI.

For most HTTP APIs, start with two SLIs: availability (non-5xx rate) and latency (requests under threshold). Add error rate if you have distinct error categories worth tracking separately.

---

Defining SLO Targets

An SLO is a target value or range for an SLI, measured over a rolling time window. The most common form:

99.5% of HTTP requests over the trailing 28 days will return a non-5xx response.

Choosing the target percentage is where most teams get into trouble. There is a real tradeoff:

SLO Target	Error Budget (28 days)	Implication
99.9%	~40 minutes	Very tight. Any real incident burns budget fast.
99.5%	~3.4 hours	Reasonable for most services.
99.0%	~6.7 hours	Loose. Customers notice degradation at this level.
95.0%	~33.6 hours	Only appropriate for batch or non-critical paths.

The right target is the one that, if violated, your users would actually notice and complain. Look at your support ticket history: at what error rates do customers contact you? At what latencies do they abandon requests? That is your SLO floor.

Tighter SLOs are not free. A 99.9% SLO with a 28-day rolling window gives you about 40 minutes of total budget. A routine deployment that causes 2 minutes of elevated errors burns 5% of your monthly budget. If your deployment pipeline is not reliable enough to absorb that, you will spend the entire month firefighting SLO compliance instead of shipping features.

Start with targets you are already hitting most months, then tighten them over time as your reliability investments compound.

---

Error Budget Calculation

The error budget is the complement of the SLO: the proportion of requests that are allowed to fail over the measurement window.

error_budget = 1 - slo_target
error_budget_minutes = total_minutes * (1 - slo_target)

For a 99.5% availability SLO over 28 days:

28 days * 24 hours * 60 minutes = 40,320 minutes
error_budget_minutes = 40,320 * 0.005 = 201.6 minutes

But the budget is better expressed in terms of request counts, because that is how Prometheus can compute it:

error_budget_requests = total_requests * (1 - slo_target)

The current remaining budget is:

remaining = total_requests - (total_requests * slo_target) - actual_errors

When remaining hits zero, the SLO is violated.

---

Multi-Window, Multi-Burn-Rate Alerting

Burn rate is the rate at which you are consuming your error budget. A burn rate of 1 means you are consuming budget exactly as fast as time passes: you will exhaust the budget at the end of the window. A burn rate of 10 means you will exhaust the 28-day budget in 2.8 days.

The Alertmanager approach recommended by the Google SRE workbook uses multiple alert windows to catch both fast burns (critical incidents that drain budget in hours) and slow burns (subtle degradation that drains budget over days). A single window cannot do both: a short window catches fast burns but misses slow ones, and a long window catches slow burns but reacts too slowly to fast ones.

Here is the Prometheus alert configuration that implements the standard four-alert pattern:

# slo-alerts.yaml
groups:
  - name: slo_burn_rate
    rules:
      # Fast burn: 14.4x burn rate sustained over 1h and 5m
      # Exhausts 28-day budget in 2 days
      - alert: SLOBurnRateFast
        expr: |
          (
            job:http_request_errors:rate1h{job="api"} / job:http_requests:rate1h{job="api"} > (14.4 * 0.005)
          )
          and
          (
            job:http_request_errors:rate5m{job="api"} / job:http_requests:rate5m{job="api"} > (14.4 * 0.005)
          )
        for: 2m
        labels:
          severity: critical
          slo: availability
        annotations:
          summary: "Fast error budget burn on {{ $labels.job }}"
          description: "Burn rate >14.4x. At this rate, 28-day error budget exhausted in ~2 days."

      # Slow burn: 6x burn rate sustained over 6h and 30m
      # Exhausts 28-day budget in ~5 days
      - alert: SLOBurnRateSlow
        expr: |
          (
            job:http_request_errors:rate6h{job="api"} / job:http_requests:rate6h{job="api"} > (6 * 0.005)
          )
          and
          (
            job:http_request_errors:rate30m{job="api"} / job:http_requests:rate30m{job="api"} > (6 * 0.005)
          )
        for: 15m
        labels:
          severity: warning
          slo: availability
        annotations:
          summary: "Slow error budget burn on {{ $labels.job }}"
          description: "Burn rate >6x. At this rate, 28-day error budget exhausted in ~5 days."

The recording rules that back those alert expressions:

# recording-rules.yaml
groups:
  - name: slo_recording
    rules:
      - record: job:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job)

      - record: job:http_request_errors:rate5m
        expr: sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)

      - record: job:http_requests:rate30m
        expr: sum(rate(http_requests_total[30m])) by (job)

      - record: job:http_request_errors:rate30m
        expr: sum(rate(http_requests_total{status=~"5.."}[30m])) by (job)

      - record: job:http_requests:rate1h
        expr: sum(rate(http_requests_total[1h])) by (job)

      - record: job:http_request_errors:rate1h
        expr: sum(rate(http_requests_total{status=~"5.."}[1h])) by (job)

      - record: job:http_requests:rate6h
        expr: sum(rate(http_requests_total[6h])) by (job)

      - record: job:http_request_errors:rate6h
        expr: sum(rate(http_requests_total{status=~"5.."}[6h])) by (job)

The two-window requirement (for example, both the 1h and 5m windows must exceed the threshold) reduces false positives. A single transient spike will not trigger the fast burn alert because the short window will reset before the long window catches up.

---

Error Budget Dashboard in TypeScript

If you are querying Prometheus programmatically to build a dashboard or feed an error budget policy system, here is a TypeScript client that calculates current budget consumption:

import axios from "axios";

interface ErrorBudgetStatus {
  sloTarget: number;
  windowDays: number;
  totalRequests: number;
  errorRequests: number;
  errorBudgetRequests: number;
  consumedRequests: number;
  remainingPercent: number;
  burnRate: number;
}

async function queryPrometheus(
  prometheusUrl: string,
  query: string,
  time?: string
): Promise<number> {
  const params: Record<string, string> = { query };
  if (time) params.time = time;

  const response = await axios.get(`${prometheusUrl}/api/v1/query`, { params });
  const result = response.data.data.result;

  if (result.length === 0) return 0;
  return parseFloat(result[0].value[1]);
}

async function getErrorBudgetStatus(
  prometheusUrl: string,
  job: string,
  sloTarget: number,
  windowDays: number
): Promise<ErrorBudgetStatus> {
  const windowSeconds = windowDays * 24 * 60 * 60;

  const [totalRequests, errorRequests] = await Promise.all([
    queryPrometheus(
      prometheusUrl,
      `sum(increase(http_requests_total{job="${job}"}[${windowSeconds}s]))`
    ),
    queryPrometheus(
      prometheusUrl,
      `sum(increase(http_requests_total{job="${job}",status=~"5.."}[${windowSeconds}s]))`
    ),
  ]);

  const errorBudgetRequests = totalRequests * (1 - sloTarget);
  const consumedRequests = errorRequests;
  const remainingRequests = errorBudgetRequests - consumedRequests;
  const remainingPercent = Math.max(
    0,
    (remainingRequests / errorBudgetRequests) * 100
  );

  // burn rate: how fast budget is consumed relative to the window
  // 1.0 = on track to exactly exhaust budget at window end
  const burnRate =
    errorBudgetRequests > 0
      ? consumedRequests / errorBudgetRequests / (1 / windowDays)
      : 0;

  return {
    sloTarget,
    windowDays,
    totalRequests,
    errorRequests,
    errorBudgetRequests,
    consumedRequests,
    remainingPercent,
    burnRate,
  };
}

// Usage
const status = await getErrorBudgetStatus(
  "http://prometheus:9090",
  "api",
  0.995, // 99.5% SLO
  28     // 28-day window
);

console.log(`Error budget remaining: ${status.remainingPercent.toFixed(1)}%`);
console.log(`Current burn rate: ${status.burnRate.toFixed(2)}x`);

---

Error Budget Policies

An error budget policy defines what happens when the budget is consumed at different levels. Without a policy, the error budget is just a number on a dashboard. With a policy, it becomes a forcing function for engineering decisions.

A practical three-tier policy:

Budget above 50%: Normal operation. Feature work and reliability investments proceed at standard ratio.

Budget between 20% and 50%: Amber state. No new feature deployments that touch the critical path. Engineering team reviews reliability work backlog in the next sprint. All incidents from the past two weeks are reviewed for systemic causes.

Budget below 20% or exhausted: Red state. Feature work stops on the affected service. All capacity goes to reliability improvements and postmortem action items. No deploys except rollbacks and hotfixes. This state is escalated to engineering leadership.

The policy creates a feedback loop that prioritizes reliability work without requiring a dedicated SRE team. When features ship reliably, the team has full budget to ship features. When reliability degrades, the system automatically redirects engineering attention.

This is also where SLO-based prioritization of reliability work pays off. Instead of arguing about which bug is “important enough to fix,” the team asks: “is this bug contributing to error budget burn?” If it is, it gets fixed before the next feature lands. If it is not, it goes into the backlog with appropriate priority.

---

Production Considerations

Multi-service SLOs vs. aggregated SLOs. If your API has 50 endpoints with very different reliability profiles, a single aggregated SLO will mask which endpoints are actually degraded. Define SLOs per critical user journey (checkout, authentication, search) rather than per service or per endpoint. Journey-level SLOs map directly to user impact and are easier to explain to non-engineering stakeholders.

Latency SLOs require percentile selection. A latency SLO like “99% of requests complete in under 500ms” is different from a p99 latency threshold. The SLO form expresses it as a ratio: 99% of requests fall under the threshold. At high traffic volumes, 1% of requests could still represent thousands of users per minute. For consumer-facing services, p99 is usually the right place to set the SLO. For internal services, p95 is often sufficient.

Rolling windows vs. calendar windows. A 28-day rolling window (also called a trailing window) gives you continuous, day-by-day visibility into budget consumption. Calendar-based windows (month boundaries) create perverse incentives: teams are tempted to burn budget late in the month and recover early in the next. Rolling windows avoid this. Most Prometheus-based setups use rolling windows because that is how rate() and increase() work natively.

The denominator problem. SLI arithmetic breaks when traffic is very low. If your service handles 10 requests during a 5-minute window and 2 of them fail, your error rate is 20%, which will fire every burn-rate alert you have. Add a minimum traffic gate to your alert expressions:

expr: |
  (
    job:http_request_errors:rate1h{job="api"} / job:http_requests:rate1h{job="api"} > (14.4 * 0.005)
  )
  and
  (
    job:http_requests:rate1h{job="api"} > 1  # at least 1 req/sec
  )

Dependency-induced budget burns. When a downstream dependency fails (a payment provider, a third-party API), your SLO burns even though the fault is external. You have two options: exclude errors caused by dependency failures from your SLI (requires propagating a specific error code or label through your stack), or accept that dependency reliability is part of your service reliability story. The first option is technically cleaner but operationally complex. The second is simpler but means you share budget with your dependencies.

---

The Tight vs. Loose SLO Tradeoff

A 99.99% SLO gives customers a strong reliability guarantee and signals that you take uptime seriously. It also means you have about 2.5 hours of total error budget per year. A single slow deployment, a bad database migration, or a region-level cloud incident can burn that entire budget in one event.

Teams that chase very tight SLOs often end up over-investing in redundancy and over-engineering deployments. They spend more time managing SLO compliance than building product. The budget is so tight that normal engineering activities (deployments, migrations, experiments) all require safety theater to avoid burning it.

A 99.5% SLO gives you about 3.4 hours per month. Deployments can fail and roll back. Incidents can last 15-20 minutes. The on-call engineer can investigate before escalating. Most services that are not payment processors, authentication systems, or safety-critical infrastructure should be in the 99.5% to 99.9% range.

The right question is not “what SLO can we technically achieve?” but “what SLO would, if violated, actually cause customers to leave?” Start there and work backwards.

---

There is a reason SRE teams track error budgets rather than uptime percentages: an uptime number tells you what happened, while an error budget tells you what you can still afford to do. That framing turns reliability from a post-incident review item into an ongoing constraint that shapes every engineering decision. Build the budget calculation, wire up the burn-rate alerts, and then write the policy that says what happens when the number hits zero. The policy is the part most teams skip, and it is the part that makes everything else matter.