Prometheus and Grafana for Startups: Setting Up Production Monitoring and Alerting from Scratch

At some point in every startup’s life, someone gets paged because a customer reported an outage that had been going on for 45 minutes. The postmortem ends with “we need better monitoring,” which everyone agrees on, and then nothing happens for three months because nobody knows where to start.

This is that guide. Not a conceptual overview, but an actual setup you can run in production this week.

Why Prometheus

Prometheus uses a pull model: your metrics server scrapes targets on a schedule, rather than your applications pushing data to a centralized collector. This has real operational consequences.

When a service goes down, Prometheus notices immediately because the scrape fails. With a push-based system, a dead service just stops sending data, and you need a separate “heartbeat” mechanism to detect that silence. Pull-based scraping makes absence of data a first-class signal.

PromQL is worth learning. It is a functional query language built specifically for time-series data, and once you stop fighting it, you can express queries like “95th percentile latency over the last 5 minutes, broken down by route, only for HTTP 5xx responses” in a single line. That kind of slicing is awkward in SQL and impossible in most log-based systems without significant infrastructure.

The ecosystem is also the practical argument. Every open-source database, message broker, and runtime ships a Prometheus exporter. You will spend more time reading documentation than wiring up integrations.

The Minimal Viable Stack

Before reaching for managed solutions, understand what you are actually running. The core stack is three components.

Prometheus scrapes metrics and stores them locally in a custom time-series database. The storage format is optimized for append-only writes and range queries, not for joins or arbitrary filters. Keep this in mind when designing your metrics schema.

Grafana provides the dashboards. It queries Prometheus (and optionally other data sources) and renders panels. Grafana itself is stateless if you store dashboards as code.

Alertmanager handles alert routing. Prometheus evaluates alert rules and fires alerts to Alertmanager, which handles deduplication, grouping, silences, and routing to destinations like Slack or PagerDuty. Do not skip Alertmanager and try to send alerts directly from Prometheus to Slack. You will regret it at 3am when you get 400 identical notifications because a pod was restarting.

A minimal Docker Compose setup:

# docker-compose.yml
version: "3.8"
services:
  prometheus:
    image: prom/prometheus:v2.51.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./rules:/etc/prometheus/rules
      - prometheus_data:/prometheus
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
      - "--storage.tsdb.retention.time=30d"
      - "--web.enable-lifecycle"
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:10.4.2
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/dashboards:/var/lib/grafana/dashboards
      - ./grafana/provisioning:/etc/grafana/provisioning
    environment:
      GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD}
      GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH: /var/lib/grafana/dashboards/overview.json
    ports:
      - "3001:3000"

  alertmanager:
    image: prom/alertmanager:v0.27.0
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml
    ports:
      - "9093:9093"

volumes:
  prometheus_data:
  grafana_data:

Instrumenting a Node.js/TypeScript Application

Install prom-client:

npm install prom-client

The right approach is to create a dedicated metrics module that initializes a registry and exports your instruments. Do not scatter new Counter() calls across your codebase.

// src/metrics/registry.ts
import { Registry, collectDefaultMetrics } from "prom-client";

export const registry = new Registry();

// Collect Node.js runtime metrics: event loop lag, heap usage, GC stats
collectDefaultMetrics({ register: registry, prefix: "node_" });

// src/metrics/http.ts
import { Counter, Histogram } from "prom-client";
import { registry } from "./registry";

export const httpRequestsTotal = new Counter({
  name: "http_requests_total",
  help: "Total number of HTTP requests",
  labelNames: ["method", "route", "status_code"],
  registers: [registry],
});

export const httpRequestDurationSeconds = new Histogram({
  name: "http_request_duration_seconds",
  help: "HTTP request duration in seconds",
  labelNames: ["method", "route", "status_code"],
  // Buckets tuned for a web API: 50ms, 100ms, 250ms, 500ms, 1s, 2.5s, 5s
  buckets: [0.05, 0.1, 0.25, 0.5, 1, 2.5, 5],
  registers: [registry],
});

// src/metrics/business.ts
import { Gauge, Counter } from "prom-client";
import { registry } from "./registry";

// Business-level metrics matter more than infrastructure metrics
export const activeCheckouts = new Gauge({
  name: "app_active_checkouts",
  help: "Number of checkout sessions currently in progress",
  registers: [registry],
});

export const paymentProcessedTotal = new Counter({
  name: "app_payments_processed_total",
  help: "Total payments processed",
  labelNames: ["currency", "provider", "result"],
  registers: [registry],
});

Wire this into an Express or Hono middleware:

// src/middleware/metrics.ts
import { Request, Response, NextFunction } from "express";
import { httpRequestsTotal, httpRequestDurationSeconds } from "../metrics/http";

export function metricsMiddleware(
  req: Request,
  res: Response,
  next: NextFunction
): void {
  const start = process.hrtime.bigint();

  // Normalize route to avoid cardinality explosion from path params
  // /users/123 and /users/456 should both be /users/:id
  const route = req.route?.path ?? req.path;

  res.on("finish", () => {
    const durationSeconds =
      Number(process.hrtime.bigint() - start) / 1_000_000_000;

    const labels = {
      method: req.method,
      route,
      status_code: String(res.statusCode),
    };

    httpRequestsTotal.inc(labels);
    httpRequestDurationSeconds.observe(labels, durationSeconds);
  });

  next();
}

// src/routes/metrics.ts
import { Router } from "express";
import { registry } from "../metrics/registry";

const router = Router();

router.get("/metrics", async (_req, res) => {
  res.set("Content-Type", registry.contentType);
  res.send(await registry.metrics());
});

export { router as metricsRouter };

One critical detail: keep label cardinality under control. If you add user_id as a label, Prometheus creates a new time series per user. At 100,000 users, you have 100,000 time series for a single metric. Use labels for categorical dimensions with bounded cardinality: method, route template, status code, region, deployment tier.

The Four Golden Signals

Google’s Site Reliability Engineering book defines four signals that, if monitored, tell you almost everything you need to know about service health. They are not comprehensive, but they are the right starting point.

Latency: How long requests take, and specifically how long failed requests take separately from successful ones. A spike in error rate with stable latency is different from a spike in latency with stable success rate.

# 95th percentile latency over 5m, by route
histogram_quantile(
  0.95,
  sum by (route, le) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

Traffic: Request rate. This is your baseline for capacity planning and for computing error rate as a ratio rather than an absolute count.

# Requests per second over 1m window
sum by (route) (rate(http_requests_total[1m]))

Errors: The rate of failed requests. Alert on error ratio, not raw count. A service handling 10 req/s with 1 error/s is in a different state than one handling 1000 req/s with 1 error/s.

# Error ratio over 5m
sum(rate(http_requests_total{status_code=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))

Saturation: How close the service is to its resource limits. For Node.js, this is event loop lag and heap utilization. For a database, it is connection pool exhaustion.

# Node.js event loop lag (prom-client default metric)
node_eventloop_lag_seconds

# Heap utilization percentage
node_heap_size_used_bytes / node_heap_size_total_bytes

Building Useful Grafana Dashboards

The most common mistake is building dashboards that look impressive in demos but tell you nothing during incidents. Avoid panels that show total requests since deployment, or uptime percentages that are always 99.9%.

A useful service dashboard has this structure:

1. Top row: SLO status - Is the service meeting its commitments right now? Red/green status panels with thresholds. Nothing more.
2. Second row: The four golden signals - Four panels: latency (p50, p95, p99), error rate, request rate, saturation. All over a 1-hour window by default.
3. Third row: Breakdown by route - Same signals broken down per endpoint. This is where you find which specific operation is degraded.
4. Bottom: Runtime internals - GC pressure, heap, event loop lag. Useful for diagnosis after you already know something is wrong.

Store dashboards as JSON and provision them from files. This makes them version-controlled and reproducible.

# grafana/provisioning/dashboards/default.yaml
apiVersion: 1
providers:
  - name: default
    folder: ""
    type: file
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

Alerting That Does Not Wake You Up Unnecessarily

Noisy alerts destroy trust in your monitoring system faster than anything else. When engineers start ignoring pages because they are usually false positives, you have lost the entire benefit of the investment.

The core principle: alert on symptoms, not causes. Alert on “the error rate is above 5% for 5 minutes,” not on “CPU is above 80%.” High CPU might be fine. A sustained error rate is never fine.

A sample Alertmanager configuration with Slack and PagerDuty routing:

# alertmanager.yml
global:
  slack_api_url: "${SLACK_WEBHOOK_URL}"
  pagerduty_url: "https://events.pagerduty.com/v2/enqueue"

route:
  receiver: slack-default
  group_by: ["alertname", "service"]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - match:
        severity: critical
      receiver: pagerduty-critical
      continue: true
    - match:
        severity: critical
      receiver: slack-critical
    - match:
        severity: warning
      receiver: slack-warnings

receivers:
  - name: slack-default
    slack_configs:
      - channel: "#alerts-dev"
        title: '{{ template "slack.title" . }}'
        text: '{{ template "slack.text" . }}'

  - name: slack-critical
    slack_configs:
      - channel: "#alerts-critical"
        title: '{{ template "slack.title" . }}'
        text: '{{ template "slack.text" . }}'

  - name: slack-warnings
    slack_configs:
      - channel: "#alerts-warnings"
        title: '{{ template "slack.title" . }}'
        text: '{{ template "slack.text" . }}'

  - name: pagerduty-critical
    pagerduty_configs:
      - routing_key: "${PAGERDUTY_ROUTING_KEY}"
        severity: critical

inhibit_rules:
  # If a service is completely down, suppress warning-level alerts for it
  - source_match:
      severity: critical
    target_match:
      severity: warning
    equal: ["service"]

Alert rules live in Prometheus, not Alertmanager:

# rules/api.yml
groups:
  - name: api.rules
    rules:
      - alert: HighErrorRate
        expr: |
          (
            sum(rate(http_requests_total{status_code=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) > 0.05
        for: 5m
        labels:
          severity: critical
          service: api
        annotations:
          summary: "API error rate above 5%"
          description: "Error rate is {{ $value | humanizePercentage }} over the last 5 minutes."

      - alert: HighLatency
        expr: |
          histogram_quantile(
            0.95,
            sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
          ) > 2
        for: 10m
        labels:
          severity: warning
          service: api
        annotations:
          summary: "API p95 latency above 2 seconds"
          description: "p95 latency is {{ $value | humanizeDuration }} over the last 5 minutes."

      - alert: ServiceDown
        expr: up{job="api"} == 0
        for: 1m
        labels:
          severity: critical
          service: api
        annotations:
          summary: "API is unreachable"
          description: "Prometheus cannot scrape the API metrics endpoint."

The for clause is important. Without it, a single bad scrape fires the alert. With for: 5m, the condition must be true for 5 consecutive minutes before the alert fires. Use shorter windows for critical issues, longer windows for degraded-but-not-broken states.

Storage: Local vs Remote Write

Local storage in Prometheus is fast and simple, but it has two constraints: disk size limits total retention, and there is no high availability. If the Prometheus pod restarts, you lose data from the scrape interval gap.

For startups, the practical options are:

Local storage only: Reasonable for teams that are not yet making decisions based on historical trends. Set --storage.tsdb.retention.time=30d and accept the constraints. This handles most monitoring needs.

Remote write to Grafana Cloud: Prometheus supports remote_write natively. You run Prometheus locally for real-time alerting, and asynchronously ship samples to Grafana Cloud for long-term storage and HA. The free tier handles roughly 10,000 active series, which is generous for most early-stage services.

# prometheus.yml addition for remote write
remote_write:
  - url: "https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push"
    basic_auth:
      username: "${GRAFANA_CLOUD_USER}"
      password: "${GRAFANA_CLOUD_API_KEY}"
    queue_config:
      max_samples_per_send: 1000
      max_shards: 2

Thanos or Mimir (self-hosted): For teams that need multi-cluster aggregation or very long retention (years) without vendor lock-in. Thanos adds a sidecar to each Prometheus instance that uploads blocks to object storage (S3, GCS). Mimir is the fully distributed version. Both are operationally non-trivial. Do not reach for these until you have outgrown local storage and have someone who can operate them.

Cost Profile and the Self-Hosted vs Managed Decision

Approach	Monthly Cost (est.)	Operational Burden	Retention	Fit
Self-hosted (local only)	~$5-20 (compute)	Low	30-90 days	Early stage, 1-5 services
Prometheus + Grafana Cloud free tier	~$0-20	Low	13 months (Grafana side)	Early stage, up to 10k series
Grafana Cloud paid	~$8 per 1k series/mo	Very low	13 months	Growing team, 10k-100k series
Datadog	~$15-23 per host/mo + APM	Very low	15 months	Series A+, needs APM depth
New Relic	~$0.30 per GB ingest	Very low	13 months	Variable cardinality workloads
Self-hosted Thanos/Mimir	~$50-200 (compute + storage)	High	Unlimited	50+ services, dedicated SRE

The cost crossover between self-hosted Prometheus and Datadog happens around 20-30 hosts. Below that, Datadog is genuinely competitive when you factor in engineering time spent operating the stack.

The argument for Datadog at startup scale is not just cost, it is scope: APM traces, logs, and metrics in one product means one query interface, one set of dashboards, and one bill. If you have a senior engineer who wants to own the observability stack and finds this kind of work interesting, self-hosted Prometheus is defensible. If you do not, pay for managed.

When to Graduate

The Prometheus-plus-Grafana stack scales further than most startups think. You should consider adding complexity when:

• You have more than 50 services and need a global view across clusters. Thanos or Mimir becomes worth the operational cost.
• You need distributed tracing to understand request flows across services. Add Jaeger or Tempo. Prometheus does not do traces.
• Your team is spending more than a few hours per month on monitoring infrastructure. Managed becomes cheaper even at Datadog prices.
• You need log correlation alongside metrics. Add Loki to the Grafana stack, or switch to a product that unifies all three signals.

Start with the minimal stack. Add components only when you can articulate exactly what problem the addition solves. The most expensive monitoring system is one that generates so much noise that engineers stop trusting it.

The four golden signals, instrumented properly and alerted on conservatively, will tell you when your production system is broken before your customers do. That is the bar to clear first.