How Prometheus Works Internally: Pull-Based Scraping, the TSDB Storage Engine, and PromQL Query Execution

Prometheus is not a general-purpose metrics store that happened to gain popularity. Every architectural decision reflects a deliberate philosophy: scrape targets at known intervals, store time series in a purpose-built local TSDB, and evaluate queries against that store using a functional query language. Understanding the internals helps you size deployments correctly, avoid cardinality explosions that crash the server, and decide when Prometheus alone is insufficient for your retention or scale requirements.

The Pull Model and Why It Matters

Most monitoring systems push metrics to a central collector. Prometheus inverts this. The server scrapes each target at a configured interval, typically 15 or 30 seconds, by issuing an HTTP GET to the target’s /metrics endpoint. Targets expose metrics in the Prometheus text format, and the server ingests the response.

The pull model has concrete operational implications. The Prometheus server controls the scrape rate, so a misbehaving target cannot flood the ingestor with data. Scrape failures are immediately visible as gaps in the time series, which surfaces network and process issues directly in the metrics. Because targets expose their state on demand, you do not need sidecar agents or forwarders on every host.

The downside is that targets must be network-reachable from the Prometheus server. Push-based systems, like the Pushgateway (intended for batch jobs that exit before a scrape can occur), exist precisely for cases where the pull model breaks down. The Pushgateway is explicitly not intended as a general aggregation point; it lacks the per-scrape metadata that makes the pull model useful.

Service Discovery Integration

Prometheus integrates with Kubernetes, Consul, EC2, GCE, Azure, DNS, and static file-based discovery. The discovery system maintains a live set of scrape targets, handling churn as pods restart or services deregister.

A typical Kubernetes scrape config looks like this:

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: 'true'
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        target_label: __address__
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: '$1:$2'
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

The relabel_configs pipeline rewrites target metadata before the first scrape. Labels prefixed with __ are internal and stripped before storage. This is where you add cluster, environment, or region labels that make federation and multi-tenancy workable. Every label you add here becomes part of every time series the target produces, which is a direct multiplier on cardinality.

The TSDB Storage Engine

Prometheus uses a custom TSDB introduced in version 2.0. The design priorities are high write throughput for sequential time series appends, fast range scans for recent data, and efficient compression.

Data Model

Every stored metric is a time series identified by a unique combination of a metric name and a set of key-value label pairs. Internally, each unique label set is assigned a numeric series ID. The fundamental storage unit is a 64-bit timestamp in milliseconds paired with a 64-bit float value.

// How Prometheus represents a series identifier internally
interface Labels {
  [key: string]: string; // e.g., { __name__: "http_requests_total", method: "GET", status: "200" }
}

interface Sample {
  timestamp: number; // Unix milliseconds
  value: number;     // float64
}

interface TimeSeries {
  labels: Labels;
  samples: Sample[];
}

Head Block: The Mutable Window

The TSDB is divided into blocks. The head block holds the most recent data, currently covering a configurable window (default: two hours). It lives primarily in memory and is the target for all incoming writes.

Inside the head block, each active series maintains a chunk of samples encoded in-memory using Gorilla-style XOR compression. The first timestamp is stored as a delta from the block’s minimum time. Subsequent timestamps store the delta-of-delta. Values use XOR encoding against the previous value, exploiting the fact that adjacent samples in a time series tend to be close to each other numerically. This yields roughly 1.37 bytes per sample in practice, compared to 16 bytes for naive storage.

// Simplified XOR value encoding (Gorilla compression)
function encodeXorValue(prev: number, curr: number): number {
  const xored = prev ^ curr; // XOR as bit patterns via DataView
  if (xored === 0) return 0;  // identical: store single control bit 0
  // Otherwise store leading/trailing zero counts + meaningful bits
  return xored;
}

When the head block’s time window is exhausted, it is flushed to disk as a persistent block and a new head block is opened. Block files are immutable once written.

Write-Ahead Log for Crash Recovery

Every sample appended to the head block is first written to a WAL on disk. The WAL uses fixed-size segment files (default: 128 MB each). On startup after a crash, Prometheus replays the WAL to reconstruct the in-memory head block.

The WAL records three types of entries: series records (new label sets), sample records (timestamp-value pairs for existing series), and tombstone records (for deletions). Series and sample records are separated so that label metadata does not repeat with every write.

One non-obvious implication: a large head block means a large WAL replay on startup. If you store 12 hours in the head block instead of the default 2, cold starts after a crash take proportionally longer. Default settings exist for a reason.

Block Compaction

On-disk blocks cover non-overlapping time ranges. The compactor runs in the background and merges adjacent small blocks into larger ones. The default compaction schedule produces blocks of progressively longer duration: two hours, then eight hours, then one day, then three days, capped at the configured retention window (default: 15 days).

Compaction applies two operations beyond merging: it removes samples covered by tombstones from deletion requests, and it removes series that have no samples in the merged block’s window. The resulting blocks are stored under data/ in directories named by a ULID that encodes the creation time.

data/
  01HXYZ.../          # block directory (ULID)
    chunks/
      000001           # raw chunk data, up to 512 MB per file
    index              # inverted posting list: label -> series IDs
    meta.json          # block metadata: minTime, maxTime, stats
    tombstones         # pending deletions
  wal/
    00000001           # WAL segments
    00000002

The index file is the key read-path structure. It contains an inverted posting list that maps each label-value pair to the set of series IDs that carry that label. A query like http_requests_total{job="api", status=~"5.."} intersects the posting lists for __name__="http_requests_total", job="api", and all status values matching the regex, producing the set of matching series IDs. Chunks for those IDs are then fetched and decoded.

PromQL Query Execution

PromQL is a functional query language that operates on sets of time series. Understanding the execution pipeline explains both its power and its cost model.

Parsing to Abstract Syntax Tree

The query engine first parses PromQL text into an AST. A query like:

rate(http_requests_total{job="api"}[5m]) > 0.1

produces an AST with a binary comparison node at the root, a rate() call on the left, and a scalar on the right. The rate() call wraps a vector selector with a 5-minute range.

Evaluation Model

Prometheus evaluates queries at a set of evaluation timestamps, either a single instant for instant queries or a sequence of timestamps at step intervals for range queries. For each evaluation timestamp t, the engine:

1. Selects the lookback window for each range vector selector. A [5m] range selector looks back from t-5m to t.
2. Fetches matching chunks from the TSDB for all series that match the label selector.
3. Applies functions bottom-up through the AST. rate() computes per-second rate from the first and last samples in the window, adjusting for counter resets. sum() aggregates across series.
4. Applies binary operators between results.

// Conceptual evaluation of rate() over a range vector
function rate(samples: Sample[], rangeSeconds: number): number | null {
  if (samples.length < 2) return null;

  const first = samples[0];
  const last = samples[samples.length - 1];
  const durationSeconds = (last.timestamp - first.timestamp) / 1000;

  if (durationSeconds === 0) return null;

  let delta = last.value - first.value;
  // Counter reset detection: if value decreased, assume reset to 0
  if (delta < 0) {
    delta = last.value + (first.value - last.value);
  }

  // Extrapolate to fill the full range window
  const extrapolationFactor = rangeSeconds / durationSeconds;
  return (delta / durationSeconds) * extrapolationFactor;
}

Range queries multiply this cost by the number of steps. A 24-hour range with a 15-second step has 5760 evaluation points. If the query touches 1000 series and each series has 300 samples per 5-minute window, you are decoding roughly 1.7 billion sample reads. This is where cardinality and range selection interact to produce out-of-memory kills.

Recording Rules for Pre-Aggregation

Recording rules evaluate a PromQL expression on a schedule and write the results back as new time series. They are the primary mechanism for making expensive aggregations affordable at query time.

groups:
  - name: api_aggregations
    interval: 30s
    rules:
      - record: job:http_requests_total:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))
      - record: job:http_request_duration_seconds:p99
        expr: |
          histogram_quantile(0.99,
            sum by (job, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )

The naming convention level:metric:operations is idiomatic. The aggregated series are stored in the TSDB like any other scraped metric. Dashboard queries hit the pre-aggregated series instead of raw data, dropping query time from seconds to milliseconds.

The Alerting Pipeline

Prometheus evaluates alerting rules against the TSDB. When a rule’s expression produces a non-empty result vector, the corresponding alert transitions to pending. If it remains pending for longer than the configured for duration, it fires. Firing alerts are forwarded to Alertmanager.

groups:
  - name: latency
    rules:
      - alert: HighP99Latency
        expr: job:http_request_duration_seconds:p99 > 0.5
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "P99 latency above 500ms for {{ $labels.job }}"
          description: "Current value: {{ $value | humanizeDuration }}"

Alertmanager handles deduplication, grouping, silencing, and routing. A single alert firing on 50 Kubernetes pods for the same root cause should produce one notification, not 50. Alertmanager groups alerts by configurable label sets, applies inhibition rules (suppress lower-severity alerts when a higher-severity one is active), and routes to receivers: PagerDuty, Slack, email, webhook.

The Prometheus-to-Alertmanager communication uses a push model. Prometheus sends alerts via HTTP POST to all configured Alertmanager instances. Alertmanager instances form a gossip cluster using the Memberlist protocol to deduplicate notifications when running in HA mode.

Federation and Multi-Cluster Setups

A single Prometheus instance does not scale horizontally. For multi-cluster or multi-datacenter setups, federation lets a top-level Prometheus scrape aggregated metrics from subordinate instances.

The subordinate Prometheus servers each scrape their own targets. The top-level server scrapes each subordinate’s /federate endpoint, selecting only pre-aggregated recording rule outputs rather than raw high-cardinality series. This keeps the cardinality of the top-level instance manageable.

# Top-level Prometheus scraping subordinate instances
scrape_configs:
  - job_name: 'federate'
    honor_labels: true
    metrics_path: '/federate'
    params:
      match[]:
        - '{__name__=~"job:.*"}'  # Only recording rule outputs
    static_configs:
      - targets:
          - 'prometheus-us-east:9090'
          - 'prometheus-eu-west:9090'

The honor_labels: true setting preserves the original job and instance labels from the source server rather than overwriting them with the federate scrape job.

For more demanding scenarios, Thanos or Grafana Mimir provide horizontally scalable query layers that read from object storage. The Prometheus instances themselves remain single-node, writing their blocks to object storage, while the query tier fans out reads across historical data.

Production Considerations

Cardinality Management

Cardinality is the number of unique time series, computed as the product of unique values across all label dimensions. A metric like http_requests_total with job (10 values), method (5), status (20), and endpoint (500) has 50,000 series before any instance-level labels are added. Add a per-pod instance label with 100 pods and you have 5 million series for a single metric.

High cardinality directly increases memory usage in the head block and WAL replay time. The practical ceiling for a single Prometheus instance with 16 GB RAM is roughly 5-10 million active series, depending on scrape interval and retention.

Use prometheus_tsdb_head_series to monitor active series count. Drop cardinality at the source: remove high-cardinality labels like request IDs or user IDs from metrics before they reach the TSDB. Use metric_relabel_configs to drop series or replace label values before storage.

scrape_configs:
  - job_name: 'api'
    static_configs:
      - targets: ['api:8080']
    metric_relabel_configs:
      # Drop debug-level internal metrics entirely
      - source_labels: [__name__]
        regex: 'go_memstats_.*'
        action: drop
      # Hash user_id to a fixed set of buckets instead of storing raw values
      - source_labels: [user_id]
        target_label: user_id
        regex: '(.+)'
        replacement: 'redacted'

Retention Tuning

The default 15-day local retention is adequate for operational dashboards. Retention is enforced by both time (delete blocks older than --storage.tsdb.retention.time) and size (delete oldest blocks when storage exceeds --storage.tsdb.retention.size). Size-based retention prevents disk saturation from cardinality spikes but can unexpectedly delete recent data if cardinality surges.

For long-term storage, remote write ships samples to an external system as they are ingested. The remote write path buffers samples in an in-memory queue and retries on failure. Common backends are Grafana Mimir, VictoriaMetrics, Thanos Receiver, Cortex, and cloud-managed options. The local TSDB continues operating independently; remote write is additive.

remote_write:
  - url: 'https://mimir.internal/api/v1/push'
    queue_config:
      max_samples_per_send: 5000
      max_shards: 30
      capacity: 10000
    write_relabel_configs:
      # Only send recording rule outputs and critical raw metrics to long-term storage
      - source_labels: [__name__]
        regex: 'job:.*|up|kube_pod_status_phase'
        action: keep

Setting max_shards too low causes the remote write queue to back up under load, eventually dropping samples if the buffer fills. Setting it too high floods the remote endpoint. Start with the default and increase based on observed queue depth metrics (prometheus_remote_storage_queue_highest_sent_timestamp_seconds).

Sizing and Operational Checklist

A starting heuristic for memory sizing: 3 KB per active series in the head block. For 1 million active series with a 2-hour head block and a 15-second scrape interval, expect roughly 3 GB just for series storage, before WAL overhead and query execution buffers.

Run Prometheus with --storage.tsdb.wal-compression enabled. WAL compression reduces WAL size by 30-50% at negligible CPU cost. Enable --web.enable-lifecycle to allow hot-reloads of configuration without restart.

Tradeoffs Table

Dimension	Prometheus	Datadog	Grafana Mimir	VictoriaMetrics	InfluxDB
Architecture	Single-node pull	SaaS hosted	Horizontally scalable, Prometheus-compatible	Single-node or cluster, drop-in Prometheus replacement	Single-node or clustered (Enterprise)
Storage model	Custom TSDB, local disk	Proprietary, cloud-managed	Object storage (S3/GCS) + ingesters	Custom LSM-based TSDB, more efficient than Prometheus	TSM engine (LSM variant)
Retention	15 days default, extendable via remote write	Configurable per plan (up to 15 months)	Unlimited via object storage	Configurable, efficient long-term local storage	Configurable, tiered in Enterprise
Query language	PromQL	MetricsQL (subset + extensions), DQL	PromQL (compatible)	MetricsQL (superset of PromQL)	Flux (powerful, steep learning curve)
Cardinality limits	~5-10M series per instance (RAM-bound)	High, managed by vendor	Scales horizontally with shard count	Higher cardinality per RAM than Prometheus	Moderate, improves with clustering
Operational cost	Low (single binary)	Zero ops, but expensive at scale	High (distributed system)	Low to medium	Medium
HA model	None natively; Thanos/Mimir for HA	Built-in	Built-in multi-replica ingesters	Cluster mode or active/active with replication	Replication in Enterprise
Agent support	Pushgateway for batch; pull-first	Agent-based push; pull via integrations	Prometheus-compatible scraping	Push and pull; Prometheus-compatible	Telegraf agent for push
Best fit	Kubernetes-native operational monitoring	Managed observability with broad integrations	Large-scale Prometheus with long retention	Cost-efficient Prometheus replacement with extra headroom	Mixed workloads needing rich write APIs

Prometheus remains the right default for Kubernetes-based infrastructure because it integrates directly with the Kubernetes API for service discovery, its operator ecosystem is mature, and the operational surface area is a single stateless binary with a local disk. Its hard limit is scale: one server, one disk, bounded RAM. When that constraint bites, the migration path to Mimir or VictoriaMetrics preserves PromQL compatibility, so the investment in recording rules, dashboards, and alert expressions carries over.