How Prometheus Works Internally: TSDB Storage, PromQL Execution, and the Pull-Based Architecture Behind Modern Monitoring

Prometheus has a reputation for being simple. One binary, one config file, and you have metrics. That simplicity is real, but it conceals a surprisingly intricate engine underneath. The pull model is a deliberate design choice with real operational consequences. The TSDB is not a general-purpose database bolted on after the fact. PromQL is not SQL with a coat of paint. Each layer has opinions baked in, and understanding those opinions is what separates an operator who fights Prometheus from one who works with it.

This article walks through how each internal component actually works, from the moment a scrape fires to the moment a query returns a result.

The Pull Model: Why Prometheus Reaches Out

Most monitoring agents push metrics to a central collector. Prometheus inverts this. Every scrape target exposes an HTTP endpoint (typically /metrics) that Prometheus polls on a configurable interval, defaulting to 15 seconds. The server reaches out, parses the exposition format, and stores what it finds.

This is a deliberate tradeoff. The pull model makes service discovery the server’s responsibility, not the target’s. Targets do not need to know the Prometheus server’s address. They also do not need to buffer metrics during network partitions or Prometheus downtime. A push-based agent that cannot reach its backend must either buffer or drop. A pull-based target just keeps incrementing its counters until someone asks.

The downside is equally clear: Prometheus must be able to reach every target. Behind NAT, inside a private subnet, or in a short-lived batch job, this becomes a problem. Prometheus addresses the batch job case with the Pushgateway (a separate component targets can push to), but it is a concession, not a first-class pattern.

Service Discovery and Relabeling

Prometheus discovers targets through service discovery integrations: Kubernetes, EC2, Consul, DNS SRV records, and static configs. Each discovery mechanism returns a list of targets with labels attached. A Kubernetes SD result might look like this:

- targets: ["10.0.1.42:8080"]
  labels:
    __meta_kubernetes_pod_name: "api-server-7d9f8b-xkv2p"
    __meta_kubernetes_namespace: "production"
    __meta_kubernetes_pod_label_app: "api-server"

Labels prefixed with __meta_ are discovery metadata. They are not stored in the TSDB. Before a scrape fires, Prometheus runs a relabeling pipeline that transforms, filters, and renames these labels into the final label set that gets persisted with every metric.

Relabeling is a series of rules applied in order. Each rule has a source label, a regex, and an action. The replace action extracts and renames, keep and drop filter targets entirely, labelmap copies labels matching a pattern, and hashmod is used for horizontal sharding across multiple Prometheus instances.

relabel_configs:
  - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_pod_label_app]
    separator: "/"
    target_label: job
    regex: "(.*)"
    replacement: "$1"
  - source_labels: [__meta_kubernetes_pod_label_environment]
    regex: "staging"
    action: drop

The relabeling pipeline runs twice: once on discovery (target relabeling) and once after the scrape (metric relabeling). Metric relabeling can drop individual time series, rename labels on scraped metrics, or extract substrings from label values. This is also where high-cardinality time series get filtered before they hit storage.

TSDB: How Prometheus Stores Time Series

The Prometheus TSDB is a custom time-series database embedded directly in the binary. It is built around one key insight: recent data is read and written at much higher rates than old data, and old data is almost never mutated. The architecture follows from that.

The Head Block

Incoming scraped samples land in the head block, an in-memory structure that represents the most recent two hours (by default) of data. The head block is mutable. New samples are appended, and it is what queries hit for recent data.

The head block is organized around a series table. Each unique combination of label names and values (called a label set) maps to a time series. The series stores a chunk of compressed samples. Prometheus uses Gorilla-style XOR delta-delta compression, a scheme where the difference between timestamps is encoded as a delta of a delta, and float64 values are XOR’d against the previous value. Adjacent metric values are usually close to each other, so XOR produces many leading zeros, and the compressed chunk is tiny.

The series lookup in the head block uses an inverted index. Every label name-value pair (job="api-server", env="production") maps to a posting list: the set of series IDs that carry that label. A label matchers query intersects these posting lists. Finding all series matching {job="api-server", env="production"} is a set intersection on two posting lists, not a scan of all series.

Persistent Blocks

When the head block fills up (after roughly two hours), it is flushed to disk as a persistent block. A block is a directory with three files:

• chunks/: Raw compressed chunk data for all series in the block, stored in 512 MB segment files.
• index: An inverted index mapping labels to series, and series to their chunk locations.
• meta.json: Block metadata, including the time range covered, ULID identifier, and compaction statistics.

Each block covers a fixed time range and is immutable once written. Queries against historical data open the relevant blocks, use the index to locate matching series, and decompress only the chunks needed to satisfy the time range.

The ULID (Universally Unique Lexicographically Sortable Identifier) assigned to each block embeds a timestamp, which means blocks sort naturally by time. This is useful when listing available blocks and during compaction.

The Write-Ahead Log

Before any sample lands in the head block’s in-memory structures, it is written to the write-ahead log (WAL). The WAL is a sequence of segment files on disk, each capped at 128 MB. Every scrape result is appended to the WAL as a record before the in-memory state is updated.

On crash and restart, Prometheus replays the WAL to reconstruct the head block. This gives crash consistency without needing to flush the head block to a persistent block on every scrape. The WAL segments are deleted only after the head block data they cover has been compacted to a persistent block.

The WAL records three types of entries:

• Series records: Map series references (integer IDs) to their label sets. Written once per new series seen.
• Sample records: Batches of (series reference, timestamp, value) tuples. Written on every scrape.
• Tombstone records: Mark samples for deletion (used by the admin delete API).

WAL replay on startup scans all segment files in order. Series records re-establish the ID-to-labels mapping. Sample records re-insert values into the head block chunks. This replay is fast because the WAL is append-only and segment files are read sequentially.

Compaction and Retention

Left unchecked, persistent blocks would accumulate forever, and queries spanning long time ranges would need to open and scan dozens of small blocks. Compaction solves this by merging small blocks into larger ones.

Prometheus compacts in levels. Fresh blocks cover two hours. After compaction, blocks can cover 6 hours, then 24 hours, then 7 days (the actual thresholds depend on configuration). At each level, the compactor merges overlapping or adjacent blocks, deduplicates series that appear in multiple source blocks (which happens after outage recovery when the WAL replay produces overlapping chunks), and rewrites the index.

Compaction is a background goroutine that runs every minute. It checks for compactable blocks using a planner, executes the merge, writes the new block atomically (via directory rename), and then removes the source blocks.

Retention is enforced during compaction. Any block whose maximum timestamp is older than the configured retention period is deleted. The default is 15 days. Storage size can also bound retention: if --storage.tsdb.retention.size is set, Prometheus deletes the oldest blocks until total size falls below the limit.

PromQL Query Execution

PromQL is a functional expression language, not SQL. It operates on vectors of time series. Understanding how a query executes explains both its power and its quirks.

Parsing and Query AST

A PromQL query string is first parsed into an AST. Lexing handles the token stream: metric names, label matchers, duration literals, and operators. The parser converts tokens into an expression tree. The expression rate(http_requests_total{job="api"}[5m]) becomes an AST with a Call node (the rate function), whose argument is a MatrixSelector node for http_requests_total{job="api"} with a range of five minutes.

The AST is then type-checked. PromQL has three expression types: instant vector (a set of time series with one sample per series at a given timestamp), range vector (each series carries a range of samples), and scalar. Functions have type signatures. rate takes a range vector and returns an instant vector. Applying rate to an instant vector is a compile-time error.

Query Evaluation

The query engine walks the AST bottom-up. Leaf nodes are selectors: they hit the TSDB to retrieve the matching series. Interior nodes are operators or function calls evaluated against the output of their subtrees.

For an instant query at timestamp T, a VectorSelector for http_requests_total{job="api"} looks up the posting lists for __name__="http_requests_total" and job="api", intersects them to get a set of series IDs, and for each series fetches the sample closest to T within the configured lookback delta (default five minutes). If no sample exists within that window, the series is not included in the result.

A MatrixSelector for http_requests_total{job="api"}[5m] does the same label lookup but retrieves all samples in the range [T - 5m, T] for each series.

rate computes per-second average increase over the range. For a counter, it uses the first and last samples in the range, handles counter resets (detected when a sample value is less than the previous one), and divides by the duration in seconds.

Binary operators between two instant vectors require a label matching step. http_requests_total / http_request_duration_seconds matches series on all labels by default. The engine groups series from both sides, pairs them by label set, and applies the operation pair-wise. The on and ignoring keywords control which labels are used for matching. group_left and group_right handle many-to-one matching.

For range queries (step queries in Grafana parlance), the engine evaluates the expression at each step interval across the requested time range. Steps are independent evaluations, but the engine batches chunk fetches across steps to avoid repeated TSDB lookups.

Staleness Handling

Prometheus uses a special staleness marker, a NaN value with a specific bit pattern, to signal that a target has disappeared. When a scrape fails or a target is removed from service discovery, Prometheus appends a stale marker for every series that was being scraped from that target. Queries encountering a stale marker treat the series as having no samples at that timestamp, rather than carrying the last-seen value forward indefinitely.

Federation and Remote Write/Read

A single Prometheus instance is intentionally scoped to a single data center or cluster. It does not natively aggregate across multiple instances. Two mechanisms extend its reach.

Federation lets one Prometheus scrape metrics from another via the /federate endpoint. A global Prometheus can pull aggregated or pre-selected metrics from per-cluster instances. Federation uses the normal pull model, which means the global instance has all of Prometheus’s push-model limitations applied to Prometheus itself: it must reach the federation endpoints, and it operates at scrape intervals that are unsuitable for high-cardinality raw metrics.

Remote write streams samples to a remote backend as they are scraped, using a protocol-buffer-encoded batch over HTTP. The write-ahead log acts as a buffer. A remote write queue reads from the WAL and batches samples for delivery. If the remote endpoint is unavailable, the queue backs up against the WAL. Once WAL segments are old enough for compaction, undelivered samples are lost. Remote write introduces a secondary write path without removing local TSDB storage; the local TSDB continues writing in parallel.

Remote read allows queries to be forwarded to a remote backend. The engine sends a series selector and time range to the remote read endpoint and receives samples back, which it integrates with locally stored data before evaluation. Remote read is rarely used in modern setups because it is less efficient than having the remote system handle queries natively.

Production Considerations

Cardinality is the main resource lever. Each unique label set is a distinct time series. A label with 1,000 distinct values multiplied across 50 other dimensions creates cardinality that exhausts memory. The head block keeps all active series in RAM. The rule: label values should be bounded, not open-ended. No user IDs, request IDs, or trace IDs in label values.

Scrape interval versus evaluation interval. Recording rules evaluate on the evaluation interval, not per scrape. Setting evaluation intervals shorter than scrape intervals means rules evaluate against staleness windows and produce gaps. Keep them aligned.

WAL sizing and restart time. WAL replay on startup must process every segment covering the head block’s time range. Large head blocks (increased by raising --storage.tsdb.head-chunks-write-queue-size or --storage.tsdb.retention.time) mean longer restarts. Kubernetes rolling deployments with large WALs can stall for minutes. Monitor prometheus_tsdb_head_chunks and prometheus_tsdb_wal_size_bytes.

Block compaction and disk I/O. Compaction rewrites blocks entirely. On spinning disks, a 7-day compaction on a large dataset generates significant I/O. Separate compaction timing from query-heavy periods or run Prometheus on SSDs.

Out-of-order samples. By default, Prometheus rejects samples with timestamps earlier than the last appended sample for a series. This breaks when scrapes arrive out of order due to clock skew or slow targets. The --storage.tsdb.allow-overlapping-compaction flag and TSDB’s out_of_order_time_window setting (added in Prometheus 2.39) allow a configurable window of out-of-order ingestion, at the cost of additional memory overhead.

Alerting evaluation lag. Alert rules evaluate on the evaluation interval. A 15-second scrape interval with a 15-second evaluation interval means an alert can lag up to 30 seconds from the actual event. Add for durations that account for this, and do not use Prometheus for sub-minute SLAs where that lag is unacceptable.

Tradeoffs Comparison

System	Storage Model	Scalability	PromQL	Retention	Operational Complexity	Best For
Prometheus	Local TSDB, pull model	Single-node, bounded by RAM/disk	Native, full feature set	15 days default, configurable	Low	Per-cluster monitoring, development, standard SREs
Thanos	Prometheus + object storage sidecar	Horizontal, multi-cluster, S3/GCS/ABS	Compatible (Thanos Query)	Unlimited (object storage)	High (many components)	Multi-cluster long-term retention, existing Prometheus investment
Grafana Mimir	Columnar TSDB, fully distributed	Horizontal, microservices architecture	Compatible (Cortex lineage)	Unlimited (object storage)	High (K8s-native)	Cloud-scale, full separation of write and read paths
VictoriaMetrics	Custom column-oriented TSDB	Horizontal (cluster mode), high ingestion	MetricsQL (PromQL superset)	Configurable, efficient compression	Low (single binary option)	High-volume ingestion, resource-constrained environments
InfluxDB	IOx columnar engine	Horizontal (InfluxDB Cloud)	Flux / InfluxQL	Configurable	Medium	Mixed time-series workloads, IoT, high write throughput
Datadog	Fully managed, proprietary	Unlimited (vendor-managed)	None (DogStatsD, custom query)	15 months standard	Very Low (SaaS)	Organizations wanting zero ops overhead, cross-signal correlation

The choice between these systems is rarely about features. It is about where you want to own operational complexity. Prometheus is an excellent choice until you need multi-cluster queries or retention beyond two weeks. At that point, you add Thanos or Mimir, and the architecture grows. VictoriaMetrics offers a middle path: better resource efficiency than Prometheus, simpler operations than Thanos. Datadog removes the ops burden entirely at a cost that scales uncomfortably with ingestion volume.

Closing

Prometheus is a well-scoped piece of infrastructure. Its pull model, in-memory head block, WAL-backed durability, and compacted persistent blocks form a coherent design where every layer exists to serve the core use case: reliable, low-latency access to recent metrics with reasonable long-term storage. The places where it struggles, multi-cluster aggregation, very long retention, high-cardinality event data, are not bugs. They are the boundary of its design. Understanding that boundary is what lets you build a monitoring stack that does not fight its own tools.