How Grafana Loki Works Internally: Label-Based Indexing, Log Chunk Storage, and the Cost-Efficient Architecture That Stores Logs Like Prometheus Stores Metrics

Most log systems solve the search problem by building a full inverted index over every token in every log line. That approach is powerful but expensive: index size routinely matches or exceeds the raw log volume, query planners need to merge posting lists across shards, and storage costs scale with both log volume and cardinality. Grafana Loki takes a different position. It indexes only metadata labels, compresses raw log lines into chunks, and ships those chunks to an object store. The tradeoff is a weaker search model (you cannot grep arbitrary tokens without scanning chunks), but the storage cost per gigabyte of logs is dramatically lower, and the operational footprint is far simpler when you already run Prometheus.

Understanding how Loki implements this architecture means understanding its write path, its read path, its indexing backends, and its compaction mechanics. All of them flow from the same core bet: labels are small, log lines are large, and object storage is cheap.

---

The Core Data Model: Labels and Chunks

Loki’s data model mirrors Prometheus. A log stream is identified by a set of key-value labels, for example {app="api", env="prod", region="us-east-1"}. Every log line that arrives must belong to exactly one stream, identified by its complete label set. Lines within a stream are ordered by timestamp and stored together.

Raw log lines for a stream are buffered in memory and written into compressed chunks. A chunk is a sequence of log entries encoded as a length-prefixed byte block and compressed with snappy or gzip (configurable). Chunks have a configured maximum size (default 256 KB uncompressed) and a maximum age (default 1 hour). When either limit is reached, the chunk is flushed to the object store and its metadata entry is written to the index.

The index stores only stream labels plus pointers to chunks: which object store path holds a given label combination between two timestamps. It does not store the text of log lines. This is the entire basis for Loki’s cost model. An Elasticsearch index for the same logs would store term posting lists, field values, and doc IDs for every searchable token. A Loki index stores only the label fingerprint and a chunk reference.

The consequence is that scanning logs for a substring pattern requires reading the relevant chunks and applying the filter in the querier. For high-cardinality searches across large time windows, this chunk scan is the cost center. For narrow time windows or filtered streams with few active chunks, it is fast.

---

The Write Path: Distributor, Ingester, WAL

Log writes arrive at the distributor. The distributor is stateless and horizontally scalable. It accepts push requests (from Promtail, the Loki client, or any compatible agent), validates the labels, enforces rate limits, and routes the write to the appropriate ingester via consistent hashing on the stream’s label fingerprint.

The distributor uses a ring (backed by a key-value store like etcd or memberlist gossip) to determine which ingester owns which stream. By default, Loki uses a replication factor of 3: each write is sent to three ingesters, and a write quorum (2 of 3) must acknowledge before the distributor returns success. This gives you durability before the chunk is flushed to object storage.

The ingester is the stateful component. It holds the in-memory chunk buffer for each stream it owns. When a write arrives, the ingester appends the log line to the head chunk for that stream. If no chunk exists yet, one is allocated.

// Simplified representation of an ingester's in-memory state
interface ChunkHead {
  labels: Record<string, string>;
  entries: Array<{ timestamp: number; line: string }>;
  sizeBytes: number;
  openedAt: number;
}

class Ingester {
  private streams = new Map<string, ChunkHead>();

  push(labels: Record<string, string>, entries: LogEntry[]): void {
    const fingerprint = labelFingerprint(labels);
    let chunk = this.streams.get(fingerprint);

    if (!chunk) {
      chunk = { labels, entries: [], sizeBytes: 0, openedAt: Date.now() };
      this.streams.set(fingerprint, chunk);
    }

    for (const entry of entries) {
      chunk.entries.push(entry);
      chunk.sizeBytes += entry.line.length;
    }

    if (chunk.sizeBytes >= MAX_CHUNK_BYTES || isExpired(chunk)) {
      this.flush(fingerprint, chunk);
    }
  }
}

Before writing to the in-memory chunk, the ingester appends to a Write-Ahead Log (WAL). The WAL records each incoming push segment as a protobuf-encoded record on local disk. On ingester crash and restart, the WAL is replayed to reconstruct in-memory state for streams that had not yet been flushed to object storage. This protects against data loss when the ingester process dies between a successful distributor quorum write and the chunk flush.

When a chunk reaches its size or age threshold, the ingester compresses it, writes it to the configured object store (S3, GCS, Azure Blob, local filesystem), and writes the index entry mapping that chunk’s label fingerprint and time range to its object store path.

---

The Index Backend: boltdb-shipper and TSDB

Loki has had multiple index backends over time. Understanding which one you are running matters for operational behavior.

boltdb-shipper was the standard index backend for several Loki versions. It uses BoltDB (an embedded key-value store using B-trees) to write index entries locally per ingester, then periodically uploads the BoltDB files to the object store. Queriers download and cache these index files locally to answer queries. The shipper model makes the index eventually consistent: there is a sync interval (typically 5 minutes to 15 minutes) between when a chunk is indexed locally and when the index file is uploaded and visible to all queriers.

TSDB (based on Prometheus’s TSDB block format) became the recommended index backend in Loki 2.8+. It writes index data as TSDB blocks directly to the object store, using the same block structure Prometheus uses for metric metadata. TSDB provides better query performance at high label cardinality, more efficient compaction, and lower memory usage on queriers than boltdb-shipper. For new deployments, TSDB is the correct choice.

Both backends store the same information: for each unique label set (stream), the set of chunk references (object store paths) that cover each time range. A query that asks for {app="api"} |= "error" first hits the index to identify which chunks contain entries for streams matching {app="api"} within the requested time window, then fetches and scans those chunks.

---

The Read Path: Query Frontend, Querier, and LogQL

The read path in Loki follows a pattern common to distributed query systems: a frontend splits and schedules work, and workers execute the pieces.

The query frontend is the entry point for queries. It accepts a LogQL query over a time range, splits the query into smaller sub-queries by time (by default, sub-queries cover 30-minute windows each), queues them, and dispatches them to available queriers. This parallelization is how Loki handles long time-range queries without timing out: a 24-hour query becomes 48 parallel 30-minute queries.

The query frontend also handles result caching. Queries for completed time windows whose results are already in the results cache (Redis or Memcached) are served without re-querying the object store.

The querier executes sub-queries. For each sub-query, the querier:

1. Consults the index to find matching chunk paths for the label selector and time range.
2. Downloads or streams the relevant chunks from the object store (or from the ingester’s in-memory buffer for very recent data).
3. Decompresses and iterates the chunk entries, applying the LogQL pipeline.
4. Returns the filtered, optionally aggregated results to the query frontend.

LogQL is the query language. It has two components: the stream selector and the pipeline. The stream selector filters by labels: {app="api", env="prod"}. The pipeline is a chain of expressions applied to each matching log line:

{app="api", env="prod"}
  | json
  | status_code >= 400
  | line_format "{{.status_code}} {{.path}}"

The | json stage parses each log line as JSON and promotes fields to labels for downstream filtering. The | status_code >= 400 filter drops lines where the parsed field is below 400. The | line_format reformats the output. All of this runs in the querier’s memory, line by line, per chunk.

Metric queries (for example, rate({app="api"}[5m])) apply a range aggregation over the log stream. The querier counts, sums, or otherwise aggregates log line counts over rolling windows and returns a time series, which can be graphed in Grafana or evaluated in alert rules via the Loki ruler.

---

Compactor: Index Compaction and Retention Enforcement

The compactor is a background component responsible for two things: compacting the index and enforcing retention.

Index compaction merges the many small BoltDB or TSDB index files that ingesters upload over time into fewer, larger files. Without compaction, queriers would need to open and merge hundreds of small files for long-range queries, which is slow. The compactor runs periodically (every 2 hours by default) and writes merged index files back to the object store.

Retention is enforced by the compactor during its compaction runs. Loki supports two retention modes: global retention (a single TTL applied to all logs) and per-stream retention using label matchers. When the compactor processes an index period that has aged past its retention boundary, it deletes the corresponding chunks from the object store and removes their index entries.

# Example Loki retention config (compactor section)
compactor:
  working_directory: /loki/compactor
  shared_store: s3
  retention_enabled: true

limits_config:
  retention_period: 744h  # 31 days global default

  per_tenant_override_config: /etc/loki/per-tenant-overrides.yaml

Per-stream retention (available in Loki 2.5+) lets you keep high-priority logs longer than general application logs:

# per-tenant-overrides.yaml
overrides:
  "tenant-a":
    retention_stream:
      - selector: '{level="error"}'
        priority: 10
        period: 2160h  # 90 days for errors
      - selector: '{level="debug"}'
        priority: 5
        period: 72h    # 3 days for debug

The compactor processes deletions at the chunk boundary, not the line boundary. A chunk that partially overlaps a retention boundary is kept until all entries in that chunk have aged past the TTL.

---

Deployment Topology: Monolithic vs. Microservices Mode

Loki can run as a single binary (-target=all) for development or small deployments, or as individual components for production scale. Each component (distributor, ingester, querier, query-frontend, compactor, ruler) is a separate process that can be scaled independently.

The microservices mode is the recommended topology for production. Ingesters are stateful and should use persistent local volumes for the WAL; they should be deployed as a StatefulSet in Kubernetes. Distributors and queriers are stateless and can scale horizontally with regular Deployments.

The Simple Scalable deployment mode (introduced in Loki 2.4) is a middle ground: it groups components into three roles (read, write, backend) that can be scaled independently without the full operational complexity of running every component separately. For most teams, Simple Scalable is the right starting point before moving to full microservices.

# Simple Scalable: write path
target: write  # runs distributor + ingester

# Simple Scalable: read path
target: read   # runs query-frontend + querier

# Simple Scalable: backend
target: backend  # runs compactor + ruler + index-gateway

---

Production Considerations

Label cardinality is the primary tuning concern. Labels in Loki serve the same purpose as labels in Prometheus: they are the sharding key for the index. Adding a high-cardinality label (user ID, request ID, trace ID) creates a new stream per unique value. Ten million unique users means ten million streams, which produces an index of enormous size and makes the distributor’s consistent hashing fan out writes across too many buckets. The rule of thumb from the Loki authors: keep label cardinality low (below a few thousand unique stream combinations), and push high-cardinality values into the log line itself to be parsed with | json or | logfmt at query time.

Chunk size and target chunk fullness matter for query performance. Small chunks (from streams with very low log volume) produce many tiny object store objects. Fetching thousands of small objects to answer a query is slower than fetching a few large ones. The chunk_target_size (default 1.5 MB compressed) and max_chunk_age (default 2 hours) together control how full chunks get before flushing. For low-volume streams, increasing max_chunk_age reduces chunk fragmentation at the cost of higher in-memory buffer time.

Ingester WAL replay time scales with write volume. On restart, each ingester must replay its WAL before it can serve writes. In high-throughput deployments, WAL replay can take minutes. Sizing ingester persistent volumes with fast disks (NVMe-backed PVCs) and tuning ingester.wal.replay_memory_ceiling reduces replay time.

Query parallelism is limited by querier count. The query frontend’s time-range splitting generates one sub-query per split interval per querier. If you have 4 queriers and a 24-hour query with 30-minute splits, you have 48 sub-queries competing for 4 workers. Adding queriers is the horizontal scaling lever for read performance. The max_outstanding_per_tenant setting in the query frontend controls how many queued queries a single tenant can hold.

The ruler enables alerting from LogQL. The Loki ruler evaluates metric queries on a schedule and fires alerts via the Alertmanager-compatible API. This means you can alert on log-derived signals (error rate, exception counts) using the same pipeline as Prometheus without shipping those metrics to a separate system.

---

Tradeoffs Comparison

Dimension	Grafana Loki	Elasticsearch	Datadog Logs	Splunk
Index model	Label metadata only	Full inverted index over all fields	Full index (managed)	Full index
Storage cost	Very low (object store + thin index)	High (index often matches raw log size)	Per-GB ingestion pricing	Per-GB ingestion pricing
Query model	Label filter + line scan; metric aggregations via LogQL	Full-text search, aggregations, KQL	Full-text search, facets, APM correlation	SPL full-text search, stats
Substring search latency	Higher for wide windows (chunk scan)	Low (inverted index lookup)	Low	Low
Operational complexity	Moderate (multiple stateful components)	High (shard management, heap tuning)	Low (fully managed)	High (forwarders, indexers, search heads)
Prometheus integration	Native (same labels, same alerting pipeline)	Separate stack required	APM/metrics integration	Separate stack required
Retention control	Per-stream label matchers, compactor-enforced	ILM policies on index	Retention tiers by index	Retention per index
Horizontal scalability	Stateless read/write path; ingesters scale with sharding	Shard rebalancing required	Automatic (managed)	Indexer cluster required
Sweet spot	Cloud-native Kubernetes deployments already running Prometheus	General-purpose log search with rich full-text query needs	Teams wanting zero-ops observability with APM integration	Enterprise compliance and security with complex correlation

---

Closing

Loki’s architecture is a deliberate rejection of the full-text index as the foundation of log storage. The bet is that most log queries are scoped by metadata (which service, which environment, which time window), and the text scan within that scope is fast enough when chunks are small and object store throughput is high. For teams running Prometheus-based monitoring stacks, the label model and query language translate directly, and the infrastructure footprint is a fraction of what Elasticsearch requires at equivalent log volume.

The constraints that follow from this design are real: high-cardinality labels break the index model, wide-window substring queries are slow, and WAL-backed ingesters require careful persistent volume management. Working within those constraints, rather than around them, is what separates a well-tuned Loki deployment from one that fights the architecture.