How Elasticsearch Works Internally: Inverted Indexes, Lucene Segments, and Distributed Search Execution

Most engineers treat Elasticsearch as a black box: you push JSON in, you get ranked results out. That model works until you hit a mapping explosion at 3 AM, or your deep-pagination query kills every data node in a cluster, or your search latency doubles after a routine index operation. Understanding what happens inside the engine makes those failure modes predictable and, more importantly, preventable.

This article covers Elasticsearch from the ground up: how Lucene builds and queries inverted indexes, how the segment model shapes write and read behavior, how a search request moves across a cluster in two phases, how relevance scoring works, and what the write path looks like from client to durable storage.

Lucene’s Inverted Index: Term Dictionaries and Postings Lists

Elasticsearch is not a search engine. Lucene is. Elasticsearch is a distributed coordination layer built on top of Lucene, and every shard in an Elasticsearch index is a standalone Lucene index. To understand Elasticsearch, you need to understand Lucene’s data structures first.

The core structure is the inverted index. For a given field, Lucene maintains a term dictionary mapping every unique term that appears in that field to a postings list. The postings list is an ordered sequence of document IDs that contain the term, optionally augmented with term frequency (how many times the term appears in the document) and position data (where in the field each occurrence sits).

A naive implementation would store the term dictionary as a hash map. Lucene uses a Finite State Transducer (FST) instead. An FST is a compressed directed acyclic graph that encodes a sorted mapping from byte sequences (terms) to values (pointers into the postings data) using shared prefix and suffix compression. For a field with millions of unique terms, the FST can fit entirely in the JVM heap while a hash map of equivalent coverage would not. Lookups are O(k) where k is the length of the search term, not O(n) in the dictionary size.

Postings lists themselves are stored using delta-encoded integers compressed with techniques like Frame of Reference (FOR) encoding. Rather than storing absolute document IDs, Lucene stores the gap between consecutive IDs. For a term that appears in documents 1, 2, 3, 4, 100, the stored deltas are 1, 1, 1, 1, 96. Small integers compress very efficiently, and most common terms appear in dense document runs where deltas are small.

Skip lists sit on top of the postings list to accelerate intersection. When computing the result for a Boolean query like A AND B, Lucene intersects the postings lists for A and B. Skip pointers every N entries let the algorithm jump forward through one list to find the next candidate that could appear in both, without scanning every entry linearly.

Numeric and keyword fields use different internal structures. Keyword fields use the same FST-based term dictionary. Numeric fields (integers, longs, doubles) use a block k-d tree (BKD tree) that supports efficient range queries. The BKD tree partitions multi-dimensional space recursively, which is also what makes geospatial range queries fast.

The Segment Model: Immutability, Refresh, and Merge

Lucene organizes its index into segments. A segment is an immutable mini-index: once written, it is never modified in place. This immutability is what makes Lucene’s concurrency model tractable. Readers never contend with writers because writers never touch existing files.

When documents are indexed, they first accumulate in an in-memory buffer. Periodically (controlled by index.refresh_interval, defaulting to one second), Elasticsearch calls IndexWriter.flush() on the underlying Lucene index. This operation serializes the in-memory buffer to a new segment on disk and makes it visible to subsequent searches. The new segment is not yet fsync-ed to durable storage at this point. That is a deliberate performance tradeoff: fsync is expensive, and making a segment searchable does not require it.

This is what “near-real-time” means in Elasticsearch. A document indexed at T=0 will be searchable at T=1s (roughly), not T=0. The refresh creates a new Lucene IndexReader that includes the new segment. The reader is reference-counted so ongoing searches complete against the old view while new searches pick up the new one.

Over time, many small segments accumulate. Each segment adds overhead to every search: queries must be executed against all segments and the results merged. Lucene’s MergePolicy (in Elasticsearch, configured via index.merge.policy.* settings) continuously merges smaller segments into larger ones in the background using a tiered merge strategy. Segments within a similar size tier are merged together, producing a new immutable segment, and the input segments are deleted once no reader holds a reference to them.

Merging is CPU and I/O intensive. You can observe merge activity via the _cat/nodes?v&h=merges.current,merges.current_docs endpoint. A cluster under heavy write load with inadequate merge throttling will accumulate segment debt: too many small segments, degraded search performance, and eventually too many open file handles.

Deletes in Lucene do not remove documents from segments. Instead, Lucene maintains a per-segment deletion bitset. A deleted document is still physically present in the segment files; it is simply masked from query results. The space is reclaimed only when a merge produces a new segment that omits the deleted documents. This is why disk usage can stay elevated after bulk deletes until merges complete.

import { Client } from "@elastic/elasticsearch";

const client = new Client({ node: "http://localhost:9200" });

// Force a segment merge to reclaim deleted document space
// Use sparingly in production — this is I/O intensive
await client.indices.forcemerge({
  index: "my-index",
  max_num_segments: 1, // merge down to a single segment (read-only indexes only)
  flush: true,
});

// Check segment count and deleted docs per shard
const stats = await client.indices.segments({ index: "my-index" });
for (const [index, data] of Object.entries(stats.indices)) {
  for (const [shardId, shards] of Object.entries(data.shards)) {
    for (const shard of shards as any[]) {
      const totalDeleted = shard.segments
        ? Object.values(shard.segments).reduce(
            (sum: number, seg: any) => sum + seg.deleted_docs,
            0
          )
        : 0;
      console.log(`Index: ${index}, Shard: ${shardId}, Deleted docs: ${totalDeleted}`);
    }
  }
}

Distributed Search: Scatter-Gather in Two Phases

An Elasticsearch index is divided into primary shards. Each primary shard is a full Lucene index. Searches execute across all shards in parallel and the results are merged at a coordinating node.

The coordinating node is whatever node receives the client request. It does not hold data; its job is to fan out the request and aggregate results. The process happens in two phases.

Query phase (scatter): The coordinating node broadcasts the query to one copy (primary or replica) of every shard in the index. Each shard executes the query locally against its Lucene index and returns a sorted list of document IDs with their relevance scores. The list length is from + size (to support pagination). No document source (_source) is fetched at this stage. The coordinating node receives n_shards * (from + size) results and merges them into a globally sorted list, keeping only the top size entries.

Fetch phase (gather): The coordinating node issues a multi-get request to the shards that hold the top size document IDs, asking for the full _source payload. Shards return the source documents. The coordinating node assembles the final response.

This two-phase design minimizes data transfer. Shipping full documents in the query phase when only a fraction of them end up in the final result set would waste I/O. The downside is the coordinating node must hold in memory the top (from + size) results from every shard simultaneously during the merge step, which creates the deep pagination memory problem described in the production considerations section.

// A basic search with explicit from/size
const response = await client.search({
  index: "products",
  body: {
    from: 0,
    size: 20,
    query: {
      bool: {
        must: [{ match: { description: "wireless headphones" } }],
        filter: [{ term: { category: "electronics" } }],
      },
    },
    sort: [{ _score: { order: "desc" } }, { updated_at: { order: "desc" } }],
  },
});

Shard Routing and Replication

By default, Elasticsearch routes a document to a primary shard using the formula:

shard = hash(_routing) % number_of_primary_shards

The _routing value defaults to the document _id. You can supply a custom routing value to colocate related documents on the same shard, which can dramatically speed up queries that always filter on the routing key (for example, queries scoped to a single tenant in a multi-tenant system).

Every primary shard has zero or more replica shards. Replicas serve two purposes: read availability and fault tolerance. Search requests are load-balanced across all copies (primary plus replicas). If a primary fails, a replica is promoted. Replicas are never on the same node as their primary (or each other) by default.

The number of primary shards is fixed at index creation. Changing it requires reindexing. The _split and _shrink APIs allow doubling or halving shard counts without a full reindex, but they have strict preconditions (read-only index, specific shard alignment).

BM25 Relevance Scoring

Elasticsearch switched from TF-IDF to BM25 as its default similarity function in version 5.0. BM25 addresses two well-known weaknesses of classic TF-IDF.

First, TF-IDF has unbounded term frequency saturation: a document mentioning a term 100 times scores much higher than one mentioning it 10 times, even if the marginal information gain is negligible. BM25 applies a saturation function controlled by the k1 parameter (default 1.2). As term frequency increases, the score increase diminishes toward an asymptote.

Second, TF-IDF penalizes long documents inconsistently. BM25 normalizes term frequency by document length using the b parameter (default 0.75), which controls how aggressively length normalization is applied. Setting b=0 disables length normalization entirely.

The per-shard BM25 score uses the IDF component computed from statistics local to that shard. In a cluster with many shards, IDF scores can diverge across shards if the document distribution is uneven, leading to relevance inconsistencies. Setting search_type=dfs_query_then_fetch forces a pre-scatter step where the coordinating node collects term frequency statistics from all shards before executing the query, producing globally consistent IDF values. It adds one extra round trip and is rarely worth it unless you are debugging relevance and score discrepancies are visible.

// DFS query for globally consistent scoring (debug / small index use)
const response = await client.search({
  index: "articles",
  search_type: "dfs_query_then_fetch",
  body: {
    query: {
      multi_match: {
        query: "distributed systems consistency",
        fields: ["title^3", "body"],
        type: "best_fields",
      },
    },
  },
});

The Write Path: From Client to Durable Storage

When a document reaches the primary shard for indexing, the following sequence occurs:

1. Parse and validate. The document is parsed against the index mapping. If dynamic mapping is enabled and the document contains unknown fields, new field mappings are derived automatically and stored in cluster state.

2. Write to translog. Before the document is added to the in-memory Lucene buffer, it is written to the translog (transaction log), a file-based write-ahead log local to the shard. The translog is fsync-ed based on the index.translog.durability setting: request (default, sync on every operation) or async (sync on a configurable interval). Async mode improves throughput at the cost of up to sync_interval of data loss on node failure.

3. Add to in-memory buffer. The document is added to Lucene’s in-memory IndexWriter buffer.

4. Forward to replicas. The primary forwards the operation to all in-sync replicas. Each replica performs steps 2 and 3 independently. The primary waits for acknowledgment from all replicas before responding to the client (controlled by wait_for_active_shards).

5. Refresh. On the next refresh interval tick, the in-memory buffer is flushed to a new Lucene segment, making the document searchable.

6. Flush (fsync). Periodically (or when the translog grows large, controlled by index.translog.flush_threshold_size, default 512 MB), Elasticsearch calls Lucene’s commit() to fsync all segment files. Once committed, the corresponding translog entries can be truncated. The translog exists to replay un-committed segment data if the node crashes.

// Bulk indexing with explicit refresh control
// refresh: "wait_for" makes documents searchable before the response returns
// but blocks until a refresh cycle completes — do not use in high-throughput paths
const body = documents.flatMap((doc) => [
  { index: { _index: "events", _id: doc.id } },
  doc,
]);

const bulkResponse = await client.bulk({
  refresh: "false", // let the normal refresh interval handle it
  body,
});

if (bulkResponse.errors) {
  const failed = bulkResponse.items.filter(
    (item: any) => item.index?.error
  );
  console.error(`${failed.length} documents failed`, failed);
}

Production Considerations

Shard Sizing

The most common mistake is creating too many shards. Each shard is a Lucene index with its own file handles, heap overhead, and merge activity. The Elasticsearch documentation suggests targeting 10-50 GB per shard for search-heavy workloads and up to a few hundred GB for logging workloads where searches are less latency-sensitive. A 100 GB index with 20 shards of 5 GB each will consistently underperform the same data in 2-4 shards of 25-50 GB.

Cluster state size scales with the number of indices and shards. A cluster managing 10,000 shards becomes sluggish during shard allocation and recovery because cluster state is synchronized across all nodes for every change. Aim for no more than 20 shards per GB of JVM heap on the master-eligible nodes.

For time-series data, use data streams backed by index lifecycle management (ILM) to automatically roll over and delete old indices. Keep hot tier indices small and frequent enough that each rollover produces shards in the target size range.

Mapping Explosions

Dynamic mapping is convenient and dangerous. If your documents contain user-generated keys or high-cardinality nested structures, enabling dynamic mapping will eventually produce an index with tens of thousands of fields. Each field requires heap-resident data structures in every shard. A mapping with 100,000 fields in a 10-shard index has been known to consume gigabytes of heap purely for field metadata.

Mitigations:

• Set dynamic: false on mappings that do not need their sub-fields indexed. The fields are stored in _source but not indexed or searchable.
• Set dynamic: strict to reject documents with unknown fields outright.
• Use flattened field type for objects where you need keyword search across keys but do not need per-field analysis. A flattened field maps the entire object as a single field with leaf values queryable as keywords.

// Mapping with explicit dynamic controls
await client.indices.create({
  index: "audit-logs",
  body: {
    mappings: {
      dynamic: "false", // unknown fields are stored but not indexed
      properties: {
        timestamp: { type: "date" },
        event_type: { type: "keyword" },
        user_id: { type: "keyword" },
        // metadata is arbitrary JSON — store it, but don't explode the mapping
        metadata: { type: "flattened" },
      },
    },
  },
});

Deep Pagination

The from + size pagination model requires the coordinating node to collect and sort the top from + size results from every shard. For from=10000, size=10, the coordinating node receives and sorts 10,010 results per shard. With 10 shards, that is 100,100 result objects in memory during the merge step. Elasticsearch enforces a hard limit via index.max_result_window (default 10,000) to prevent this from causing OOM failures.

For deep pagination use cases (exports, scrolling through large result sets), there are two alternatives:

Search After: Pass the sort values of the last result as search_after in the next request. This is stateless and safe for concurrent writers. It does not support jumping to an arbitrary page, only sequential forward traversal.

Point In Time (PIT) + Search After: Open a PIT with POST /index/_pit?keep_alive=1m. The PIT freezes a consistent view of the index across multiple requests. Combine it with search_after for stable pagination even when documents are being added or deleted. Close the PIT when done to release resources.

// Open a Point In Time for stable deep pagination
const pit = await client.openPointInTime({
  index: "orders",
  keep_alive: "2m",
});

let searchAfter: any[] | undefined;
const allResults: any[] = [];

while (true) {
  const page = await client.search({
    body: {
      size: 1000,
      query: { range: { created_at: { gte: "2026-01-01" } } },
      sort: [{ created_at: { order: "asc" } }, { _id: { order: "asc" } }],
      pit: { id: pit.id, keep_alive: "2m" },
      ...(searchAfter ? { search_after: searchAfter } : {}),
    },
  });

  const hits = page.hits.hits;
  if (hits.length === 0) break;

  allResults.push(...hits.map((h: any) => h._source));
  searchAfter = hits[hits.length - 1].sort;
}

await client.closePointInTime({ body: { id: pit.id } });

Cluster Stability

Master nodes manage cluster state: shard allocation, index metadata, node membership. They are not designed to handle search or indexing load. Dedicate master-eligible nodes and give them no data roles. Three master-eligible nodes is the standard configuration for production: it tolerates one master node failure while still achieving the quorum needed for elections.

Heap sizing for data nodes follows the rule of allocating no more than 50% of available RAM to the JVM heap, capped at 30-31 GB. Beyond 31 GB, the JVM can no longer use compressed ordinary object pointers, and heap memory becomes less efficient. The remaining RAM goes to the OS page cache, which Lucene depends on heavily for segment reads.

Monitor jvm.mem.heap_used_percent and shard allocation health (_cluster/health). A cluster in yellow state has all primaries assigned but some replicas unassigned, usually due to insufficient nodes. A cluster in red state has at least one unassigned primary, meaning some documents are not searchable at all.

Tradeoffs: Elasticsearch vs. Alternatives

Dimension	Elasticsearch	OpenSearch	Apache Solr	Meilisearch	Typesense
Lineage	Elastic (commercial)	AWS fork of ES 7.10	Apache Foundation	Independent, open-source	Independent, open-source
Query model	Query DSL (JSON), SQL plugin	Query DSL (JSON), SQL plugin	Solr query syntax, JSON API	Simple JSON, limited DSL	Simple JSON, typo-tolerant
Relevance tuning	BM25, custom similarity, scripted scoring	BM25, k-NN with FAISS	BM25, LTR plugin	BM25 with built-in typo tolerance	BM25 with typo correction
Horizontal scaling	Built-in, multi-shard, multi-node	Built-in (same as ES)	Built-in via SolrCloud	Single-node (multi-node in beta)	Multi-node with Raft consensus
Write throughput	High with bulk API	High (same as ES)	High with SolrJ	Moderate	Moderate
Operational complexity	High: JVM, cluster state, ILM, shard planning	High (same as ES)	High: ZooKeeper or embedded, config-heavy	Low: single binary	Low: single binary
Vector search	kNN via HNSW (8.x+)	kNN via FAISS/HNSW	Dense vector field, HNSW	Experimental hybrid	Built-in vector search
Licensing	SSPL / Elastic License 2.0 (source-available)	Apache 2.0	Apache 2.0	MIT (source)	GPL v3 / commercial
Sweet spot	Large-scale log analytics, enterprise search, complex query DSL	ES workloads needing Apache 2.0 license	Legacy Lucene workloads, SolrCloud shops	Developer-friendly instant search, small to medium datasets	Instant search, SaaS, simple ops, small to medium datasets

Closing

Elasticsearch’s performance characteristics follow directly from Lucene’s design choices: the FST-based term dictionary keeps the search index queryable at heap-resident speeds, segment immutability makes concurrent reads and writes straightforward, and the two-phase scatter-gather model minimizes cross-node data transfer at the cost of coordinating node memory pressure during deep pagination. The failure modes that catch teams off guard, mapping explosions, shard proliferation, translog durability misconfiguration, are predictable once you understand the data structures behind them. The engine is not magic; it is a set of explicit tradeoffs expressed in file formats and network protocols.