How LLM Inference Engines Work: KV Caches, PagedAttention, and Continuous Batching From Prompt to Token

Running a large language model in production is not the same as running it in a Jupyter notebook. The gap between “it works locally” and “it handles 200 concurrent requests at acceptable latency” is where inference engine design lives. Understanding what vLLM, TGI, and similar systems actually do internally helps you make informed decisions about deployment architecture, hardware sizing, and which knobs to turn when something is slow.

This article covers the full stack: from how autoregressive generation works at the GPU level, through KV cache design and PagedAttention, continuous batching, speculative decoding, multi-GPU parallelism, and quantization tradeoffs.

The Autoregressive Loop

Transformer-based language models generate text one token at a time. Each forward pass through the model takes the full sequence of tokens so far and outputs a probability distribution over the vocabulary. You sample or greedily select the next token, append it, and repeat.

The problem is quadratic attention. For a sequence of length N, the attention mechanism computes N×N attention scores. Each new token extends the sequence, which grows the computation. For a 2048-token context, this is manageable. For 32K or 128K contexts, it is expensive.

The key optimization that makes this tractable is the KV cache.

KV Caches: What They Are and Why Memory Is the Bottleneck

During a forward pass, each attention layer computes key and value tensors for every token in the sequence. On subsequent tokens, those tensors for earlier positions do not change. Instead of recomputing them, the engine stores them in GPU memory and reads them for each new token. This is the KV cache.

The memory footprint is significant. For a single inference request with a 4096-token context on a 70B parameter model at float16:

KV cache per layer = 2 (K and V) × context_length × num_heads × head_dim × bytes_per_element
                   = 2 × 4096 × 64 × 128 × 2 bytes
                   = 134 MB per layer

With 80 layers, that is roughly 10.7 GB just for the KV cache of one request. The model weights for a 70B model at float16 are around 140 GB. A GPU with 80 GB of HBM3 memory has to fit both the weights and the KV caches of all concurrent requests.

This is why memory management is the central problem of LLM serving, not compute.

Memory Fragmentation Under Static Allocation

Early inference systems allocated KV cache memory statically: reserve a fixed buffer per request equal to the maximum sequence length. If you set max sequence length to 4096 tokens and get a request that only generates 200 tokens, you wasted 95% of that allocation.

Worse, because requests finish at unpredictable times, the free memory between active allocations becomes fragmented. You might have 10 GB of free GPU memory but be unable to accept a new request because no contiguous block is large enough to hold its maximum-length KV cache.

PagedAttention: Virtual Memory for KV Caches

PagedAttention, introduced in the vLLM paper, applies a familiar idea from OS virtual memory to KV cache management. Instead of allocating one contiguous block per request, the cache is divided into fixed-size pages (blocks), typically 16 or 32 tokens per block.

Each request has a logical page table mapping its token positions to physical GPU memory pages. As a request generates tokens, the engine allocates physical pages on demand. When the request completes, the pages are returned to a free pool and immediately reusable by any other request.

// Simplified model of the block manager data structures
interface PhysicalBlock {
  blockId: number;
  refCount: number;
  tokens: number[]; // token IDs stored in this block
}

interface SequenceGroup {
  requestId: string;
  logicalBlocks: number[]; // indices into physical block table
  generatedTokens: number;
}

class BlockManager {
  private freeBlocks: number[] = [];
  private physicalBlocks: Map<number, PhysicalBlock> = new Map();
  private blockSize: number;

  constructor(totalBlocks: number, blockSize: number) {
    this.blockSize = blockSize;
    for (let i = 0; i < totalBlocks; i++) {
      this.freeBlocks.push(i);
    }
  }

  allocate(sequence: SequenceGroup): boolean {
    if (this.freeBlocks.length === 0) return false;
    const blockId = this.freeBlocks.pop()!;
    this.physicalBlocks.set(blockId, {
      blockId,
      refCount: 1,
      tokens: [],
    });
    sequence.logicalBlocks.push(blockId);
    return true;
  }

  free(sequence: SequenceGroup): void {
    for (const blockId of sequence.logicalBlocks) {
      const block = this.physicalBlocks.get(blockId)!;
      block.refCount -= 1;
      if (block.refCount === 0) {
        this.freeBlocks.push(blockId);
        this.physicalBlocks.delete(blockId);
      }
    }
  }
}

The non-contiguous layout requires the attention kernel to gather K and V tensors from non-adjacent memory locations during computation. vLLM ships custom CUDA kernels that handle this gather operation efficiently.

PagedAttention also enables prefix caching: if two requests share a common prefix (a system prompt, for example), the physical pages for that prefix can be shared between them with copy-on-write semantics. One copy of the prompt KV cache can serve hundreds of requests. This is how vLLM’s prefix caching feature achieves dramatic speedups for workloads with shared system prompts.

Static Batching vs. Continuous Batching

The Throughput Problem with Static Batching

The naive approach to batching is to collect N requests, run a forward pass over all of them simultaneously, then return results. This works well when all requests finish at the same time. In practice, they do not.

Consider a batch of 8 requests where one request generates 5 tokens and another generates 500 tokens. With static batching, the 7 short requests finish early but their GPU slots sit idle while the long request continues. GPU utilization tracks the slowest request, not the average.

Continuous Batching (Iteration-Level Scheduling)

Continuous batching, also called iteration-level scheduling, changes when new requests join the batch. Instead of waiting for the entire batch to complete before accepting new requests, the scheduler inserts new requests into the next forward pass the moment a slot frees up.

Each forward pass can contain a mix of:

• Requests in their prefill phase (processing the prompt in parallel)
• Requests in their decode phase (generating one token at a time)

The scheduler runs between every forward pass. It decides which sequence groups to run, which to preempt (swap to CPU memory if GPU is full), and which new requests to admit.

type RequestPhase = "prefill" | "decode";

interface ScheduledRequest {
  requestId: string;
  phase: RequestPhase;
  promptTokens: number[];
  generatedTokens: number[];
  maxNewTokens: number;
}

interface SchedulerOutput {
  prefillRequests: ScheduledRequest[];
  decodeRequests: ScheduledRequest[];
  preempted: string[];
}

function schedule(
  waiting: ScheduledRequest[],
  running: ScheduledRequest[],
  freeBlocks: number,
  blockSize: number
): SchedulerOutput {
  const blocksNeeded = (req: ScheduledRequest) =>
    Math.ceil(
      (req.promptTokens.length + req.generatedTokens.length + 1) / blockSize
    );

  const output: SchedulerOutput = {
    prefillRequests: [],
    decodeRequests: [],
    preempted: [],
  };

  // Admit new prefill requests if memory allows
  for (const req of waiting) {
    if (blocksNeeded(req) <= freeBlocks) {
      output.prefillRequests.push(req);
      freeBlocks -= blocksNeeded(req);
    }
  }

  // All running requests continue decode
  for (const req of running) {
    output.decodeRequests.push(req);
  }

  return output;
}

The throughput gain from continuous batching is substantial. Benchmarks show 5-10x higher request throughput on typical chat workloads compared to static batching, because GPU utilization is nearly constant rather than pulsing with batch boundaries.

Speculative Decoding

Decode is memory-bandwidth-bound, not compute-bound. For each token, the model reads all its weights from HBM, does a small matmul, and writes back a single token. The hardware is underutilized.

Speculative decoding exploits this slack by using a small draft model to propose multiple tokens ahead, then verifying them in parallel with the large target model.

The sequence:

1. A small model (e.g., 1B parameters) generates K candidate tokens speculatively.
2. The large target model runs a single forward pass over the original context plus all K candidates in parallel.
3. The target model’s outputs are compared token-by-token against the candidates using rejection sampling.
4. All accepted tokens are committed. If a token is rejected, the target model’s own distribution is used for that position.

Because the verification pass processes K tokens in parallel rather than one at a time, it takes roughly the same time as a single decode step. If K tokens are typically accepted (e.g., 3-4 out of 5 speculated), throughput improves proportionally.

The draft model must be aligned with the target model’s distribution for acceptance rates to be high. Common choices: smaller models from the same model family, or n-gram language models that just repeat observed patterns.

Speculative decoding reduces latency for individual requests without degrading output quality. The output distribution of accepted tokens is identical to what the target model would have produced alone, by construction of the rejection sampling algorithm.

Tensor Parallelism and Pipeline Parallelism

Single GPUs can no longer fit the largest models. Multi-GPU inference requires partitioning the computation.

Tensor Parallelism

Tensor parallelism splits individual weight matrices across GPUs. For a transformer’s MLP layer with a weight matrix of shape [d_model, 4d_model], you can split it column-wise across N GPUs so each holds [d_model, 4d_model / N]. Each GPU computes a partial result; an allreduce collective operation sums them.

vLLM uses Megatron-style tensor parallelism: the QKV projection and MLP layers are split across the GPU’s head dimension and intermediate dimension respectively. Each GPU holds a complete set of model layers but a fraction of each layer’s width.

Tensor parallelism requires tight GPU interconnects. NVLink (600 GB/s on H100 NVLink) makes this practical within a node. PCIe (64 GB/s) adds latency that degrades scaling efficiency.

Pipeline Parallelism

Pipeline parallelism assigns different transformer layers to different GPUs. GPU 0 handles layers 0-19, GPU 1 handles layers 20-39, and so on. The output tensor from each stage is passed to the next GPU.

Pipeline parallelism works across nodes with slower interconnects (InfiniBand) but introduces pipeline bubbles: GPUs earlier in the pipeline sit idle while later stages catch up. Micro-batching reduces bubbles by interleaving multiple requests through the pipeline, at the cost of additional latency.

In practice, most deployments use tensor parallelism within a node and pipeline parallelism across nodes when serving models too large for one node’s memory.

Quantization: GPTQ, AWQ, and GGUF

Quantization reduces memory usage and increases throughput by representing weights in lower precision. The tradeoffs vary significantly by format.

Format	Bit Width	Method	Speed	Quality	Typical Use Case
GPTQ	4-bit	Post-training, layer-wise	Fast (GPU)	Good, some degradation at edge cases	vLLM, TGI on GPU
AWQ	4-bit	Activation-aware weight quantization	Faster than GPTQ	Better than GPTQ at same bit width	vLLM, TGI on GPU
GGUF	2-8 bit (mixed)	CPU-friendly packing with optional GPU offload	Moderate (CPU/MPS)	Good at Q4_K_M and above	Ollama, llama.cpp on laptop/edge
FP8	8-bit float	Native hardware (H100)	Fastest on H100	Near-identical to FP16	vLLM on H100 for throughput
GPTQ-int8	8-bit	Dynamic quantization	Fast	Very close to FP16	Memory-constrained GPU

GPTQ quantizes each layer by solving an optimization problem over a calibration dataset, minimizing the reconstruction error of each weight matrix. The result is a 4-bit integer weight matrix plus per-group scaling factors.

AWQ improves on GPTQ by observing that not all weights are equally sensitive to quantization. A small fraction of weights (those corresponding to large activation channels) account for most of the reconstruction error. AWQ scales these channels up before quantization and back down during inference, protecting them from the precision loss without changing the bit width.

GGUF (formerly GGML) packs weights in formats designed for CPU memory layouts with optional GPU offloading. The Q4_K_M variant uses 4-bit quantization with K-means clustering per block, which improves perplexity compared to naive 4-bit rounding. It is the practical choice for running models on Apple Silicon or CPU-only servers where you need to stay off expensive GPU cloud instances.

Production Considerations

Memory planning. Calculate your maximum KV cache memory before deploying. The rule of thumb: subtract model weight memory from total GPU memory, then subtract 1-2 GB for fragmentation overhead. What remains is available for KV pages. Set gpu_memory_utilization in vLLM to 0.90 or lower to leave a safety buffer; hitting 100% triggers OOM kills, not graceful degradation.

Prefill and decode are different bottlenecks. Prefill (processing the prompt) is compute-bound; it benefits from large batches and tensor parallelism. Decode (generating tokens) is memory-bandwidth-bound; it benefits from continuous batching to keep the GPU busy across requests. If your P50 latency is high but GPU utilization is low during decode, the answer is more concurrent requests, not more GPUs.

Prefix caching for shared prompts. If you have a system prompt used in every request, enable prefix caching in vLLM (enable_prefix_caching=True). The memory savings can be dramatic: a 2000-token system prompt shared across 100 concurrent requests reduces KV cache memory by 200,000 tokens worth of pages.

Choosing between vLLM, TGI, and Ollama. This is a deployment context decision, not a quality decision:

Concern	vLLM	TGI (Text Generation Inference)	Ollama
Throughput	Highest (PagedAttention + continuous batching)	High (also continuous batching)	Low to moderate
GPU support	NVIDIA (CUDA), AMD (ROCm), TPU	NVIDIA, AMD	NVIDIA, Apple MPS, CPU
Quantization	GPTQ, AWQ, FP8, GGUF (limited)	GPTQ, AWQ, FP8, BitsAndBytes	GGUF (primary format)
Multi-GPU	Tensor + pipeline parallelism	Tensor parallelism	Limited
Ease of setup	Moderate	Moderate	Easiest
Best for	Production API serving at scale	Hugging Face model ecosystem	Local dev, edge, laptop
OpenAI API compatibility	Yes	Yes	Yes

Use vLLM when throughput per GPU-hour is the primary concern. Use TGI when you are deeply in the Hugging Face ecosystem and want tight integration with model hub tooling. Use Ollama for local development and cases where you are running on Apple Silicon or CPU servers without CUDA.

Speculative decoding in production. Speculative decoding helps most for chat-style workloads where output is conversational prose (high acceptance rates on speculated tokens). It helps less for code generation, structured JSON output, or tasks with high token diversity. Benchmark acceptance rates on your specific workload before enabling it in production; a low acceptance rate wastes the overhead of running the draft model.

Quantization quality checks. Do not rely on perplexity alone to evaluate quantized models. Perplexity tracks average quality; production failures happen at the tail. Evaluate on task-specific benchmarks, especially for multi-step reasoning, instruction following, and any domain-specific knowledge the model must have. A model that looks fine on perplexity can fail noticeably on structured output tasks at Q4 quantization.

Monitoring. The metrics that matter for inference serving: time to first token (TTFT), time per output token (TPOT), token throughput (tokens/sec), KV cache utilization (%), and request queue depth. KV cache utilization above 95% consistently means you are either memory-constrained or receiving more load than your deployment can sustain. GPU utilization below 60% during peak load usually indicates a batching or scheduling configuration problem.

Closing

LLM inference is fundamentally a resource management problem. PagedAttention solves fragmentation. Continuous batching solves GPU idle time. Speculative decoding recovers the wasted memory bandwidth of single-token decode. Quantization buys you more model capacity per GB of memory. Understanding which problem each technique solves makes it straightforward to reason about why your deployment behaves the way it does and which lever to pull next.