How LLM Tokenizers Work: BPE, Vocabulary Construction, and Why Token Economics Matter for Production AI

Every time you call a language model API, your request is billed in tokens, your context window fills in tokens, and your model’s latency is a function of tokens generated. Yet most engineers working with LLMs treat tokenization as an opaque preprocessing step. That opacity has real production costs: inefficient prompts, misjudged RAG chunk sizes, unexpected multilingual cost spikes, and context windows that overflow in ways that are hard to debug. Understanding how tokenizers actually work fixes all of that.

What a Tokenizer Does

A tokenizer converts a raw string into a sequence of integer IDs. Those IDs index into a vocabulary table that the model’s embedding layer uses to look up dense vector representations. The model never sees raw text. It sees a list of integers. That means the tokenizer is not a detail — it is the interface between language and the model’s internal arithmetic.

Three decisions shape a tokenizer’s behavior: the algorithm used to learn the vocabulary, the size of that vocabulary, and how the algorithm handles bytes versus characters versus Unicode codepoints. The dominant approaches in production LLMs today are Byte Pair Encoding (BPE) and Unigram Language Model (used by SentencePiece). Understand those two and you understand 95% of what matters in practice.

Byte Pair Encoding: The Algorithm Step by Step

BPE was originally a data compression algorithm. It was adapted for NLP by Sennrich et al. in 2016 and has since become the standard tokenization algorithm for GPT-2, GPT-3, GPT-4, Claude, LLaMA, and most other mainstream models.

The core idea: start with a base vocabulary of individual bytes (or characters), then repeatedly merge the most frequent pair of adjacent tokens into a new token. Repeat until you reach the target vocabulary size.

Here is a concrete example. Suppose your training corpus has the following word frequencies:

"low": 5, "lower": 2, "newest": 6, "widest": 3

Initial character vocabulary (ignoring end-of-word markers for clarity): l, o, w, e, r, n, s, t, i, d.

Step 1: Count all adjacent pairs. The pair e, s appears in “newest” (6) and “widest” (3), totaling 9. That is the most frequent pair. Merge it: create a new token es.

Step 2: Recount. Now es, t appears 9 times (in “newest” and “widest”). Merge it: create est.

Step 3: Continue. Eventually tokens like low, er, new, est, wid emerge as atomic units. Each merge reduces the average sequence length while capturing common subword patterns.

In practice, modern BPE tokenizers operate on UTF-8 byte sequences directly rather than Unicode characters. This is critical: it means every string is tokenizable regardless of the character set. Japanese, Arabic, Hindi, and emoji all reduce to bytes. The model is never “stuck” on an unknown character — but tokenization efficiency varies dramatically across scripts, as we will see later.

How tiktoken Builds Its Vocabulary

OpenAI’s tiktoken library implements BPE with a few specific design choices. Each encoding (cl100k_base for GPT-4 and GPT-3.5-turbo, o200k_base for GPT-4o) ships with a pre-trained merge table and a vocabulary file. At runtime, tiktoken does not re-learn anything. It applies the pre-trained merge rules to your input string.

The merge rules are applied via a regex presplit before BPE runs. The pattern splits input text into natural units (words, contractions, whitespace, numbers, punctuation) before tokenization begins. This prevents BPE from merging across word boundaries in ways that produce unintuitive tokens. For example, "don't" is split into ["don", "'t"] before BPE runs, so the apostrophe and t never merge with adjacent words.

Here is how to use tiktoken to count tokens and inspect the tokenization of a string:

import Tiktoken from "tiktoken";

const enc = Tiktoken.get_encoding("cl100k_base");

function countTokens(text: string): number {
  const tokens = enc.encode(text);
  return tokens.length;
}

function inspectTokens(text: string): Array<{ id: number; text: string }> {
  const tokens = enc.encode(text);
  return Array.from(tokens).map((id) => ({
    id,
    text: new TextDecoder().decode(enc.decode(new Uint32Array([id]))),
  }));
}

// Token count for a system prompt
const systemPrompt = `You are a helpful assistant. Answer questions concisely.`;
console.log(countTokens(systemPrompt)); // 13

// Inspect a surprising example
const surprising = "tokenization";
inspectTokens(surprising).forEach(({ id, text }) => {
  console.log(`${id}: "${text}"`);
});
// Output:
// 5964: "token"
// 1082: "ization"

The word “tokenization” becomes two tokens. “token” is common enough to be a single token. “ization” is a common suffix. But “tokenization” as a single unit did not appear frequently enough in the BPE training corpus to earn its own merge.

How SentencePiece Differs

Google’s SentencePiece is used by T5, mT5, ALBERT, LLaMA (via its own BPE variant), Gemma, and others. The key architectural difference: SentencePiece treats the input as a raw byte stream and includes whitespace as part of the token. In BPE-based tokenizers like tiktoken, whitespace is handled by the presplit regex. In SentencePiece, a space before a word becomes part of the word’s token representation.

That matters when you are constructing prompts manually or concatenating strings, because adding or removing leading spaces can change tokenization. A word that appears at the start of a sentence (no preceding space) may tokenize differently than the same word in the middle of a sentence (preceding space). If you are computing token counts for partial strings and concatenating them, the boundary space changes the count.

SentencePiece also supports the Unigram Language Model algorithm as an alternative to BPE. Unigram starts with a large vocabulary and prunes it, using a probabilistic model to select the most likely tokenization of a string rather than the longest-match greedy approach BPE uses. This produces more consistent segmentation across morphologically rich languages.

Why Tokens Don’t Map 1:1 to Words

The ratio of tokens to words (or characters) is not constant. It depends on the language, the word frequency in the training corpus, and the structure of the text. Here are concrete examples using cl100k_base:

Text	Characters	Tokens	Tokens/Char
"hello world"	11	2	0.18
"Hello, World!"	13	4	0.31
"pneumonoultramicroscopicsilicovolcanoconiosis"	45	13	0.29
"2024-12-31T23:59:59.999Z"	24	11	0.46
"こんにちは世界" (Japanese: “hello world”)	7 chars	9	1.29
"مرحبا بالعالم" (Arabic: “hello world”)	13 chars	11	0.85
"{\n \"key\": \"value\"\n}"	20	10	0.50

Three patterns stand out. First, common English words tokenize efficiently. “hello” and “world” are each one token. Second, non-Latin scripts tokenize less efficiently because BPE training corpora are English-heavy, so the merge rules favor English subword patterns. Japanese text can use two to four times as many tokens as an equivalent English string conveying the same information. Third, structured data (JSON, ISO dates, UUIDs) is expensive because the specific format strings appear rarely in training data and fall back to character-level tokenization.

The practical implication: if you are running a multilingual RAG system and sizing chunks in tokens, a chunk limit that is comfortable for English may truncate non-English content aggressively.

Vocabulary Size Tradeoffs

Vocabulary size is a hyperparameter set at training time. You cannot change it without retraining the model. The choice involves real tradeoffs:

Vocabulary Size	Representative Models	Avg Tokens per English Word	Multilingual Efficiency	Embedding Table Size
~32K	GPT-2, early BERT	~1.3	Poor	Small
~50K	GPT-3 (gpt2 encoding)	~1.3	Poor	Small
~100K	GPT-4 (cl100k_base)	~1.1	Moderate	Medium
~200K	GPT-4o (o200k_base)	~1.05	Good	Larger
~256K	Gemma 3, LLaMA 3	~1.05	Good	Larger

A larger vocabulary means more common sequences get their own token, which reduces sequence length for the same input. Shorter sequences mean: lower context usage, faster attention computation (attention is O(n²) in sequence length), and lower API cost. The downside is a larger embedding table that consumes GPU memory and adds weight to the model’s first and last layer.

For multilingual text, a larger vocabulary specifically means more tokens from non-English scripts earn their own entries rather than falling back to byte-level tokens. Going from 50K to 256K vocabulary can cut Japanese token counts by 30-40%.

Context Window Economics

Context window size is measured in tokens. When a model advertises a 128K context window, that means it can process at most 128,000 tokens of combined prompt and completion. Cost at most providers is billed per input token and per output token, usually at different rates.

This creates a budget problem you need to manage explicitly in production. Here is a TypeScript implementation of a simple token budget tracker:

import Tiktoken from "tiktoken";

interface TokenBudget {
  contextLimit: number;
  inputCostPerMillion: number;
  outputCostPerMillion: number;
  encoding: string;
}

interface BudgetAllocation {
  system: number;
  history: number;
  context: number; // RAG chunks or injected content
  userMessage: number;
  reservedForOutput: number;
}

class TokenBudgetManager {
  private enc: ReturnType<typeof Tiktoken.get_encoding>;
  private budget: TokenBudget;

  constructor(budget: TokenBudget) {
    this.budget = budget;
    this.enc = Tiktoken.get_encoding(budget.encoding as Parameters<typeof Tiktoken.get_encoding>[0]);
  }

  count(text: string): number {
    return this.enc.encode(text).length;
  }

  // Returns how many tokens remain for RAG context given fixed allocations
  availableForContext(
    system: string,
    history: Array<{ role: string; content: string }>,
    userMessage: string,
    reservedForOutput: number
  ): number {
    const systemTokens = this.count(system);
    const historyTokens = history.reduce(
      (sum, msg) => sum + this.count(msg.content) + 4, // 4 tokens overhead per message
      0
    );
    const userTokens = this.count(userMessage) + 4;

    const used = systemTokens + historyTokens + userTokens + reservedForOutput;
    return Math.max(0, this.budget.contextLimit - used);
  }

  // Trim RAG chunks to fit within token budget, preserving highest-scored first
  fitChunks(
    chunks: Array<{ text: string; score: number }>,
    tokenBudget: number
  ): Array<{ text: string; score: number }> {
    const sorted = [...chunks].sort((a, b) => b.score - a.score);
    const result: Array<{ text: string; score: number }> = [];
    let remaining = tokenBudget;

    for (const chunk of sorted) {
      const tokens = this.count(chunk.text);
      if (tokens <= remaining) {
        result.push(chunk);
        remaining -= tokens;
      }
    }

    return result;
  }

  estimateCost(inputTokens: number, outputTokens: number): number {
    return (
      (inputTokens / 1_000_000) * this.budget.inputCostPerMillion +
      (outputTokens / 1_000_000) * this.budget.outputCostPerMillion
    );
  }
}

// Example: GPT-4o budget configuration
const manager = new TokenBudgetManager({
  contextLimit: 128_000,
  inputCostPerMillion: 2.5,
  outputCostPerMillion: 10.0,
  encoding: "o200k_base",
});

const systemPrompt = "You are an expert assistant. Answer based only on the provided context.";
const userMessage = "What is the refund policy for enterprise contracts?";
const conversationHistory = [
  { role: "user", content: "Can you help me understand the contract terms?" },
  { role: "assistant", content: "Of course. What specifically would you like to know?" },
];

const availableForRAG = manager.availableForContext(
  systemPrompt,
  conversationHistory,
  userMessage,
  1_000 // reserve 1K tokens for output
);

console.log(`Available for RAG context: ${availableForRAG} tokens`);
// Approximately 127,900 tokens available in this case

The reservedForOutput parameter is important. If you fill the context window to capacity with input, the model has no room to generate a response. Always reserve headroom.

RAG Chunk Sizing

In retrieval-augmented generation, you split a document corpus into chunks, embed those chunks, and retrieve the top-k chunks at query time. Chunk size is one of the most consequential decisions in a RAG pipeline, and it is measured in tokens.

The tension: smaller chunks improve retrieval precision (each chunk covers a tighter semantic area), but a small chunk may lack enough context to be useful when injected into a prompt. Larger chunks carry more context but retrieve noisier results.

Common starting points: 256-512 tokens for dense factual content (documentation, product specs), 512-1024 tokens for narrative content (policies, articles). Overlap of 10-15% between adjacent chunks is standard to avoid losing context at chunk boundaries.

The chunk size you choose should account for the tokenizer of your embedding model, not just your generation model. Many embedding models (text-embedding-3-small, for instance) have a 8,191-token input limit. If your generation model uses cl100k_base and your embedding model uses the same encoding, chunk sizes transfer cleanly. If they differ, you need to count tokens for both separately.

function chunkDocument(
  text: string,
  chunkSize: number,
  overlap: number,
  enc: ReturnType<typeof Tiktoken.get_encoding>
): string[] {
  const allTokens = enc.encode(text);
  const chunks: string[] = [];
  const decoder = new TextDecoder();
  let start = 0;

  while (start < allTokens.length) {
    const end = Math.min(start + chunkSize, allTokens.length);
    const chunkTokens = allTokens.slice(start, end);
    chunks.push(decoder.decode(enc.decode(chunkTokens)));
    start += chunkSize - overlap;
  }

  return chunks;
}

Chunking on token boundaries rather than character boundaries ensures your chunks are exactly the size you specified. Chunking on characters and then counting tokens after the fact produces chunks that vary in token count, which makes budget management unreliable.

Multilingual Efficiency and Cost Implications

If you operate in multilingual markets, tokenization efficiency is a real cost driver. A customer support system handling Spanish inquiries pays roughly the same as English. A system handling Japanese or Arabic pays measurably more per unit of meaning communicated.

This compounds across the full stack. A Japanese user query that is 15 tokens in Japanese might convey the same information as an 8-token English query. Your RAG retrieval pulls chunks that are also in Japanese, costing more tokens. The output is in Japanese, costing more output tokens. At scale, this multiplier produces billing surprises if you sized your cost model against English-only benchmarks.

The mitigation is not to switch languages. It is to budget correctly: benchmark token counts per language in your system, apply per-language multipliers to your cost projections, and size your chunk limits per language if the gap is large enough to matter.

Tradeoffs at a Glance

Dimension	BPE (tiktoken)	SentencePiece Unigram	Character-level
Vocabulary learning	Greedy merge from byte pairs	Probabilistic pruning from large initial vocab	None (fixed alphabet)
Handling of unknown text	Falls back to byte-level	Falls back to character-level	Always character-level
Whitespace treatment	Excluded from tokens (presplit regex)	Included as part of tokens	Included
Multilingual efficiency	Moderate (English-biased merges)	Better (language-agnostic objective)	Poor (long sequences)
Tokenization consistency for partial strings	Splits can change at concatenation points	Splits can change at concatenation points	Stable
Adoption in mainstream models	GPT-4, GPT-4o, Claude, Mistral	LLaMA 3, Gemma, T5, mT5	Rare in large models
Token count predictability	High (deterministic given merge table)	High (deterministic)	Exact

Production Considerations

Count tokens before you send, not after. Tiktoken is synchronous, fast, and runs locally. There is no reason to discover a prompt is too long when the API call fails. Count tokens at construction time and truncate or compress before sending.

Use the right encoding for the model. cl100k_base for GPT-4 and GPT-3.5-turbo, o200k_base for GPT-4o. Using the wrong encoding gives token counts that differ by 10-20% for the same string. If you are switching models, update the encoding in your budget manager.

Account for message overhead. The chat completions format wraps each message in a structure that consumes additional tokens. OpenAI’s guidance: add 4 tokens per message (for role, content boundaries, and turn delimiters), and add 2 tokens to the total for the reply priming. If you omit this, your count is consistently short.

Structure your prompts to be token-efficient. Verbose system prompts cost tokens on every request. A system prompt that is 800 tokens rather than 200 costs an extra 600 tokens per call. At 1M calls per month, that is 600B extra input tokens. The fix is not to make prompts worse — it is to audit them for redundancy and compress where you can.

Monitor token distributions in production. Track p50, p90, and p99 input token counts per request type. Spikes in token counts often indicate prompt construction bugs (loops that append context repeatedly) or user input patterns you did not anticipate. An alert on p99 input tokens crossing 80% of your context limit gives you early warning before truncation starts affecting responses.

Do not assume token counts are stable across model updates. OpenAI has changed tokenizers between model versions. When you upgrade a model version, re-run your token count benchmarks and verify that your budget assumptions still hold.

The tokenizer sits at the boundary between human language and model computation. It determines what the model can see, how much it costs, and whether your context fits. Treating it as a black box is a choice with concrete dollar and quality consequences. Understanding it takes one afternoon and saves you from a class of production problems that are otherwise surprisingly hard to diagnose.