Fine-Tuning vs RAG: How to Choose the Right LLM Customization Strategy

The default recommendation in most LLM tutorials is: “Use RAG to add knowledge, fine-tune to change behavior.” That framing is mostly right, but it glosses over enough nuance that teams make expensive mistakes from it. Teams RAG their way into prompt bloat when they should have fine-tuned. Teams fine-tune on factual data and then wonder why the model hallucinates updated facts six months later.

This article is a practical breakdown of both approaches: what each actually does at the architecture level, where each breaks down, real cost and latency numbers to set expectations, TypeScript implementation examples for both paths, and a decision framework you can apply to a real problem today.

The Core Problem Each Approach Solves

Before comparing implementation, you need to be clear about what you are trying to change in the model.

A base LLM has two things baked into its weights: factual knowledge absorbed from training data, and behavioral patterns (tone, reasoning style, output format, domain vocabulary). These are not stored separately in the architecture. They are entangled in the same parameter space. But they are conceptually distinct, and that distinction is the key to choosing the right customization approach.

RAG adds facts at inference time. You retrieve relevant documents, inject them into the prompt, and the model reasons over them. The model’s weights do not change. Everything the model “knows” from your data exists only inside the context window of a single request.

Fine-tuning changes the weights. You update the model’s parameters using supervised examples. You are shifting the probability distributions inside the network. This is how you change how the model responds, not what documents it can look up.

That distinction has direct architectural consequences. RAG is inherently stateless per-request and scales horizontally. Fine-tuning produces a new model artifact that you then serve, typically at higher infrastructure cost. Getting this wrong wastes money and engineering time.

RAG Architecture: What Is Actually Happening

At its core, a RAG pipeline is a two-stage retrieval-augmented completion:

1. Retrieval: embed the user query, search a vector store for semantically similar chunks, optionally apply metadata filters and reranking.
2. Generation: construct a prompt that includes the retrieved chunks as context, send to the LLM, return the response.

The model itself is untouched. You are using its reasoning capability to synthesize information you provide at runtime.

import Anthropic from "@anthropic-ai/sdk";

interface Chunk {
  content: string;
  source: string;
  score: number;
}

async function ragQuery(
  userQuery: string,
  retrievedChunks: Chunk[],
): Promise<string> {
  const client = new Anthropic();

  const context = retrievedChunks
    .map((c, i) => `[Source ${i + 1}: ${c.source}]\n${c.content}`)
    .join("\n\n---\n\n");

  const prompt = `You are answering questions using the provided documents only.
If the answer is not in the documents, say "I don't have that information."
Do not add details beyond what is in the documents.

${context}

Question: ${userQuery}

Answer:`;

  const response = await client.messages.create({
    model: "claude-3-5-sonnet-20241022",
    max_tokens: 1024,
    messages: [{ role: "user", content: prompt }],
  });

  return response.content[0].type === "text" ? response.content[0].text : "";
}

The strengths of this approach are real: your knowledge base is updatable without retraining, you can inspect exactly what context the model used, and you can swap the underlying LLM without losing your knowledge corpus. The weaknesses are equally real: retrieval can miss, the context window caps how much knowledge a single response can use, and complex reasoning chains across many documents are fragile.

Fine-Tuning Architecture: What Is Actually Happening

Fine-tuning takes a pre-trained model and continues training it on a new, smaller dataset. You are not training from scratch. You are nudging the existing weight distribution toward behaviors present in your training examples.

The most common technique for production fine-tuning today is LoRA (Low-Rank Adaptation). Instead of updating all billions of parameters, LoRA inserts small trainable matrices at specific layers and trains only those. This reduces the number of trainable parameters by 10-1000x, dramatically cutting training cost and time, while preserving most of the base model’s capability.

The key inputs are:

• A base model (open-source is the practical path: Llama 3, Mistral, Phi-3, Qwen).
• A training dataset of (input, output) pairs that demonstrate the behavior you want.
• Compute: GPU hours, either on your own hardware or rented from a cloud provider.

// This is what your training data structure looks like.
// the actual training loop runs in Python/PyTorch, not TypeScript.
// But you will generate, validate, and ship these examples from your TS stack.

interface TrainingExample {
  messages: Array<{
    role: "system" | "user" | "assistant";
    content: string;
  }>;
}

function buildTrainingExample(
  systemPrompt: string,
  userInput: string,
  idealOutput: string,
): TrainingExample {
  return {
    messages: [
      { role: "system", content: systemPrompt },
      { role: "user", content: userInput },
      { role: "assistant", content: idealOutput },
    ],
  };
}

// Validate dataset before submitting to training
function validateDataset(examples: TrainingExample[]): {
  valid: boolean;
  errors: string[];
} {
  const errors: string[] = [];

  for (const [i, ex] of examples.entries()) {
    if (ex.messages.length < 2) {
      errors.push(`Example ${i}: too few messages`);
    }

    const lastMsg = ex.messages[ex.messages.length - 1];
    if (lastMsg.role !== "assistant") {
      errors.push(`Example ${i}: last message must be from assistant`);
    }

    const assistantContent = ex.messages
      .filter((m) => m.role === "assistant")
      .map((m) => m.content)
      .join(" ");

    if (assistantContent.length < 20) {
      errors.push(`Example ${i}: assistant output suspiciously short`);
    }
  }

  return { valid: errors.length === 0, errors };
}

After training completes, you have a model (or a set of LoRA adapter weights that merge with the base model at serving time). Every inference call after that reflects the behavioral changes baked in during training. No retrieval step, no context injection.

Tradeoffs: Real Numbers

Abstract comparisons are not useful. Here are concrete numbers based on real production deployments to calibrate expectations.

Dimension	RAG	Fine-Tuning
Setup cost	Low: embeddings, vector store, pipeline code	High: dataset curation (200-1000+ examples minimum), GPU hours
Training cost	None	$50-$500 for a 7B model with LoRA on consumer cloud GPUs; $1,000-$10,000 for larger models
Inference latency added	50-300ms for retrieval + reranking, before LLM call	Zero: no retrieval step; total latency is just the LLM call
Knowledge freshness	Real-time: update vector store, no retraining	Stale: requires retraining to update facts
Knowledge capacity	Unbounded: corpus can be terabytes	Limited to what fits in training distribution
Behavioral change	Weak: prompt-based, inconsistent, easy to override	Strong: baked into weights, consistent across inputs
Factual accuracy	High for in-corpus facts	Prone to hallucination on facts not well-represented in training data
Auditability	High: retrieved chunks are inspectable	Low: behavior emerges from weights, not inspectable
Infrastructure complexity	Vector store, embedding pipeline, retrieval logic	Model serving (GPU required), possibly adapter merging

The latency number is worth unpacking. RAG adds 50-300ms of retrieval overhead per request before the LLM even starts generating. At p99, that can be 400-600ms. Fine-tuned model inference has no retrieval step, so total latency is purely generation time. For latency-sensitive applications, this matters.

The cost comparison flips depending on volume. Fine-tuning is a one-time cost. RAG has ongoing per-request costs for embedding and vector search (small, but real). At high QPS with a stable knowledge domain, fine-tuning can be cheaper over 12 months despite the higher upfront cost.

The Behavioral vs. Factual Knowledge Split

This is the axis that most cleanly separates when to use each approach.

Behavioral knowledge is how the model should act: its tone, its domain vocabulary, its reasoning patterns, what it should never say, how it should format outputs. Teaching a model to always respond in JSON with a specific schema, to use your internal API naming conventions, to adopt the communication style of your support team, to consistently refuse certain categories of requests: all of this is behavioral.

Fine-tuning is the right tool for behavioral knowledge. You can engineer this into a prompt to some extent, but prompts are brittle. Users can accidentally override them. The model’s base tendencies leak through under adversarial or edge-case inputs. A fine-tuned behavioral pattern is load-bearing in a way that a system prompt is not.

Factual knowledge is what the model should know: your product documentation, your company’s internal procedures, a legal corpus, a technical specification. Anything that is represented as documents, that changes over time, and where you need to point to a source when the model gets it wrong.

RAG is the right tool for factual knowledge. The documents are inspectable. When the model says something wrong, you can debug the retrieval layer. When the documents change, you update the vector store. Fine-tuning factual knowledge into weights is a path to hard-to-debug hallucinations and expensive retraining cycles every time facts change.

The overlap is real, and it is where teams get into trouble. Some use cases involve both: a customer support system that needs to respond in a specific tone (behavioral) and answer questions about a frequently updated product catalog (factual). The right answer there is often both: fine-tune for behavior, RAG for facts.

Implementing a Fine-Tuned Model Pipeline in TypeScript

Once you have a fine-tuned model hosted (whether self-hosted with vLLM, or via a provider that supports custom model hosting), the inference call from TypeScript is nearly identical to calling a base model. The behavioral changes are already in the weights.

import OpenAI from "openai";

// vLLM exposes an OpenAI-compatible API, so the same client works
// whether you are calling a base model or your fine-tuned variant.
const client = new OpenAI({
  baseURL: process.env.VLLM_BASE_URL ?? "http://localhost:8000/v1",
  apiKey: process.env.VLLM_API_KEY ?? "placeholder",
});

interface SupportTicket {
  subject: string;
  body: string;
  customerTier: "free" | "pro" | "enterprise";
}

interface ClassifiedTicket {
  category: string;
  priority: "low" | "medium" | "high" | "critical";
  suggestedResponse: string;
}

async function classifyTicket(
  ticket: SupportTicket,
): Promise<ClassifiedTicket> {
  // No system prompt needed for tone or format (that is in the weights).
  // Just provide the task input.
  const response = await client.chat.completions.create({
    model: process.env.FINE_TUNED_MODEL_ID ?? "ft-support-classifier-v2",
    messages: [
      {
        role: "user",
        content: JSON.stringify({
          subject: ticket.subject,
          body: ticket.body,
          tier: ticket.customerTier,
        }),
      },
    ],
    response_format: { type: "json_object" },
    temperature: 0.1, // Low temperature for classification tasks
  });

  const raw = response.choices[0].message.content ?? "{}";
  return JSON.parse(raw) as ClassifiedTicket;
}

The notable difference from RAG: there is no context injection, no retrieval latency, and no prompt engineering for tone or format. The model already knows what output structure you expect because it was trained on examples of that structure.

Implementing a Hybrid Strategy

For the use cases that need both, the architecture is straightforward: fine-tune the model for behavioral patterns, then use RAG to supply it with current factual context.

async function hybridQuery(
  userQuery: string,
  retrievedChunks: Chunk[],
): Promise<string> {
  const client = new OpenAI({
    baseURL: process.env.VLLM_BASE_URL,
    apiKey: process.env.VLLM_API_KEY ?? "placeholder",
  });

  // Context injection still happens, but the system prompt is minimal
  // because tone, format, and domain vocabulary are in the weights.
  const context = retrievedChunks
    .map((c) => `[${c.source}]\n${c.content}`)
    .join("\n\n");

  const response = await client.chat.completions.create({
    model: process.env.FINE_TUNED_MODEL_ID ?? "ft-support-v2",
    messages: [
      {
        role: "system",
        content:
          "Use the provided documents to answer the question. Stay within the documents.",
      },
      {
        role: "user",
        content: `Documents:\n${context}\n\nQuestion: ${userQuery}`,
      },
    ],
    temperature: 0.2,
  });

  return response.choices[0].message.content ?? "";
}

The system prompt here is minimal because the model already knows your domain language and response format from training. You are only using RAG’s retrieval layer for current facts.

Decision Framework

Apply these questions in order:

1. What kind of knowledge are you adding?

If it is facts that exist in documents and might change: start with RAG. If it is behavioral patterns (tone, output format, domain vocabulary, refusal policies): fine-tuning is the right path.

2. How often does the knowledge change?

If the knowledge changes more than once a month, fine-tuning becomes operationally expensive. You will be running training jobs on a recurring basis to keep facts current. RAG handles this with a vector store update.

3. Can you tolerate retrieval failures?

RAG can miss. If your application is in a domain where “I don’t have that information” is an unacceptable answer (emergency response, medical triage, financial compliance), the retrieval dependency is a hard liability. A fine-tuned model with known behavior boundaries may be safer.

4. What is your volume and latency budget?

At high volume with stable knowledge, the economics favor fine-tuning after the break-even point (roughly 6-12 months of RAG operational costs depending on corpus size). At lower volume or with frequently changing content, RAG stays cheaper indefinitely.

5. Do you need auditability?

If regulators, lawyers, or customers need to know why the model said what it said, RAG’s inspectable retrieval chain is a significant advantage. Fine-tuned model behavior is not easily explainable at the weight level.

A summary of the decision paths:

Scenario	Recommendation
Adding product documentation that updates weekly	RAG only
Teaching the model a specific output schema	Fine-tuning only
Customer support with changing policies and branded tone	Fine-tuning for tone + RAG for policy
Internal code assistant with company conventions	Fine-tuning only
Legal research over a large, stable corpus	RAG only
Low-latency classification at high QPS	Fine-tuning only
Knowledge base Q&A with audit requirements	RAG only
Domain-specific reasoning patterns (medical, legal)	Fine-tuning, possibly with RAG for current facts

Production Considerations

Dataset quality matters more than dataset size for fine-tuning. 500 high-quality, diverse examples outperform 5,000 noisy or redundant ones. The most common failure mode is training on examples that do not actually represent the full distribution of inputs the model will see in production. Build your training set from real production inputs where possible.

Monitor your fine-tuned model for behavioral drift. Unlike a RAG pipeline where you can inspect what the model was given, a fine-tuned model can surprise you. Run an evaluation suite before every deployment. Track outputs on a fixed regression set across model versions.

RAG pipelines degrade as the corpus ages. The embedding model you used at ingest time is the embedding model you must use at query time. If you update your embedding model, you re-index from scratch. Plan for this at schema design time: keep your source documents separate from your chunk index, track chunk provenance with a relational mapping, and script the re-indexing process before you need it.

Serving fine-tuned models requires GPU infrastructure. LoRA adapters are small (often under 1GB), but you still need the full base model in VRAM at serving time. A 7B parameter model requires roughly 14GB VRAM in fp16 or 7GB in 4-bit quantization. Plan serving costs before committing to fine-tuning as your primary path.

Closing Insight

The teams that get into trouble treat this as a technology choice when it is actually a knowledge architecture question. Before you pick a tool, ask what kind of knowledge you are adding and who owns keeping it current.

If the answer is “it lives in documents and a human maintains them,” RAG is almost certainly the right path. If the answer is “it is a pattern in how experts respond and we can capture it in labeled examples,” fine-tuning is the right path. If the answer involves both, build both layers and keep them cleanly separated.

The worst outcome is fine-tuning factual knowledge into weights and then debugging hallucinations about facts that changed six months ago. The second worst is using RAG to teach a model behavioral patterns and then fighting prompt injections and tone inconsistency in production. Getting the knowledge type right upfront saves weeks of downstream debugging.