LLM Fine-Tuning Infrastructure: Training Pipelines, Dataset Management, and Deployment Strategies for Custom Models

Fine-tuning sounds straightforward until you run it in production. The first training run is the easy part. The hard parts are managing dataset versions across experiments, ensuring your eval set does not leak into training, building a rollback path when the fine-tuned model regresses on edge cases, and not spending $2,000 on a GPU run that needed three more hours of data cleaning before it started.

This article covers the full pipeline: deciding whether fine-tuning is actually the right move, preparing and versioning datasets, running LoRA/QLoRA training jobs, evaluating against the base model, deploying through a model registry, and serving fine-tuned models at production latency targets.

When Fine-Tuning Actually Makes Sense

The honest answer is: less often than you think, and later than most teams attempt it.

Start with prompt engineering. A well-structured system prompt with a few-shot template covers 80% of behavior customization requirements. It is free to iterate, takes minutes to deploy, and requires no GPU. If you can solve the problem with prompting, do that first.

RAG covers most of the remaining cases. When the model needs to know facts it was not trained on (your product docs, internal policies, recent events), retrieval is the right primitive. Fine-tuning does not teach a model facts reliably; it teaches the model a style, format, or reasoning pattern.

Fine-tuning makes sense when:

• The task requires consistent output format that few-shot examples do not enforce reliably (structured extraction from noisy documents, domain-specific code generation).
• You have a specific reasoning pattern, tone, or domain vocabulary that is underrepresented in the base model’s training data.
• You are paying token costs on a high-volume endpoint and a smaller fine-tuned model can match quality at a fraction of the inference cost.
• Latency requirements exclude the large models that handle the task well with prompting alone.

Fine-tuning does not make sense when:

• Your dataset is under 500 examples. Below this, the noise overwhelms the signal. Collect more data or use RAG.
• The task changes frequently. A fine-tuned model is not easy to update. If your domain knowledge changes monthly, RAG is more maintainable.
• You have not exhausted prompt engineering. This sounds obvious. Surprisingly many teams skip it.

Method	When to use	Iteration speed	Cost
Prompt engineering	Format, style, few-shot behavior	Minutes	Near zero
RAG	Dynamic knowledge, factual grounding	Hours	Low (retrieval infra)
Fine-tuning	Consistent format, domain reasoning, cost reduction at scale	Days	Medium-high
Full pretraining	New domain knowledge from scratch	Weeks	Very high

Dataset Preparation

Data quality is the only variable that matters before training starts. A clean dataset of 1,000 examples outperforms a dirty dataset of 10,000 every time.

Format and Conversion

Most fine-tuning frameworks use conversational format (Chat Markup Language or the Hugging Face conversations schema). If your source data is not in this format, convert it before versioning.

from datasets import Dataset
from typing import Any

def convert_to_chat_format(
    raw_examples: list[dict[str, Any]],
    system_prompt: str,
) -> list[dict[str, Any]]:
    """
    Convert raw input/output pairs to multi-turn chat format.
    Expects each example to have 'input' and 'expected_output' keys.
    """
    formatted = []
    for ex in raw_examples:
        formatted.append({
            "conversations": [
                {"role": "system", "content": system_prompt},
                {"role": "user", "content": ex["input"]},
                {"role": "assistant", "content": ex["expected_output"]},
            ]
        })
    return formatted

dataset = Dataset.from_list(
    convert_to_chat_format(raw_data, system_prompt=SYSTEM_PROMPT)
)
dataset.save_to_disk("data/processed/v1")

Deduplication

Near-duplicate examples cause overfitting on specific phrasings. Use MinHash LSH for fuzzy deduplication at scale, or cosine similarity on embeddings for smaller datasets.

from datasketch import MinHash, MinHashLSH
import re

def compute_minhash(text: str, num_perm: int = 128) -> MinHash:
    m = MinHash(num_perm=num_perm)
    tokens = set(re.findall(r"\w+", text.lower()))
    for token in tokens:
        m.update(token.encode("utf-8"))
    return m

def deduplicate(examples: list[dict], threshold: float = 0.85) -> list[dict]:
    lsh = MinHashLSH(threshold=threshold, num_perm=128)
    kept = []
    for i, ex in enumerate(examples):
        key = str(i)
        mh = compute_minhash(ex["conversations"][1]["content"])  # user turn
        candidates = lsh.query(mh)
        if not candidates:
            lsh.insert(key, mh)
            kept.append(ex)
    return kept

After deduplication, run a quality filter. The minimum bar is: non-empty assistant responses, minimum token length on both turns, no obvious templating artifacts (responses that end with {placeholder} or similar).

Train/Val/Test Splits

The split order matters. Shuffle first, then split. Never create splits after any form of filtering that might introduce correlation between splits.

Use an 80/10/10 ratio for datasets under 5,000 examples. For larger datasets, you can afford a smaller validation and test set in absolute terms (500-1,000 examples each). The test set must be held out completely until final evaluation; it is not for hyperparameter tuning.

Data contamination check: compute pairwise similarity between your test set and training set. Any test example with cosine similarity above 0.92 to a training example should be moved or discarded.

import numpy as np
from sentence_transformers import SentenceTransformer

def check_contamination(
    train_texts: list[str],
    test_texts: list[str],
    threshold: float = 0.92,
) -> list[int]:
    """Returns indices of contaminated test examples."""
    model = SentenceTransformer("all-MiniLM-L6-v2")
    train_embeddings = model.encode(train_texts, batch_size=64, normalize_embeddings=True)
    test_embeddings = model.encode(test_texts, batch_size=64, normalize_embeddings=True)

    contaminated = []
    for i, test_emb in enumerate(test_embeddings):
        sims = np.dot(train_embeddings, test_emb)
        if sims.max() > threshold:
            contaminated.append(i)
    return contaminated

Dataset Versioning with DVC

Once you have a clean dataset, version it. Raw JSON in a git repo does not scale. Use DVC to track dataset artifacts separately from code, backed by S3 or GCS.

# Initialize DVC in your training repo
dvc init
dvc remote add -d s3remote s3://your-bucket/dvc-cache

# Track a processed dataset
dvc add data/processed/v1
git add data/processed/v1.dvc .gitignore
git commit -m "add processed training dataset v1"
dvc push

In your training script, pin the dataset version by its DVC hash. This ties a specific model checkpoint to the exact data it was trained on, which matters when you are debugging a regression six weeks later.

# In CI, reproduce a specific dataset version
dvc pull data/processed/v1

Tag each dataset version with a metadata file describing the source, filters applied, split sizes, deduplication threshold, and creation date. This is not optional for teams with more than one person running experiments.

Training: LoRA and QLoRA

Full fine-tuning updates all model weights. For a 7B parameter model in bfloat16, that is 14GB of weights plus optimizer states (Adam needs 2x weight storage for first and second moments, so ~42GB GPU memory minimum before activations). This requires A100 80GB or H100 territory.

LoRA (Low-Rank Adaptation) inserts small trainable matrices into the attention layers instead. For a 7B model with rank 16 adapters, you train roughly 20-40M parameters instead of 7B. This reduces GPU memory requirements to A100 40GB or even 2-4x A10G instances for 7B models.

QLoRA adds 4-bit quantization to the frozen base model, which gets the 7B model down to around 4GB of base weight memory. The total training footprint fits on a single A10G 24GB for models up to 13B parameters.

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model, TaskType
import torch

# QLoRA: load base model in 4-bit
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    quantization_config=bnb_config,
    device_map="auto",
)

# LoRA adapter config
lora_config = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=16,                          # rank: higher = more capacity, more memory
    lora_alpha=32,                 # scaling factor; alpha/r is the effective LR multiplier
    target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
    lora_dropout=0.05,
    bias="none",
)

model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# trainable params: 20,185,088 || all params: 8,051,232,768 || trainable%: 0.25

Key hyperparameter decisions for LoRA:

• Rank (r): 8-16 for format adaptation. 32-64 for more complex reasoning tasks. Higher rank increases capacity but also overfitting risk.
• Target modules: At minimum, q_proj and v_proj. Adding k_proj, o_proj, and MLP layers increases capacity; benchmark whether they add value for your task.
• Learning rate: 2e-4 is a reasonable starting point for QLoRA. Full fine-tuning starts lower at 1e-5 to 2e-5.
• Epochs: 1-3 for large datasets. More than 3 epochs on small datasets almost always overfits.

Multi-GPU Training

For 13B+ models or larger datasets, you need multi-GPU. Hugging Face Accelerate handles this without rewriting your training script.

# launch_config.yaml
compute_environment: LOCAL_MACHINE
distributed_type: MULTI_GPU
num_machines: 1
num_processes: 4       # one per GPU
gpu_ids: 0,1,2,3
mixed_precision: bf16

accelerate launch --config_file launch_config.yaml train.py

For batch jobs on cloud GPUs, prefer DeepSpeed ZeRO-3 for models above 30B. ZeRO-3 shards optimizer states, gradients, and parameters across GPUs, which is the only way to fit a 70B model on a 4x A100 80GB node without model parallelism.

Compute Provisioning

GPU spot instances cut training costs by 60-70% compared to on-demand. The trade-off is interruption: spot instances can be reclaimed with 2 minutes of notice. Mitigate this with checkpoint saves every 100-200 steps and resume logic in your training loop.

from transformers import TrainerCallback

class CheckpointOnInterruptCallback(TrainerCallback):
    def on_step_end(self, args, state, control, **kwargs):
        if state.global_step % 100 == 0:
            control.should_save = True
        return control

Cost comparison for a 7B LoRA fine-tuning run (10,000 examples, 3 epochs, ~6 hours):

GPU	Provider	On-demand/hr	Spot/hr	Est. total (spot)
A10G 24GB	AWS (g5.xlarge)	$1.01	~$0.35	~$2.10
A100 40GB	Lambda Labs	$1.29	N/A	~$7.74
A100 80GB	AWS (p4d)	$3.06	~$1.10	~$6.60
H100 80GB	CoreWeave	$2.79	~$1.20	~$7.20

For 7B QLoRA, a single A10G is sufficient. For 13B QLoRA, two A10Gs work. For 70B LoRA (not QLoRA), you need 4x A100 80GB minimum. Use spot for training, on-demand for inference.

Evaluation

Evaluating a fine-tuned model is harder than evaluating general capabilities. You need three layers:

Automated benchmarks: Task-specific metrics that you define before training starts. For classification: F1, precision, recall. For structured extraction: exact match on fields, partial credit for near-matches. For generation: ROUGE scores are a rough proxy; LLM-as-judge is better for open-ended tasks.

Regression against the base model: The fine-tuned model should not regress on capabilities you care about outside the training domain. Run the base model’s standard benchmarks (or a subset) against the fine-tuned adapter. A model that gets better at extraction but loses general instruction-following is not a net win.

Human evaluation: Automated metrics miss things. For the first few fine-tuning runs, have domain experts rate 50-100 outputs on a 1-5 scale. This calibrates whether your automated metrics are tracking what actually matters.

from openai import OpenAI
from typing import Literal

client = OpenAI()

def llm_judge_score(
    prompt: str,
    reference: str,
    model_output: str,
    criteria: str,
) -> dict[str, int | str]:
    """
    LLM-as-judge for open-ended generation tasks.
    Returns a score (1-5) and rationale.
    """
    judge_prompt = f"""You are evaluating model output quality.

Criteria: {criteria}

User prompt: {prompt}
Reference answer: {reference}
Model output: {model_output}

Score the model output 1-5 where:
1 = completely wrong or unhelpful
3 = partially correct, missing key details
5 = accurate, complete, and matches the reference closely

Respond in JSON: {{"score": <int>, "rationale": "<string>"}}"""

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": judge_prompt}],
        response_format={"type": "json_object"},
        temperature=0,
    )
    import json
    return json.loads(response.choices[0].message.content)

Track scores across training runs in a structured log, not a spreadsheet. Git commit the eval results alongside the model checkpoint reference.

Model Registry and Versioning

A model registry is a database of checkpoints with metadata: base model, dataset version, training config, eval scores, and deployment status. Hugging Face Hub works for teams comfortable with public or private repos. For internal infrastructure, MLflow Model Registry or a simple S3 + DynamoDB setup is sufficient.

The minimum metadata record per checkpoint:

interface ModelCheckpoint {
  id: string;                    // e.g., "llama-3.1-8b-v1.2.0"
  baseModel: string;             // "meta-llama/Llama-3.1-8B-Instruct"
  adapterPath: string;           // S3 URI to LoRA adapter weights
  datasetVersion: string;        // DVC hash or dataset tag
  trainingConfig: {
    loraRank: number;
    learningRate: number;
    epochs: number;
    batchSize: number;
  };
  evalScores: Record<string, number>;  // e.g., { "f1": 0.87, "exactMatch": 0.74 }
  baselineScores: Record<string, number>; // base model scores on same eval set
  status: "candidate" | "staging" | "production" | "deprecated";
  createdAt: string;
  deployedAt?: string;
}

Never put a checkpoint into production without an associated eval record. This is a policy, not a technical constraint, but enforcing it at the registry level prevents shortcuts under deadline pressure.

Deployment: Serving Fine-Tuned Models

LoRA adapters are not standalone models. You need the base model loaded plus the adapter weights merged or applied at inference time. There are two approaches:

Merge at deployment: Use peft’s merge_and_unload() to fuse the adapter into the base model weights. The result is a standard model file, compatible with any serving framework. This is the simpler path if you are not switching adapters dynamically.

from peft import PeftModel
from transformers import AutoModelForCausalLM

base_model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    torch_dtype="bfloat16",
)
model = PeftModel.from_pretrained(base_model, "path/to/lora-adapter")
merged = model.merge_and_unload()
merged.save_pretrained("merged-model/")

Load adapter at serving time: vLLM supports LoRA adapters as a first-class concept via --enable-lora. This lets you serve the base model and multiple adapters from one GPU, switching adapters per request. Useful when you have multiple fine-tuned variants for different tasks.

docker run --runtime nvidia --gpus all \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --enable-lora \
  --lora-modules task-a=/adapters/task-a,task-b=/adapters/task-b \
  --max-lora-rank 64

Requests then specify the adapter via the model field:

const response = await openai.chat.completions.create({
  model: "task-a",  // routes to the task-a LoRA adapter
  messages: [{ role: "user", content: userMessage }],
});

A/B Testing Between Base and Fine-Tuned Models

Route a percentage of traffic to the fine-tuned model and compare quality metrics against the base model in production. The evaluation signal in production is noisier than your offline eval, but it catches real distribution shift.

import { createHash } from "crypto";

function selectModelVariant(
  userId: string,
  finetuneRolloutPct: number,
): "base" | "finetuned" {
  const bucket = parseInt(
    createHash("sha256").update(userId + "model-ab-v1").digest("hex").slice(0, 4),
    16,
  ) % 100;
  return bucket < finetuneRolloutPct ? "finetuned" : "base";
}

const modelEndpoint =
  selectModelVariant(userId, 10) === "finetuned"
    ? process.env.FINETUNED_MODEL_URL
    : process.env.BASE_MODEL_URL;

Start at 5-10% traffic. Monitor latency, error rate, and any user-facing quality signal (thumbs up/down, task completion rate, downstream conversion). Hold at each traffic level for 24-48 hours before increasing. The deterministic bucket hash ensures the same user consistently hits the same variant, which matters for tasks with session context.

Rollback Strategy

Fine-tuned models need a fast rollback path. If your serving stack uses vLLM with adapter switching, rollback is updating a config value and reloading the adapter list. If you deployed a merged model, rollback means routing traffic back to the base model endpoint or a previous checkpoint.

Keep the previous production checkpoint’s merged weights in storage. Rollback should take under five minutes and require no re-training.

Define rollback triggers before deployment:

• Error rate increase greater than 0.5% compared to the 24-hour pre-deployment baseline
• P95 latency regression greater than 20%
• Manual trigger from any engineer with production access

Log rollbacks with a reason code. If you roll back more than twice from the same checkpoint, investigate the eval process rather than the model. The eval missed something real.

Decision Framework

Situation	Recommendation
Task works with a 5-shot prompt	Ship the prompt. Do not fine-tune.
Task needs dynamic knowledge	Use RAG.
Task needs consistent structure, dataset exists and is clean	Fine-tune with LoRA/QLoRA
Model size too large for latency budget	Fine-tune a smaller model to match quality
Dataset under 500 examples	Collect more data first
Frequent domain changes (weekly)	RAG over fine-tuning
Multiple task variants, shared base model	vLLM multi-adapter serving
Cost reduction at high throughput	Fine-tune 7B to replace 70B calls

Fine-tuning infrastructure is not complicated, but it has a lot of pieces that need to work together: data versioning, deduplication, contamination checks, multi-GPU orchestration, eval tracking, and a deployment pipeline with rollback. Teams that skip steps in the middle spend weeks debugging regressions that were caused by a dirty dataset or an eval set that leaked into training. The infrastructure overhead is the bulk of the work. The actual training is the easy part.