Deploying Open-Source LLMs in Production: Model Serving, Quantization, and Infrastructure Choices

The pitch for self-hosted LLMs sounds simple: avoid per-token billing, keep data off third-party infrastructure, tune the model for your use case. The reality involves GPU provisioning decisions with real cost consequences, serving stack choices that affect P99 latency by 3x or more, and quantization tradeoffs that trade accuracy for throughput in non-obvious ways.

This article covers the full stack: picking a serving framework, choosing a quantization format, provisioning GPUs, tuning batching behavior, load balancing across inference nodes, and building observability into the system. Code examples use the HTTP APIs you will actually interact with in production, plus Pulumi infrastructure definitions.

When Self-Hosting Actually Makes Sense

Start with the honest version of the build-vs-buy analysis. Self-hosting is not automatically cheaper. At low volumes, managed API providers win on total cost because you avoid idle GPU time, engineering overhead, and ops burden.

Self-hosting becomes competitive when:

• Throughput is high enough that reserved GPU capacity costs less than per-token pricing. For GPT-4o-class models at current pricing, this crossover is roughly 5-10M tokens/day depending on your GPU efficiency.
• Your data cannot leave your infrastructure: healthcare, finance, legal, or specific contractual requirements.
• You need to fine-tune the model, modify sampling behavior, or add custom preprocessing that managed APIs do not expose.
• Latency requirements are sub-100ms P99, and you need to co-locate the model with your application.

If none of these apply, a managed provider is the right call for now. Come back when usage scales.

Choosing a Serving Framework

Three frameworks dominate production deployments: vLLM, Text Generation Inference (TGI), and Ollama. They solve different problems.

vLLM is the right default for production systems. Its PagedAttention implementation handles KV cache memory efficiently, which translates directly to higher throughput under concurrent load. It exposes an OpenAI-compatible API, supports continuous batching, and has native tensor parallelism for multi-GPU setups. Start here unless you have a specific reason not to.

TGI (Hugging Face Text Generation Inference) is a strong alternative with broader model support and good Flash Attention integration. It performs comparably to vLLM on many workloads and is worth benchmarking head-to-head if you are running Mistral or Falcon variants.

Ollama is optimized for developer machines and single-node local deployments. It handles quantized GGUF models well and is simple to operate. It is not the right tool for production inference servers under load: its concurrency model and batching behavior are not tuned for throughput.

Framework	Best For	Concurrency Model	Multi-GPU	OpenAI API Compatible
vLLM	Production inference, high throughput	Continuous batching	Yes (tensor parallel)	Yes
TGI	Broad model support, HuggingFace ecosystem	Continuous batching	Yes	Yes (partial)
Ollama	Local dev, single-node, GGUF models	Limited	No	Yes
llama.cpp server	CPU inference, edge	Request-based	No	Yes

Starting a vLLM Inference Server

A minimal production vLLM deployment on a single A100 80GB looks like this:

docker run --runtime nvidia --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct \
  --tensor-parallel-size 1 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --enable-prefix-caching \
  --api-key your-secret-key

The flags that matter most:

• --gpu-memory-utilization 0.90: Leaves 10% for the CUDA context and overhead. Pushing this to 0.95 gains you more KV cache space but risks OOM under burst load.
• --max-model-len 8192: Capping context length reduces peak KV cache consumption and lets you serve more concurrent requests. If your use case does not need 128K context, do not allocate for it.
• --enable-prefix-caching: Reuses KV cache for identical prompt prefixes across requests. This is a meaningful throughput gain for RAG workloads where system prompts are shared.

Calling it from your application uses the standard OpenAI client:

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "http://your-inference-server:8000/v1",
  apiKey: process.env.INFERENCE_API_KEY,
});

async function generate(prompt: string): Promise<string> {
  const response = await client.chat.completions.create({
    model: "meta-llama/Llama-3.1-8B-Instruct",
    messages: [{ role: "user", content: prompt }],
    max_tokens: 512,
    temperature: 0.1,
  });

  return response.choices[0].message.content ?? "";
}

Quantization: GPTQ, AWQ, and GGUF

Running a 70B model in full BF16 precision requires roughly 140GB VRAM. Two A100 80GBs can hold it with tensor parallelism, but that is expensive. Quantization compresses the model weights to reduce memory footprint, at the cost of some accuracy.

GPTQ (Post-Training Quantization) quantizes weights to 4-bit or 8-bit using activation statistics. 4-bit GPTQ reduces a 70B model to about 35GB, fitting on a single A100. Inference speed is faster than full precision due to reduced memory bandwidth pressure. Accuracy loss for most tasks is small but detectable on reasoning benchmarks.

AWQ (Activation-Aware Weight Quantization) identifies which weights are most sensitive to quantization and preserves their precision. In practice, AWQ at INT4 typically outperforms GPTQ at the same bit width by 1-3 points on MMLU-style benchmarks. If you are going to 4-bit, prefer AWQ.

GGUF is the quantization format used by llama.cpp and Ollama. It supports a range of quantization levels from Q2_K to Q8_0. Q4_K_M is the common production choice: roughly 40% memory reduction versus FP16 with acceptable quality loss. GGUF runs well on CPU and Apple Silicon, which makes it useful for edge or developer tooling. For GPU inference servers, GPTQ and AWQ are generally better fits.

Format	Typical Size (7B)	Typical Size (70B)	Speed vs FP16	Accuracy vs FP16	Runtime
FP16	14 GB	140 GB	1x	baseline	vLLM, TGI
INT8	7 GB	70 GB	~1.1x	-0.5-1%	vLLM, TGI
GPTQ INT4	4 GB	38 GB	~1.3x	-1-3%	vLLM, TGI
AWQ INT4	4 GB	38 GB	~1.4x	-0.5-2%	vLLM
GGUF Q4_K_M	4.1 GB	40 GB	varies	-1-3%	llama.cpp, Ollama

Loading an AWQ model in vLLM requires only setting the quantization flag:

--model TheBloke/Llama-3.1-70B-Instruct-AWQ \
--quantization awq \
--tensor-parallel-size 1

GPU Provisioning and Cost

GPU choice depends on the model size after quantization and your throughput target. The practical options:

• A10G (24GB): Fits 7B FP16 or 13B INT4. Cheapest useful GPU for inference. Available on-demand at around $1.00-1.50/hr.
• A100 40GB: Fits 30B INT4 or 13B FP16. Good balance of VRAM and price. Around $2.50-3.50/hr.
• A100 80GB: Fits 70B INT4 or 30B FP16. Best single-GPU option for mid-size models. Around $3.50-5.00/hr.
• H100 80GB: Roughly 3x the throughput of A100 for transformer inference due to FP8 hardware acceleration. Around $7-10/hr. Worth it for production systems where token throughput is the constraint.

For a 7B model serving a high-traffic product endpoint, a pair of A10Gs behind a load balancer gives good cost-throughput balance. For 70B models, start with a single A100 80GB and add more nodes horizontally.

Infrastructure as code using Pulumi:

import * as aws from "@pulumi/aws";
import * as pulumi from "@pulumi/pulumi";

const config = new pulumi.Config();

const inferenceInstance = new aws.ec2.Instance("llm-inference", {
  ami: "ami-xxxxxxxxxxxxxxxxx", // Deep Learning AMI with CUDA
  instanceType: "g5.xlarge",   // 1x A10G 24GB
  keyName: config.require("keyName"),
  vpcSecurityGroupIds: [inferenceSecurityGroup.id],
  subnetId: privateSubnet.id,
  rootBlockDevice: {
    volumeSize: 200,
    volumeType: "gp3",
  },
  tags: {
    Name: "llm-inference-node",
    Environment: "production",
  },
  userData: pulumi.interpolate`#!/bin/bash
    docker run -d --runtime nvidia --gpus all \
      -v /opt/models:/root/.cache/huggingface \
      -p 8000:8000 \
      --ipc=host \
      --restart unless-stopped \
      vllm/vllm-openai:latest \
      --model meta-llama/Llama-3.1-8B-Instruct \
      --gpu-memory-utilization 0.90 \
      --api-key ${config.requireSecret("inferenceApiKey")}
  `,
});

export const inferencePrivateIp = inferenceInstance.privateIp;

For autoscaling, GPU instance warm-up time (model loading) is 2-5 minutes for 7-13B models and 5-10 minutes for 70B models. Do not rely on reactive autoscaling for latency-sensitive paths. Use predictive scaling or keep a minimum of two instances running.

Batching Strategies

Continuous batching (also called iteration-level scheduling) is the key throughput technique in vLLM and TGI. Unlike static batching where you wait for N requests before processing, continuous batching slots new requests into the batch as previous ones complete their generation. This keeps GPU utilization high and avoids the head-of-line blocking that kills throughput in naive request-per-request serving.

vLLM handles this automatically. What you control is --max-num-seqs (maximum concurrent sequences in a batch, default 256) and --max-num-batched-tokens (total tokens across all sequences in one scheduler step).

For a latency-sensitive application, lower --max-num-seqs to reduce batching wait time at the cost of throughput. For a background processing pipeline, increase it to maximize GPU utilization.

Async batching at the application layer is useful when you have a queue of requests that do not need immediate responses:

interface BatchRequest {
  id: string;
  prompt: string;
  resolve: (result: string) => void;
  reject: (err: Error) => void;
}

class AsyncBatcher {
  private queue: BatchRequest[] = [];
  private timer: NodeJS.Timeout | null = null;
  private readonly maxBatchSize: number;
  private readonly maxWaitMs: number;
  private readonly client: OpenAI;

  constructor(client: OpenAI, maxBatchSize = 20, maxWaitMs = 50) {
    this.client = client;
    this.maxBatchSize = maxBatchSize;
    this.maxWaitMs = maxWaitMs;
  }

  enqueue(prompt: string): Promise<string> {
    return new Promise((resolve, reject) => {
      this.queue.push({ id: crypto.randomUUID(), prompt, resolve, reject });

      if (this.queue.length >= this.maxBatchSize) {
        this.flush();
      } else if (!this.timer) {
        this.timer = setTimeout(() => this.flush(), this.maxWaitMs);
      }
    });
  }

  private async flush(): Promise<void> {
    if (this.timer) {
      clearTimeout(this.timer);
      this.timer = null;
    }

    const batch = this.queue.splice(0, this.maxBatchSize);
    if (batch.length === 0) return;

    try {
      // vLLM processes concurrent requests via its own scheduler
      const results = await Promise.all(
        batch.map(async (req) => {
          const response = await this.client.chat.completions.create({
            model: "meta-llama/Llama-3.1-8B-Instruct",
            messages: [{ role: "user", content: req.prompt }],
            max_tokens: 256,
          });
          return {
            id: req.id,
            content: response.choices[0].message.content ?? "",
            resolve: req.resolve,
          };
        })
      );

      results.forEach((r) => r.resolve(r.content));
    } catch (err) {
      batch.forEach((req) => req.reject(err as Error));
    }
  }
}

Load Balancing Inference Nodes

Multiple inference nodes behind a load balancer is the standard horizontal scaling approach. There are two things that make LLM load balancing different from normal HTTP services.

First, requests have wildly different processing times. A request that generates 2 tokens finishes in milliseconds; one that generates 2000 tokens takes seconds. Round-robin routing sends long-running requests to the same nodes that handle short requests, which creates uneven load. Least-connections routing works better in practice.

Second, KV cache prefix caching (if enabled) means that requests with identical system prompts benefit from being routed to the same node. This is a minor optimization but worth noting if you have a common system prompt across many users.

An nginx configuration for a vLLM cluster:

upstream llm_inference {
    least_conn;
    server inference-1.internal:8000;
    server inference-2.internal:8000;
    server inference-3.internal:8000;
    keepalive 32;
}

server {
    listen 443 ssl;
    server_name inference.internal;

    location /v1/ {
        proxy_pass http://llm_inference;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        proxy_read_timeout 120s;    # Long enough for slow generations
        proxy_send_timeout 30s;
        proxy_buffering off;        # Required for streaming responses
    }
}

For streaming responses, proxy_buffering off is required. Without it, nginx buffers the entire response before forwarding, which breaks the streaming UX entirely.

Monitoring Latency and Throughput

The metrics that matter for LLM inference are different from standard HTTP services.

Time to First Token (TTFT): How long from request receipt to first generated token. This is what users perceive as latency. Driven by prompt processing time, which scales with input token count.

Inter-Token Latency (ITL): Time between successive tokens. This is a function of model size, GPU speed, and current batch load. Target sub-50ms ITL for a smooth streaming UX.

Tokens per Second (TPS): Total generation throughput across all requests. This is your GPU utilization efficiency metric.

Request queue depth: How many requests are waiting to be scheduled. A persistently non-zero queue means you need more capacity.

vLLM exposes a Prometheus metrics endpoint at /metrics. The key metrics to scrape:

// Prometheus scrape config fragment
const prometheusConfig = {
  scrape_configs: [
    {
      job_name: "vllm",
      static_configs: [
        {
          targets: [
            "inference-1.internal:8000",
            "inference-2.internal:8000",
          ],
        },
      ],
      metrics_path: "/metrics",
      scrape_interval: "10s",
    },
  ],
};

// Key metric names from vLLM
const KEY_METRICS = [
  "vllm:time_to_first_token_seconds",       // TTFT histogram
  "vllm:time_per_output_token_seconds",      // ITL histogram
  "vllm:e2e_request_latency_seconds",        // End-to-end latency
  "vllm:request_success_total",              // Success counter
  "vllm:num_requests_running",               // Currently processing
  "vllm:num_requests_waiting",               // Queue depth
  "vllm:gpu_cache_usage_perc",               // KV cache utilization
  "vllm:num_preemptions_total",              // KV cache evictions (bad)
];

Set alerts on: P95 TTFT exceeding your SLA, queue depth growing beyond 10 for more than 60 seconds, and num_preemptions_total increasing (preemptions mean requests are being paused and restarted due to KV cache pressure, which tanks throughput).

Common Production Issues

OOM under burst load: The KV cache fills up. Reduce --max-model-len, lower --gpu-memory-utilization slightly to give more headroom, or add capacity. Preemptions are the early warning sign.

Slow cold start: Model loading takes 2-5 minutes. Use a startup probe, not a readiness probe, to hold traffic until the model is loaded. With Kubernetes:

readinessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 120
  periodSeconds: 10
  failureThreshold: 3
startupProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 60
  periodSeconds: 15
  failureThreshold: 20   # Allows up to 5 minutes for model load

Quantization accuracy regression: Measure on your actual task distribution before deploying. MMLU scores are a useful proxy but your production prompts may be more or less sensitive to quantization than general benchmarks suggest. Run a parallel evaluation with a sample of real requests against both the quantized and full-precision model before cutting over.

The Honest Cost Comparison

At 10M tokens/day with a 7B model on a single A10G ($1.20/hr on spot):

• GPU cost: ~$29/day
• At GPT-4o-mini pricing ($0.60/1M output tokens, $0.15/1M input tokens): roughly $4-8/day assuming a mix of input/output
• GPT-4o pricing at the same volume: $25-100/day depending on token split

At this scale, the managed API is likely cheaper when you factor in engineering time, ops overhead, and the cost of running spare capacity. The calculus changes at 100M tokens/day or when the data isolation requirement is non-negotiable.

The right mental model: self-hosting is an infrastructure bet that pays off at scale. Before you are at scale, it is a liability.

What to Ship First

Start with vLLM, an AWQ-quantized 7B or 8B model (Llama 3.1 8B or Mistral 7B v0.3 are good defaults), a single A10G node, prefix caching enabled, and the Prometheus metrics endpoint scraped into your existing monitoring stack. Get TTFT and ITL data on your actual workload. That data tells you whether the bottleneck is GPU throughput, network, or application-side overhead before you spend money scaling the wrong thing.

Model serving infrastructure is not a one-size decision. The right stack depends on your model size, latency requirements, and concurrency patterns. The frameworks are stable, the quantization options are well-understood, and the observability tooling is there. The missing piece for most teams is measurement: knowing what is actually slow before reaching for more hardware.