GPU Infrastructure for AI Workloads: Provisioning, Cost Optimization, and Autoscaling Inference Servers

Running an LLM in production is not the same problem as running one in a notebook. The notebook tolerates 30-second first-token latency. Your users do not. The notebook runs on a single GPU you rented for an afternoon. Your inference server needs to handle bursty traffic, survive instance preemptions, and not invoice you into insolvency.

GPU compute is genuinely expensive. A single NVIDIA A100 80GB instance on AWS (p4d.24xlarge) costs around $32/hour on-demand. A 7B parameter model loaded in fp16 consumes roughly 14GB of VRAM. A 70B model needs four A100s just to fit in memory. If your inference server idles at 5% utilization because you over-provisioned to handle peak traffic, you are burning money at a rate that eliminates most startup margins. The gap between “working” and “economically viable” GPU infrastructure is where most teams get stuck.

This article covers the decisions that actually matter: which GPU for which workload, where to provision it, how to reduce cost without degrading quality, how to autoscale inference correctly, and what to monitor so you know what’s breaking before your users do.

GPU Selection: Match the Workload to the Hardware

NVIDIA’s data center GPU lineup is not interchangeable. Each chip has a different balance of memory bandwidth, VRAM capacity, Tensor Core generation, and per-hour cost. Picking the wrong one is expensive in both directions.

NVIDIA T4 (16GB VRAM) is the floor for production inference on smaller models. It runs on AWS g4dn instances and GCP n1 + T4 instances at roughly $0.50-0.75/hour. The T4 has Turing-era Tensor Cores and 320 GB/s memory bandwidth. It handles 7B models comfortably in int8, 13B models are a stretch. Use T4s for batch-tolerant inference, embedding generation, and classifier serving where latency is not a first-order constraint. Never use T4s for fine-tuning anything serious.

NVIDIA A100 40GB/80GB is the workhorse for serious training and high-throughput inference. The 80GB variant can hold a 70B parameter model in fp16, or a 34B model with comfortable headroom for KV cache growth. NVLink connectivity on multi-GPU setups gives near-linear scaling for training. AWS p4d.24xlarge gives you 8x A100 80GB for $32/hour, GCP a2-highgpu-8g gives 8x A100 40GB. Use A100s for fine-tuning, training runs under 100B parameters, and latency-sensitive inference on large models.

NVIDIA H100 80GB (SXM5/PCIe) is the current performance apex for training. It has 3.35 TB/s HBM3 memory bandwidth (vs 2 TB/s on A100) and fourth-generation Tensor Cores with FP8 support. The step-up matters most for large training runs and for models where KV cache pressure is severe. AWS p5.48xlarge gives 8x H100 SXM5 for $98/hour. That price makes H100s appropriate for training, not for inference serving unless you need to saturate throughput on a model that is already consuming full A100 VRAM. Inference ROI on H100 vs A100 rarely justifies the 3x cost difference unless you are running at very high QPS.

NVIDIA L4 (24GB VRAM) is the inference-optimized middle tier. It uses Ada Lovelace architecture with dedicated optical flow and encoder hardware, but the relevant feature is its 72W TDP vs 300W for A100. That lower power draw makes L4 much cheaper to run per-hour, and GCP’s g2-standard instances make L4 competitive at scale. L4 is purpose-built for serving: it handles 7B and 13B models in fp16, 7B in fp32 if you want, and larger models in quantized form. Use L4 for production inference where you’re cost-conscious and models fit within 24GB.

The short version:

GPU	VRAM	Best for	Avoid for	Approx on-demand/hr
T4	16GB	Embedding, small model inference	Fine-tuning, large models	$0.50-0.75
L4	24GB	Production inference, 7B-13B	Training, 70B+ models	$0.70-1.20
A100	80GB	Training, large model inference	Cost-sensitive small models	$3.00-4.00
H100	80GB	Large-scale training, max QPS	Inference ROI unless very high load	$10-15

Fine-tuning sits between training and inference in terms of requirements. LoRA and QLoRA have reduced the VRAM requirements dramatically. A 7B model fine-tuned with QLoRA fits on a single A100 40GB or two L4s. Full fine-tuning a 70B model still requires 4-8 A100 80GB with gradient checkpointing.

Provisioning Strategy: Cloud vs Specialized Providers

AWS, GCP, and Azure all offer GPU compute, but GPU availability is a real constraint. A100 and H100 capacity is frequently exhausted in major regions. The major clouds charge a premium for that availability guarantee.

AWS is the safe choice for organizations already embedded in the AWS ecosystem. The p4d (A100), p5 (H100), and g4dn (T4) families integrate with existing IAM, VPC, and EKS workflows. The operational overhead of using a different provider often exceeds the cost savings. Spot interruption rates for GPU instances are higher than for CPU, typically 15-40% in popular regions, which shapes your resilience architecture.

GCP competes seriously on L4 availability and price. The A3 series (H100) and A2 (A100) are comparable to AWS. GCP’s TPU v5e is a legitimate alternative for transformer inference at scale if you’re willing to invest in JAX or Pax compatibility, but most teams aren’t.

Lambda Labs runs at 60-70% of AWS on-demand pricing for A100 and H100 instances. The tradeoff is a thinner operational layer. You get a VM, not a managed Kubernetes control plane with native autoscaling hooks. Lambda is excellent for batch training jobs and experimentation where you want cheap raw GPU hours and can tolerate the operational simplicity constraints.

CoreWeave is the option that comes up most for serious ML infrastructure teams. Their A100 and H100 availability is better than major clouds in practice, and they run Kubernetes natively. If you need a cluster of 8+ A100s for training and can’t get AWS capacity, CoreWeave is worth evaluating seriously.

RunPod occupies the bottom of the price range: community cloud instances where you rent unused consumer GPUs (RTX 4090s, RTX 3090s). Fine for experimentation and low-stakes fine-tuning. Not appropriate for production inference where SLA matters.

For production inference specifically, the major clouds win on reliability and integration. For batch training jobs, Lambda Labs or CoreWeave on reserved instances often beats AWS on-demand by 40-60%.

Cost Optimization: Four Levers That Actually Move the Number

Spot and Preemptible Instances

GPU spot instances are the single highest-leverage cost reduction available. AWS spot pricing for p3.2xlarge (V100) runs 60-70% below on-demand. The problem is interruption handling. Training jobs need checkpoint-and-resume logic. Inference servers need graceful drain so in-flight requests complete before the instance terminates.

A minimal Kubernetes deployment that handles spot interruption for an inference workload:

// spot-inference-deployment.ts
import { KubernetesManifest } from 'aws-cdk-lib/aws-eks';
import { Stack } from 'aws-cdk-lib';

export function createSpotInferenceDeployment(stack: Stack, cluster: any) {
  return new KubernetesManifest(stack, 'SpotInferenceDeployment', {
    cluster,
    manifest: [{
      apiVersion: 'apps/v1',
      kind: 'Deployment',
      metadata: { name: 'llm-inference', namespace: 'ml' },
      spec: {
        replicas: 3,
        selector: { matchLabels: { app: 'llm-inference' } },
        template: {
          metadata: { labels: { app: 'llm-inference' } },
          spec: {
            // Prefer spot, fall back to on-demand
            nodeSelector: { 'karpenter.sh/capacity-type': 'spot' },
            tolerations: [{
              key: 'nvidia.com/gpu',
              operator: 'Exists',
              effect: 'NoSchedule',
            }],
            terminationGracePeriodSeconds: 60, // complete in-flight requests
            containers: [{
              name: 'vllm',
              image: 'vllm/vllm-openai:latest',
              resources: {
                limits: { 'nvidia.com/gpu': '1' },
                requests: { 'nvidia.com/gpu': '1' },
              },
              lifecycle: {
                preStop: {
                  // drain before termination signal reaches the process
                  exec: { command: ['/bin/sh', '-c', 'sleep 15'] },
                },
              },
            }],
          },
        },
      },
    }],
  });
}

The terminationGracePeriodSeconds and preStop hook together give the inference server time to drain. Without these, spot terminations silently drop requests.

Model Quantization

Quantization reduces VRAM requirements, which means you can serve on smaller (cheaper) GPUs or serve more concurrent requests on the same hardware. The practical tradeoffs:

Precision	Size vs fp16	Quality loss	Use case
fp16	1x baseline	None (reference)	Training, quality-critical inference
int8	0.5x	Minimal (0.1-0.5% on benchmarks)	Production inference default
int4 (AWQ/GPTQ)	0.25x	Small (1-3% on benchmarks)	Memory-constrained inference
GGUF Q4	0.25x	Moderate	Edge/CPU fallback, dev environments

A 70B model in fp16 needs 140GB of VRAM (4x A100 80GB). The same model in int4 fits on a single A100 80GB with room for KV cache. That’s a 4x cost reduction on hardware for a 1-3% quality hit on most tasks. For most production use cases, int8 is the right default: near-fp16 quality, half the VRAM requirement.

vLLM handles quantization at load time:

// model-config.ts
export interface ModelConfig {
  modelId: string;
  quantization: 'none' | 'awq' | 'gptq' | 'squeezellm';
  dtype: 'float16' | 'bfloat16' | 'float32';
  maxModelLen: number;
  gpuMemoryUtilization: number; // 0.0-1.0, leave headroom for KV cache
}

export const INFERENCE_CONFIGS: Record<string, ModelConfig> = {
  'llama3-70b-production': {
    modelId: 'meta-llama/Meta-Llama-3-70B-Instruct',
    quantization: 'awq',
    dtype: 'float16',
    maxModelLen: 8192,
    gpuMemoryUtilization: 0.88, // 12% headroom for KV cache growth
  },
  'llama3-8b-high-throughput': {
    modelId: 'meta-llama/Meta-Llama-3-8B-Instruct',
    quantization: 'none',
    dtype: 'float16',
    maxModelLen: 16384,
    gpuMemoryUtilization: 0.90,
  },
};

Setting gpuMemoryUtilization too high causes OOM on long context requests. 0.85-0.90 is the production range; go higher only after profiling under realistic traffic.

Request Batching

The difference between a GPU at 5% utilization and 80% utilization is almost entirely batching. GPUs are parallel compute engines. A single inference request uses a small fraction of available compute. Continuous batching (as implemented in vLLM and TensorRT-LLM) packs multiple requests into each forward pass, amortizing the per-token cost across concurrent users.

The lever you control is maximum batch size and batch wait timeout. A tight timeout (5ms) optimizes for latency. A looser timeout (50ms) optimizes for throughput. For user-facing chat interfaces, tune for latency. For batch document processing, tune for throughput.

// inference-client.ts
export class InferenceClient {
  private readonly baseUrl: string;
  private readonly timeoutMs: number;

  constructor(baseUrl: string, timeoutMs = 30_000) {
    this.baseUrl = baseUrl;
    this.timeoutMs = timeoutMs;
  }

  async complete(
    prompt: string,
    options: { maxTokens?: number; temperature?: number } = {}
  ): Promise<string> {
    const controller = new AbortController();
    const timer = setTimeout(() => controller.abort(), this.timeoutMs);

    try {
      const response = await fetch(`${this.baseUrl}/v1/completions`, {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({
          model: 'default',
          prompt,
          max_tokens: options.maxTokens ?? 512,
          temperature: options.temperature ?? 0.0,
          stream: false,
        }),
        signal: controller.signal,
      });

      if (!response.ok) {
        throw new Error(`Inference failed: ${response.status} ${response.statusText}`);
      }

      const data = await response.json() as { choices: Array<{ text: string }> };
      return data.choices[0].text;
    } finally {
      clearTimeout(timer);
    }
  }
}

GPU Time-Sharing

NVIDIA’s Multi-Instance GPU (MIG) feature on A100 and H100 allows physical partitioning of a single GPU into up to 7 isolated slices, each with their own VRAM and compute allocation. For serving multiple smaller models or isolated inference workloads on the same hardware, MIG gives you real isolation without the overhead of separate VMs.

NVIDIA’s Multi-Process Service (MPS) is a softer form of sharing without hardware isolation. It’s appropriate when you have multiple processes that don’t individually saturate the GPU but don’t require strict VRAM isolation.

Autoscaling Inference Servers

The naive approach is to scale on CPU utilization, the default for most Kubernetes HPA configurations. CPU utilization on a GPU inference server is almost meaningless. The GPU is doing the work. You need to scale on signals that reflect actual GPU pressure.

Scale on request queue depth for batch workloads. If your inference endpoint sits behind an SQS queue or Kafka topic, scale on message backlog depth. This is the clearest signal: the queue is growing faster than you’re draining it, so add replicas.

Scale on p95 latency for interactive workloads. User-facing applications care about latency, not throughput. If p95 time-to-first-token crosses your SLO threshold (typically 500ms-2s for chat), scale out. KEDA (Kubernetes Event Driven Autoscaler) supports custom metrics including Prometheus, making this straightforward to wire up.

Scale on GPU utilization only as a secondary signal. GPU utilization from nvidia-smi tells you when you’re saturated, but it’s a lagging indicator for interactive workloads. By the time GPU utilization is at 90%, latency is already degraded.

// keda-scaledobject.ts
export function createInferenceScaler(namespace: string) {
  return {
    apiVersion: 'keda.sh/v1alpha1',
    kind: 'ScaledObject',
    metadata: { name: 'llm-inference-scaler', namespace },
    spec: {
      scaleTargetRef: { name: 'llm-inference' },
      minReplicaCount: 1,
      maxReplicaCount: 8,
      cooldownPeriod: 120, // seconds before scale-down after traffic drops
      pollingInterval: 15,
      triggers: [
        {
          type: 'prometheus',
          metadata: {
            serverAddress: 'http://prometheus.monitoring.svc:9090',
            // vLLM exposes this metric natively
            query: 'vllm:num_requests_waiting{namespace="ml"}',
            threshold: '5', // scale up if >5 requests waiting
          },
        },
        {
          type: 'prometheus',
          metadata: {
            serverAddress: 'http://prometheus.monitoring.svc:9090',
            query: 'histogram_quantile(0.95, rate(vllm:e2e_request_latency_seconds_bucket[2m]))',
            threshold: '2.0', // scale up if p95 latency > 2s
          },
        },
      ],
    },
  };
}

Cold Start: The Real Problem with Large Models

A 70B quantized model is roughly 35GB of weights. Loading that from S3 into VRAM on a fresh instance takes 3-8 minutes depending on network bandwidth. A Kubernetes pod that takes 5 minutes to become ready is not useful for handling a traffic spike.

The production mitigations:

1. Keep a warm minimum replica running at all times. Scale to zero sounds appealing on paper. For models over 7B, it’s not viable in practice unless your traffic patterns allow 5+ minute cold starts.

2. Use instance store or node-local NVMe for model caching. Pulling weights from S3 every cold start is slow. Mount model weights onto node-local storage using a DaemonSet that pre-warms the cache when a node joins the cluster.

3. Separate the model loading path from the readiness probe. A pod that is “Running” but still loading weights should not receive traffic. Your readiness probe must check the actual inference endpoint, not just the process health.

// readiness-probe.ts — minimal health check for vLLM
export async function checkInferenceReady(endpoint: string): Promise<boolean> {
  try {
    const response = await fetch(`${endpoint}/health`, {
      signal: AbortSignal.timeout(2000),
    });
    return response.ok;
  } catch {
    return false;
  }
}

# readinessProbe in pod spec
readinessProbe:
  httpGet:
    path: /health
    port: 8000
  initialDelaySeconds: 120  # give the model time to load
  periodSeconds: 10
  failureThreshold: 30      # 5 minutes total wait before giving up

The Monitoring Layer

Kubernetes default metrics will not tell you what’s wrong with a GPU inference server. You need GPU-specific observability.

NVIDIA DCGM Exporter runs as a DaemonSet and exposes Prometheus metrics for GPU utilization, memory used/free, memory bandwidth, power draw, temperature, and SM clock speed. Install it first; everything else builds on it.

The metrics that matter in production:

Metric	Alert threshold	What it means
DCGM_FI_DEV_GPU_UTIL	>90% sustained 5m	Compute saturated, add replicas
DCGM_FI_DEV_MEM_COPY_UTIL	>80% sustained	Memory bandwidth bottleneck
DCGM_FI_DEV_FB_USED	>90% of VRAM	OOM risk, reduce batch size or add instances
DCGM_FI_DEV_GPU_TEMP	>83°C	Thermal throttling, check cooling/instance type
DCGM_FI_DEV_POWER_USAGE	>TDP spec	Power capping may be throttling performance
vllm:num_requests_waiting	>10	Queue building, autoscaler should have triggered

Thermal throttling is a real production issue in some cloud environments, particularly on older hardware. When GPU temperature exceeds the throttle threshold (typically 83-87°C for data center GPUs), the SM clock speed drops and inference throughput degrades silently. You see this as latency degradation that doesn’t correlate with GPU utilization or memory pressure. Always alert on temperature.

Memory pressure kills inference servers in a different way. If a long-context request grows the KV cache past available VRAM, vLLM will OOM-kill the request and potentially destabilize the server. Set gpu_memory_utilization conservatively, monitor DCGM_FI_DEV_FB_USED, and alert before you hit 90%.

// gpu-metrics-alert-rules.ts
export const GPU_ALERT_RULES = [
  {
    alert: 'GPUMemoryPressure',
    expr: 'DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_FREE > 0.90',
    for: '2m',
    labels: { severity: 'warning' },
    annotations: {
      summary: 'GPU memory usage above 90% on {{ $labels.instance }}',
      description: 'VRAM utilization: {{ $value | humanizePercentage }}. Risk of OOM on large requests.',
    },
  },
  {
    alert: 'GPUThermalThrottling',
    expr: 'DCGM_FI_DEV_GPU_TEMP > 83',
    for: '5m',
    labels: { severity: 'warning' },
    annotations: {
      summary: 'GPU temperature above throttle threshold on {{ $labels.instance }}',
      description: 'Temperature: {{ $value }}°C. Clock speeds may be reduced.',
    },
  },
  {
    alert: 'InferenceQueueBacklog',
    expr: 'vllm:num_requests_waiting > 20',
    for: '1m',
    labels: { severity: 'critical' },
    annotations: {
      summary: 'Inference queue backlog exceeding 20 requests',
      description: 'Autoscaler may not be scaling fast enough. Check cold start times.',
    },
  },
];

Production Decisions That Bite Later

Oversizing the instance for safety. Running a 7B model on an A100 because “we might need it later” costs $3/hour vs $0.70/hour for an L4. Profile actual VRAM usage under peak load with realistic context lengths before committing to instance size.

Not accounting for KV cache in VRAM estimates. Model weights are static. KV cache grows with context length and concurrent requests. A 13B model in fp16 uses 26GB for weights. Under 100 concurrent requests at 4K context length each, KV cache adds another 20-30GB. Budget accordingly.

Using on-demand pricing for batch training. There is almost never a reason to run a training job on on-demand GPU instances. Spot interruptions are a real cost, but checkpoint-and-resume adds one day of engineering once, and the 60-70% cost reduction is permanent.

Single-replica inference in production. Even if your traffic doesn’t justify two replicas for throughput, run two for availability. Single-GPU inference pods have no fault tolerance for instance failures or spot preemptions.

The cost reality of GPU infrastructure forces every trade-off to be explicit. You cannot afford to provision defensively the way you might with CPU workloads. The teams that run GPU infrastructure efficiently are the ones who measure VRAM utilization per model, set autoscaling on the right signals, and treat spot interruptions as a design constraint rather than an edge case. Everything else follows from taking the cost seriously.