Kubernetes Autoscaling in Production: HPA, VPA, KEDA, and Custom Metrics for Cost-Efficient Scaling

Most Kubernetes clusters end up either over-provisioned because operators pad resource requests defensively, or under-provisioned because they over-trusted autoscaling and never validated the configuration under load. The autoscaling layer in Kubernetes is powerful but it exposes enough sharp edges that running it naively in production leads to one of two failure modes: runaway costs or cascading pod evictions during traffic spikes.

This article covers the three main autoscaling mechanisms, when each one applies, how they interact (and where they conflict), and the configuration patterns that actually work at scale. The target is a cluster running production workloads where cost and reliability are both real concerns, not a toy environment.

The Three Autoscalers and What They Actually Do

Before writing any YAML, get the model right. Kubernetes offers three distinct scaling mechanisms that operate at different levels.

Horizontal Pod Autoscaler (HPA) scales the number of pod replicas in a Deployment or StatefulSet. It polls a metrics source on a configurable interval (default: 15 seconds) and computes the desired replica count using the formula: desiredReplicas = ceil(currentReplicas * (currentMetricValue / desiredMetricValue)). It supports CPU utilization, memory utilization, and custom metrics from the Metrics API.

Vertical Pod Autoscaler (VPA) adjusts the resources.requests and resources.limits on individual pods. It operates in three modes: Off (recommendations only), Initial (applies recommendations at pod creation, never during runtime), and Auto (evicts pods and recreates them with updated requests). VPA does not change replica count.

KEDA (Kubernetes Event-Driven Autoscaling) extends HPA by adding scalers that pull metrics from external sources: SQS queue depth, Kafka consumer group lag, Prometheus queries, HTTP request rate, cron schedules, and dozens more. Under the hood, KEDA implements the Kubernetes External Metrics API and creates an HPA object that the control plane manages. KEDA can also scale to zero, which HPA alone cannot do.

The cluster autoscaler is a separate component that provisions and deprovisions nodes based on pod scheduling pressure. It is not a pod-level mechanism, but its behavior is tightly coupled to how you configure the pod-level autoscalers.

When to Use Each One

Scenario	Use
Stateless web service, CPU or latency bound	HPA on CPU utilization or custom latency metric
Consumer reading from a queue	KEDA on queue depth, scale to zero overnight
Service with highly variable request rate, unknown peak	HPA + VPA in recommendation mode to calibrate requests
Job processing batch workloads	KEDA with cron scaler or job scaler
Service where resource requests are consistently wrong	VPA in Initial mode to right-size at deploy time
Long-running, stateful workload	VPA in Auto mode is risky; use Recommendation only
Mixed CPU and memory pressure	HPA on custom metric (e.g., queue depth, active connections)

VPA and HPA should not be configured simultaneously on the same resource type. Using both CPU-based HPA and VPA Auto on the same deployment causes a fight: HPA scales out because utilization is high, VPA evicts the pods to update requests, HPA scales out again. Use VPA recommendations to calibrate your resource requests manually, then let HPA drive replica count.

Configuring HPA with Custom Prometheus Metrics

CPU utilization is a poor scaling signal for most production services. A service processing complex requests may saturate at 30% CPU while a trivially fast endpoint saturates at 90%. Scaling on a business-level metric (requests per second, queue depth, active connections) is more reliable.

This requires three components: a Prometheus instance scraping your service, the prometheus-adapter installed in the cluster, and an HPA that references the custom metric.

Configure prometheus-adapter to expose the metric:

# prometheus-adapter configmap (excerpt)
rules:
  custom:
    - seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
      resources:
        overrides:
          namespace:
            resource: namespace
          pod:
            resource: pod
      name:
        matches: "^(.*)_total$"
        as: "${1}_per_second"
      metricsQuery: 'rate(<<.Series>>{<<.LabelMatchers>>}[2m])'

HPA referencing the custom metric:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-service-hpa
  namespace: production
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-service
  minReplicas: 3
  maxReplicas: 50
  metrics:
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: 100
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Pods
          value: 5
          periodSeconds: 60

The behavior block is critical and often omitted in tutorials. Without it, scale-down uses a default stabilization window of 5 minutes but scale-up is aggressive. The configuration above allows 100% scale-up in 60 seconds (doubles replicas each minute under sustained load) and limits scale-down to 5 pods per minute with a 5-minute stabilization window. This asymmetry is intentional: fast scale-up, slow scale-down.

VPA in Recommendation Mode

Running VPA in Off mode is the safest way to get value from it without risk. It observes your workload for 8 days (default) and builds a recommendation based on observed CPU and memory usage at various percentiles.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-service-vpa
  namespace: production
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-service
  updatePolicy:
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
      - containerName: api-service
        minAllowed:
          cpu: 100m
          memory: 128Mi
        maxAllowed:
          cpu: 4
          memory: 4Gi
        controlledResources: ["cpu", "memory"]

After a week of normal traffic, check the recommendations:

kubectl describe vpa api-service-vpa -n production

The output includes lowerBound, target, and upperBound for both CPU and memory. The target is the recommended request; the upperBound is what VPA would set as the limit. If your current requests are 2x or more than the recommended target, you are paying for headroom you do not need.

The most common finding is that teams set CPU requests at 500m because it “feels safe” and VPA recommends 80m because that workload is actually I/O bound, not compute bound. Over 50 pods, that is a meaningful cluster cost difference.

KEDA Scalers for Real Workloads

SQS Queue Consumer

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: sqs-consumer
  namespace: production
spec:
  scaleTargetRef:
    name: sqs-worker-deployment
  pollingInterval: 15
  cooldownPeriod: 30
  minReplicaCount: 0
  maxReplicaCount: 20
  triggers:
    - type: aws-sqs-queue
      authenticationRef:
        name: keda-aws-credentials
      metadata:
        queueURL: https://sqs.us-east-1.amazonaws.com/123456789/worker-queue
        queueLength: "5"
        awsRegion: us-east-1
        scaleOnInFlight: "true"

queueLength: "5" means KEDA will target one replica per 5 messages in the queue. With 100 messages, it schedules 20 replicas. With 0 messages after the cooldown period, it scales to zero. Setting scaleOnInFlight: "true" includes messages currently being processed in the depth calculation, which prevents premature scale-down while work is still in flight.

Kafka Consumer Group Lag

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: kafka-consumer
  namespace: production
spec:
  scaleTargetRef:
    name: kafka-processor-deployment
  pollingInterval: 10
  minReplicaCount: 1
  maxReplicaCount: 30
  triggers:
    - type: kafka
      metadata:
        bootstrapServers: kafka-broker:9092
        consumerGroup: event-processor-group
        topic: domain-events
        lagThreshold: "50"
        offsetResetPolicy: latest

One replica per 50 messages of consumer group lag. minReplicaCount: 1 keeps the consumer group offset active even during quiet periods. Setting this to 0 means Kafka assumes the consumer is gone and may rebalance aggressively when scale-up happens.

HTTP Request Rate Scaler

KEDA’s HTTP add-on enables scaling to zero for HTTP services. It requires an interceptor proxy in front of your service. For services that receive no traffic overnight (internal tools, cron-triggered APIs), this reduces idle replica costs significantly.

apiVersion: http.keda.sh/v1alpha1
kind: HTTPScaledObject
metadata:
  name: internal-api
  namespace: production
spec:
  hosts:
    - internal-api.company.internal
  pathPrefixes:
    - /
  scaleTargetRef:
    name: internal-api-deployment
    port: 8080
  replicas:
    min: 0
    max: 10
  scalingMetric:
    requestRate:
      granularity: 1s
      targetValue: 20
      window: 1m

Cold start latency is the tradeoff here. When scaled to zero, the first request will wait for pod startup. If startup time is 5+ seconds, internal tools are a much better fit for this pattern than user-facing APIs.

HPA vs VPA vs KEDA: Tradeoffs

Dimension	HPA	VPA	KEDA
What it scales	Replica count	Pod resource requests	Replica count (via HPA)
Primary input	CPU, memory, custom metrics	Observed resource usage over time	External event sources
Scale to zero	No	No	Yes
Pod disruption	None	Yes (in Auto mode, evicts pods)	None
Latency of response	15-30 seconds	Minutes to hours	Configurable (10-30s typical)
Setup complexity	Low	Medium	Medium-High
VPA conflict risk	Yes, if both use CPU	N/A	Low (KEDA owns the HPA object)
Best for	Stateless HTTP services	Right-sizing resource requests	Queue consumers, event-driven workers
Main risk	Thrashing on noisy metrics	Evictions during traffic spikes	Cold start latency when scaling from zero

Cluster Autoscaler Integration

The cluster autoscaler reacts to pods in Pending state caused by insufficient node capacity. It provisions a new node and the scheduler places the pending pod. This loop works reliably when your pod-level autoscalers are configured correctly, but breaks down in two common ways.

Problem 1: Scale-up is slow because cluster autoscaler lags behind HPA. HPA schedules new pods in 30 seconds. If no node has capacity, those pods sit Pending until the cluster autoscaler provisions a new node (1-3 minutes on most cloud providers). During that window, your service is under-scaled. The mitigation is to maintain a small amount of excess node capacity using pod priority or overprovisioning pods.

An overprovision deployment schedules low-priority placeholder pods that occupy node capacity:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: cluster-overprovisioner
spec:
  replicas: 2
  template:
    spec:
      priorityClassName: overprovisioning
      containers:
        - name: pause
          image: registry.k8s.io/pause:3.9
          resources:
            requests:
              cpu: 500m
              memory: 500Mi

When a real pod needs capacity, it preempts the pause pod. The pause pod becomes Pending, which triggers the cluster autoscaler to provision a new node.

Problem 2: Nodes do not scale down because pods have no PodDisruptionBudget. The cluster autoscaler drains nodes before terminating them. If your pods block eviction (no PDB, or PDB that prevents any eviction), the cluster autoscaler cannot remove underutilized nodes. Define a PDB for every production deployment:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-service-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: api-service

This allows eviction while guaranteeing at least 2 replicas stay available during drains.

Common Production Pitfalls

Metric thrashing on HPA. A scaling metric that oscillates (request rate that spikes every few seconds) causes HPA to scale up and down repeatedly. The stabilization window on scale-down helps, but the root cause is often a metric with too short a window. Use a 2-5 minute rate window in Prometheus rather than a 30-second window. The signal should describe sustained load, not momentary spikes.

VPA Auto mode on stateful or latency-sensitive services. VPA in Auto mode evicts pods to apply new resource recommendations. For a database sidecar, a cache warmer, or any service with significant startup time, this causes unnecessary disruption. Use Initial or Off.

Resource requests set without profiling. The single most common cost inefficiency in Kubernetes is over-padded resource requests. Teams set cpu: 500m, memory: 512Mi as a default across all services regardless of actual usage. Run VPA in Off mode for two weeks and you will almost always find services where actual p95 CPU usage is 5-10x lower than the request. The cluster autoscaler decisions, pod scheduling density, and node costs all flow from resource requests, so misconfiguration here has compounding effects.

HPA minReplicas: 1 for services that need to tolerate a node failure. A single replica provides no availability guarantee. With one pod and a node failure, your service is down until the scheduler places a new pod, which can take 30-60 seconds. Set minReplicas to at least 2 for any service where a 30-second outage is unacceptable, and spread them across availability zones using pod topology spread constraints:

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: topology.kubernetes.io/zone
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: api-service

Scale-down delays masking resource waste. The default HPA scale-down stabilization window is 5 minutes. KEDA’s cooldown period defaults to 300 seconds as well. This is correct for production services, but for batch workers that run jobs overnight and are idle during the day, these defaults mean the cluster holds onto excess replicas for minutes after the queue drains. For KEDA queue consumers, lower the cooldownPeriod to 60 seconds and monitor for oscillation before settling on a value.

Cost Implications

The cost case for autoscaling is straightforward: you pay for provisioned capacity, not for requests handled. A service that scales from 3 to 30 replicas during peak hours and back to 3 overnight costs roughly 10x less on replica compute than a static 30-replica deployment.

The less obvious cost consideration is the cluster autoscaler’s interaction with node pool configuration. Spot or preemptible nodes cost 70-90% less than on-demand. HPA and KEDA work equally well on spot nodes, but pods must handle termination signals correctly and start up quickly enough that the cluster autoscaler can replace spot capacity before the service degrades.

Right-sizing resource requests through VPA recommendations often reduces the number of nodes needed at any given replica count. If VPA reveals that your pods need 200m CPU instead of 1000m, you can pack 5x more pods onto the same node, reducing the baseline cluster cost significantly before autoscaling even runs.

A Note on Observability

Autoscaling decisions are only debuggable if you instrument them. Track these metrics in whatever monitoring system you use:

• HPA current replicas vs desired replicas over time
• HPA scaling events (both up and down) with the trigger metric value at the time
• VPA recommendations vs current requests per deployment
• KEDA scaler metric values (queue depth, lag) alongside replica count
• Cluster autoscaler scale-up latency (time from Pending pod to pod Running)

Without these metrics, you will not know whether your autoscaling configuration is working correctly until it fails under a load pattern you have not seen before.

Autoscaling in Kubernetes is not a set-and-forget configuration. It is an ongoing calibration of scaling signals, thresholds, and policies against observed workload behavior. Start with VPA in recommendation mode to fix resource requests, add HPA with a sensible custom metric, deploy KEDA for any queue-driven consumers, and revisit the configuration after every significant traffic event. The configuration that was correct at 100 RPS needs review at 10,000 RPS.