Designing an Autoscaling System: Metrics-Driven Scaling, Cooldown Policies, and Predictive Capacity Planning

Most autoscaling failures are not scaling failures. They are metric selection failures, policy configuration failures, or cooldown misconfiguration failures. The autoscaling machinery works fine. The inputs to that machinery are wrong.

This article covers how autoscaling systems actually work at the infrastructure level, what the real tradeoffs are between each approach, and where the configuration decisions that matter most tend to get underspecified.

The Scaling Control Loop

Every autoscaling system is a control loop. At some interval, it reads a metric, compares it to a target, computes a desired replica count or capacity level, and applies a change. The details vary across clouds and orchestrators, but the loop is always the same:

observe metric -> compare to target -> compute desired state -> apply change -> wait

The wait is the cooldown. Almost everyone gets it wrong.

The core challenge is that infrastructure changes are not instantaneous. A new instance takes 60 to 120 seconds to boot, pass health checks, and receive traffic. If the control loop fires again before the first change has stabilized the metric, you get oscillation: scale up, scale up again, metric drops, scale down, scale down again. This is flapping, and it is expensive and destabilizing.

Reactive Scaling: Metrics That Actually Work

CPU and Memory as Scaling Signals

CPU utilization is the most common scaling signal and also the most commonly misused. The specific failure is using average CPU across instances rather than the CPU of the busiest instance.

If you have 10 instances and one is at 90% CPU while the others are at 10%, your average is 18%. A threshold-based scaler sees no problem. Your users hitting the overloaded instance see a problem.

p95 CPU across instances is a better signal for most workloads. For latency-sensitive services, use request latency as a scaling metric rather than CPU at all. CPU is a proxy. Latency is what you actually care about.

Memory is a worse scaling signal than CPU for most web services. Memory usage is often sticky: it rises as services warm up their caches and does not fall proportionally when load drops. Autoscaling on memory tends to produce systems that scale up and never scale back down. Use memory as a floor (scale up if memory pressure is high) but not as a primary driver.

Custom Metrics

Custom metrics unlock the right scaling signal for queue-based and async systems. If your service consumes from a queue, the correct scaling signal is queue depth (or queue depth divided by processing rate). CPU on the consumer is a trailing indicator. By the time CPU is high, the queue has already grown to a problematic size.

Here is a TypeScript example of computing a target replica count from a queue metric:

interface ScalingContext {
  currentReplicas: number;
  queueDepth: number;
  targetQueueDepthPerReplica: number;
  minReplicas: number;
  maxReplicas: number;
}

function computeDesiredReplicas(ctx: ScalingContext): number {
  const { currentReplicas, queueDepth, targetQueueDepthPerReplica, minReplicas, maxReplicas } = ctx;

  if (queueDepth === 0) {
    return minReplicas;
  }

  const desired = Math.ceil(queueDepth / targetQueueDepthPerReplica);
  return Math.min(Math.max(desired, minReplicas), maxReplicas);
}

// Example: 500 messages in queue, target 50 messages per replica
const result = computeDesiredReplicas({
  currentReplicas: 4,
  queueDepth: 500,
  targetQueueDepthPerReplica: 50,
  minReplicas: 2,
  maxReplicas: 20,
});
// result: 10

This is the same logic that Kubernetes KEDA uses internally. The metric source is pluggable (SQS, Kafka, Redis, Prometheus) but the replica computation is always ceil(metricValue / targetMetricValuePerPod).

Scaling Policies: Step vs Target Tracking

Step Scaling

Step scaling fires when a metric crosses a threshold and applies a fixed change: add 2 instances, or add 20% capacity. Multiple steps can be configured to react differently to different severity levels: add 1 instance when CPU is between 70 and 80%, add 3 instances when CPU exceeds 80%.

The problem with step scaling is that the right step size depends on how many instances you currently have. Adding 2 instances when you have 4 is a 50% increase. Adding 2 instances when you have 40 is a 5% increase. The step is not proportional to the problem.

Step scaling is useful when your workload has known discrete states (batch job starts, end-of-month processing) and you want explicit control over how capacity changes. For most stateless web services it is the wrong tool.

Target Tracking

Target tracking scaling tells the autoscaler what metric value you want to maintain, and lets it compute the change size. You set a target CPU utilization of 60%, and the scaler continuously adjusts capacity to hold that target. The math is straightforward:

desiredReplicas = currentReplicas * (currentMetricValue / targetMetricValue)

This is proportional. If CPU is at 90% and the target is 60%, the desired count is current * (90 / 60) = current * 1.5. If you have 4 replicas, you get 6. If you have 20, you get 30. The response scales with the problem.

Target tracking also handles scale-down correctly. When load drops and CPU falls below target, the formula produces a smaller replica count. The scaler applies that change after the scale-down cooldown expires.

The critical configuration decision in target tracking is the target value itself. Setting target CPU at 80% leaves almost no headroom. A sudden traffic spike that takes CPU from 78% to 95% while new instances are booting creates a gap where the service is degraded. Set your target 20 to 30 percentage points below the point at which the service starts to struggle. For a service that degrades at 85% CPU, a 60% target gives roughly 30 to 90 seconds of headroom while instances spin up.

Cooldown Periods and Flapping Prevention

Cooldown is the minimum time between scaling actions. It exists because the metric does not immediately respond to a capacity change. If you add instances and the metric has not recovered yet, you should wait before adding more. If you remove instances and the metric has not risen yet, you should wait before removing more.

Most platforms give you separate scale-up and scale-down cooldowns. This is correct: they serve different purposes and need different values.

Scale-up cooldown should be at least as long as it takes a new instance to be healthy and serving traffic. For most cloud VMs this is 90 to 180 seconds. For containers in Kubernetes, it depends on your readiness probe configuration but 60 to 90 seconds is typical.

Scale-down cooldown should be longer. Scale-down carries risk (you might need to scale back up quickly) and the cost savings from slightly delayed scale-down are small. Set scale-down cooldown to 5 to 15 minutes. This is not over-cautious; it is recognizing that brief traffic dips should not trigger a scale-down that you will immediately regret.

Here is a simple cooldown guard in TypeScript that you might use in a custom autoscaling controller:

interface ScalingState {
  lastScaleUpAt: number | null;
  lastScaleDownAt: number | null;
  scaleUpCooldownMs: number;
  scaleDownCooldownMs: number;
}

type ScalingDirection = "up" | "down" | "none";

function isWithinCooldown(state: ScalingState, direction: ScalingDirection, now: number): boolean {
  if (direction === "up" && state.lastScaleUpAt !== null) {
    return now - state.lastScaleUpAt < state.scaleUpCooldownMs;
  }
  if (direction === "down" && state.lastScaleDownAt !== null) {
    return now - state.lastScaleDownAt < state.scaleDownCooldownMs;
  }
  return false;
}

function shouldScale(
  desired: number,
  current: number,
  state: ScalingState,
  now: number
): ScalingDirection {
  if (desired > current && !isWithinCooldown(state, "up", now)) {
    return "up";
  }
  if (desired < current && !isWithinCooldown(state, "down", now)) {
    return "down";
  }
  return "none";
}

Flapping prevention goes beyond cooldowns. Some platforms allow you to define a stabilization window: the scaler only acts on the maximum desired count observed over the last N seconds (for scale-up) or the minimum (for scale-down). This prevents a momentary spike from triggering scale-up, and a momentary dip from triggering premature scale-down. Kubernetes HPA has this built in as behavior.scaleUp.stabilizationWindowSeconds and behavior.scaleDown.stabilizationWindowSeconds.

Kubernetes HPA and VPA

Horizontal Pod Autoscaler

HPA scales the number of pod replicas based on observed metrics. The default metric is CPU utilization measured against the pod’s CPU request. The formula is exactly the target tracking formula above.

The most important thing to get right with HPA is the CPU request. HPA computes utilization as actual CPU usage / CPU request. If you set a CPU request that is too low, HPA thinks pods are always busy and keeps adding replicas. If the request is too high, HPA thinks pods are idle and will scale down too aggressively.

Set CPU requests to roughly the p50 CPU usage of the pod under normal load, and set CPU limits to the p99. Profile under load before setting these values.

Custom metrics with HPA require a metrics adapter (Prometheus Adapter, KEDA, or cloud-native equivalents). KEDA is the most practical choice because it ships with adapters for the most common external metric sources and handles the ScaledObject lifecycle cleanly.

A ScaledObject for an SQS queue worker looks like this (expressed as the TypeScript equivalent of what you would configure):

interface KedaScaledObject {
  name: string;
  deploymentName: string;
  minReplicaCount: number;
  maxReplicaCount: number;
  triggers: KedaTrigger[];
}

interface KedaTrigger {
  type: "aws-sqs-queue";
  queueURL: string;
  queueLength: string; // target messages per replica
  awsRegion: string;
}

const workerScaling: KedaScaledObject = {
  name: "order-processor-scaler",
  deploymentName: "order-processor",
  minReplicaCount: 0, // scale to zero when queue is empty
  maxReplicaCount: 50,
  triggers: [
    {
      type: "aws-sqs-queue",
      queueURL: "https://sqs.us-east-1.amazonaws.com/123456789/order-queue",
      queueLength: "10",
      awsRegion: "us-east-1",
    },
  ],
};

Note minReplicaCount: 0. KEDA supports scale-to-zero for workloads that have periods of no activity. HPA cannot scale below 1.

Vertical Pod Autoscaler

VPA adjusts the CPU and memory requests of individual pods rather than the count. It watches actual resource usage over time and recommends or automatically applies updated request values.

VPA operates in three modes: Off (recommendations only), Initial (applies recommendations only when pods are created), and Auto (restarts pods to apply updated requests).

Auto mode is disruptive. When VPA updates requests, pods are evicted and rescheduled. For stateless services this is acceptable if your PodDisruptionBudget is configured to maintain minimum availability. For stateful or latency-sensitive services, Initial mode is safer: resources are set correctly at pod creation, and you update requests during planned deployments rather than at VPA’s discretion.

HPA and VPA cannot both operate on CPU for the same deployment. VPA changing CPU requests changes the denominator HPA uses to compute utilization, causing both controllers to fight. The standard approach is: use VPA on memory (it is safer to let VPA right-size memory), and HPA on CPU or a custom metric.

Predictive and Scheduled Scaling

Reactive scaling always lags the load. The metric has to rise, the scaler has to detect it, new instances have to start, and health checks have to pass. For services with predictable traffic patterns, this lag is avoidable.

Scheduled scaling sets capacity at a specific level before the known load spike arrives. A service that processes payroll runs every Friday night gets scaled to 3x capacity at 5pm Friday and scaled back at 2am Saturday. No metric, no reactive logic, just a calendar.

Predictive scaling uses historical data to forecast future load and provisions capacity ahead of time. AWS Auto Scaling has a predictive scaling mode that analyzes 14 days of CloudWatch metrics and generates an hourly scaling schedule for the next 48 hours. It adjusts that schedule daily based on observed actuals.

The TypeScript below shows how you might implement a simple forecasting layer that generates a scheduled scaling action:

interface LoadForecast {
  timestamp: number;
  predictedReplicas: number;
  confidence: number;
}

interface HistoricalDataPoint {
  timestamp: number;
  replicas: number;
  cpuUtilization: number;
}

function computeWeeklyAverageForecast(
  history: HistoricalDataPoint[],
  targetHour: number,
  targetDayOfWeek: number,
  bufferMultiplier: number = 1.2
): LoadForecast {
  const matching = history.filter((dp) => {
    const date = new Date(dp.timestamp);
    return date.getHours() === targetHour && date.getDay() === targetDayOfWeek;
  });

  if (matching.length === 0) {
    return { timestamp: Date.now(), predictedReplicas: 1, confidence: 0 };
  }

  const avgReplicas = matching.reduce((sum, dp) => sum + dp.replicas, 0) / matching.length;
  const buffered = Math.ceil(avgReplicas * bufferMultiplier);

  return {
    timestamp: Date.now(),
    predictedReplicas: buffered,
    confidence: Math.min(matching.length / 10, 1),
  };
}

The bufferMultiplier is deliberate. A forecast gives you the expected average, not the peak. Scaling to the average means you will be under-provisioned during the peak portion of that window. Scale to the average plus 20% as a starting point, and tighten it once you have measured actual versus predicted.

Serverless Autoscaling

Serverless functions (AWS Lambda, Cloudflare Workers, Google Cloud Run) autoscale differently from VM or container-based infrastructure.

The autoscaling is implicit. The platform creates a new execution environment for each concurrent request, subject to configured concurrency limits and account-level limits. There are no cooldowns in the traditional sense. The platform scales to zero when there is no traffic and scales to the concurrency limit nearly instantaneously.

The scaling properties you actually control in serverless are:

• Concurrency limits: maximum simultaneous invocations (Lambda’s reserved concurrency, Cloud Run’s max instances)
• Provisioned concurrency: pre-warmed instances that eliminate cold starts, at the cost of paying for idle capacity
• Timeout: maximum duration per invocation, which affects how many concurrent invocations you need for a given throughput

The cold start problem is real for latency-sensitive workloads. Provisioned concurrency solves it, but it is essentially the same tradeoff as minimum replicas in a container autoscaler: you pay for capacity even when it is not being used, to eliminate startup latency.

Tradeoffs Summary

Approach	Response time	Cost	Flapping risk	Best for
Step scaling	Fast (explicit)	Moderate	High if thresholds are narrow	Discrete, predictable load changes
Target tracking	90 to 180s lag	Efficient	Low with proper cooldown	Stateless web services with proportional load
Custom metric (queue)	60 to 120s lag	Very efficient	Low	Async workers, batch consumers
Scheduled scaling	Zero (proactive)	Higher (reserves capacity)	None	Predictable traffic patterns
Predictive scaling	Zero (proactive)	Moderate overhead	Low	Weekly/daily patterns, large fleets
VPA (memory only)	Pod restart cost	Efficient over time	None	Right-sizing long-running pods
Serverless implicit	Near-instant	Pay-per-use	None	Spiky, unpredictable, short-duration workloads

Production Considerations

Set a minimum replica count that reflects your actual availability requirements, not cost minimization. A minimum of 1 replica means a single failure takes your service to zero capacity while a replacement starts. For most services, 2 is the practical minimum. For services with SLAs, 3.

Alarm on scale-to-max events. When your autoscaler hits the maximum replica count, it stops scaling. At that point, your service is shedding load or degrading. That event should page someone, not be silently swallowed by the autoscaling infrastructure.

Test scale-up under load before it matters. Run a load test that triggers your autoscaler. Measure actual time from trigger to healthy new instances receiving traffic. That number is your actual gap time, and it should inform both your scale-up cooldown and your headroom target.

Separate scaling policies by workload type. A monolith autoscaler that responds to an average of CPU across a mixed workload is much less precise than separate services with separate scaling policies. If your API handles both synchronous requests and background processing, separate the two. They have different scaling signals and different acceptable latency for capacity changes.

Watch for metric lag in scale-down decisions. Some metrics are reported with a delay. CloudWatch metrics for EC2 can lag by 1 to 5 minutes depending on the resolution you have configured. A 5-minute lag plus a 5-minute scale-down cooldown means your autoscaler is making scale-down decisions based on 10-minute-old data. Either increase resolution (at cost) or increase your scale-down cooldown to account for it.

Closing

Autoscaling is infrastructure plumbing. It works correctly when you have the right metric, the right target value, and cooldowns that reflect the actual time your infrastructure takes to stabilize. Every exotic scaling algorithm in the world cannot compensate for a metric that lags the problem by five minutes, or a cooldown that expires before new instances are healthy.

Start with target tracking on a metric that reflects user experience, not infrastructure internals. Set generous cooldowns. Alarm on scale-to-max. Everything else is tuning.