Progressive Delivery with Argo Rollouts: Canary Analysis, Automated Blue-Green, and Metrics-Driven Deployment Gates in Kubernetes

Kubernetes Deployment objects ship all-or-nothing. You bump the image tag, the rolling update starts, and within a few minutes all pods are running the new version. If something breaks at 100% traffic, you roll back. The blast radius is everyone.

Progressive delivery flips that model. You route a fraction of traffic to the new version, run automated checks against real production signals, and promote or abort based on what the data says. Argo Rollouts is the Kubernetes-native controller that makes this concrete: Rollout CRDs, AnalysisTemplate CRDs, and first-class integrations with Istio, NGINX, AWS ALB, and the major metrics backends.

This article is a production playbook. You will leave with working YAML for Rollout and AnalysisTemplate resources, a TypeScript snippet for a custom promotion webhook, and a clear map of where the failure modes live.

Rollout vs. Deployment: What Actually Changes

A Rollout is a drop-in replacement for a Deployment at the spec level. The pod template, replicas, selectors, and resource limits move over unchanged. What changes is the update strategy field.

A Deployment uses RollingUpdate or Recreate. A Rollout uses canary or blueGreen, and those strategies unlock step-based promotion, traffic weight control, and analysis hooks that the core Deployment controller does not have.

The Argo Rollouts controller watches Rollout objects and manages the underlying ReplicaSets directly. It also manages a traffic routing layer by patching VirtualService weights (Istio), Ingress annotations (NGINX), or TargetGroup weights (ALB), depending on what you configure. When no traffic shaping provider is configured, it falls back to pod-count-based weight approximation.

One important detail: if you are migrating from a Deployment, you need to scale the old Deployment to zero or delete it. Two controllers managing the same pod selector will fight.

Canary Strategy: Steps, Weights, and Traffic Shaping

The canonical canary configuration is a series of steps. Each step either sets a traffic weight, pauses for a duration or manual promotion, or triggers an analysis run.

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: payments-api
  namespace: production
spec:
  replicas: 10
  selector:
    matchLabels:
      app: payments-api
  template:
    metadata:
      labels:
        app: payments-api
    spec:
      containers:
        - name: payments-api
          image: payments-api:v2.4.1
          ports:
            - containerPort: 8080
  strategy:
    canary:
      canaryService: payments-api-canary
      stableService: payments-api-stable
      trafficRouting:
        istio:
          virtualService:
            name: payments-api-vs
            routes:
              - primary
      steps:
        - setWeight: 5
        - analysis:
            templates:
              - templateName: error-rate-check
            args:
              - name: service-name
                value: payments-api
        - pause: { duration: 10m }
        - setWeight: 25
        - pause: { duration: 10m }
        - setWeight: 50
        - analysis:
            templates:
              - templateName: latency-check
            args:
              - name: service-name
                value: payments-api
        - setWeight: 100

The canaryService and stableService are separate Service objects that the controller patches. The VirtualService routes traffic by weight between those two services. At 5% weight, Istio sends 5% of requests to the canary pods regardless of how many canary pods exist. This is the key distinction from pod-count approximation: with Istio (or NGINX weighted routing), you get deterministic traffic percentages. Without a traffic shaping provider, the weight approximates the pod fraction, so at 5% weight with 10 replicas you get 1 canary pod and 9 stable pods, which is actually 10%, not 5%.

When a Full Service Mesh Is Required

Use Istio or another service mesh when you need:

• Sub-10% traffic weights with meaningful pod counts
• Header-based routing for internal QA traffic on the canary
• mTLS between services during a canary
• Per-request observability (traces that distinguish canary vs. stable)

For simple cases where weight approximation at pod-count granularity is acceptable, NGINX ingress weighted routing or ALB target group weighting is sufficient and much cheaper operationally.

Header-based routing uses an Istio VirtualService match rule that is separate from the weight-based rule:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: payments-api-vs
  namespace: production
spec:
  hosts:
    - payments-api
  http:
    - name: canary-header
      match:
        - headers:
            x-canary:
              exact: "true"
      route:
        - destination:
            host: payments-api-canary
    - name: primary
      route:
        - destination:
            host: payments-api-stable
          weight: 95
        - destination:
            host: payments-api-canary
          weight: 5

This lets QA engineers send requests to the canary by adding x-canary: true without affecting production traffic weights.

AnalysisTemplate: The Metrics Gate

An AnalysisTemplate defines what success looks like during a step. The controller creates an AnalysisRun when the step fires, the run queries the metrics backend on a configurable interval, and the result is Successful, Failed, or Inconclusive.

apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate-check
  namespace: production
spec:
  args:
    - name: service-name
  metrics:
    - name: error-rate
      interval: 1m
      count: 5
      successCondition: result[0] < 0.01
      failureCondition: result[0] >= 0.05
      failureLimit: 1
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc.cluster.local:9090
          query: |
            sum(rate(http_requests_total{
              service="{{ args.service-name }}",
              status=~"5.."
            }[2m]))
            /
            sum(rate(http_requests_total{
              service="{{ args.service-name }}"
            }[2m]))

    - name: p99-latency
      interval: 1m
      count: 5
      successCondition: result[0] < 0.5
      failureCondition: result[0] >= 1.0
      failureLimit: 2
      provider:
        prometheus:
          address: http://prometheus.monitoring.svc.cluster.local:9090
          query: |
            histogram_quantile(0.99, sum(rate(
              http_request_duration_seconds_bucket{
                service="{{ args.service-name }}"
              }[2m]
            )) by (le))

count: 5 with interval: 1m means the analysis runs for at least 5 minutes, querying once per minute. failureLimit: 1 means one failure measurement aborts the rollout immediately. failureCondition and successCondition are Go templates evaluated against the query result.

The AnalysisRun lifecycle has four terminal states: Successful (all metrics passed), Failed (failure limit exceeded), Inconclusive (result was within neither success nor failure bands), and Error (the query itself failed). Only Successful promotes the rollout to the next step. The other three abort by default, though you can configure inconclusive to pause for manual intervention instead.

Datadog and Web Endpoint Providers

Prometheus is the most common provider, but Argo Rollouts ships providers for Datadog, New Relic, CloudWatch, and a generic web endpoint.

For a web endpoint, the controller sends a POST with the current rollout state and expects a JSON response with a status field. This is how you wire in custom business logic: conversion rate checks, A/B test significance, external anomaly detection.

provider:
  web:
    url: https://analysis.internal/api/check
    headers:
      - key: Authorization
        value: "Bearer {{ args.token }}"
    jsonPath: "{$.status}"
    successCondition: result == "pass"

Blue-Green Strategy

Blue-green with Argo Rollouts maintains two full replica sets: the active (blue) and the preview (green). Traffic goes 100% to active until you explicitly promote.

strategy:
  blueGreen:
    activeService: payments-api-active
    previewService: payments-api-preview
    autoPromotionEnabled: false
    scaleDownDelaySeconds: 30
    prePromotionAnalysis:
      templates:
        - templateName: smoke-test
      args:
        - name: service-name
          value: payments-api-preview
    postPromotionAnalysis:
      templates:
        - templateName: error-rate-check
      args:
        - name: service-name
          value: payments-api-active

prePromotionAnalysis runs against the preview service before any traffic shift. postPromotionAnalysis runs after the cutover while the old replica set is still alive (before scaleDownDelaySeconds expires). If post-promotion analysis fails, the controller cuts traffic back to the old replica set. This is the strongest rollback guarantee Argo Rollouts offers: the old version is still warm and available.

autoPromotionEnabled: false requires a manual promotion via kubectl argo rollouts promote payments-api or an API call. Setting autoPromotionSeconds to a number enables automatic promotion after that many seconds if analysis passes.

Traffic Splitting Math

With a service mesh doing weight-based routing:

• Weight applies to request percentage, not pod count.
• 5% weight means 5% of requests go to canary, regardless of replica counts on each side.
• The number of canary pods affects latency and capacity, not the traffic percentage.

With pod-count approximation (no traffic shaping provider):

• Weight is approximated by the ratio of canary pods to total pods.
• setWeight: 20 with 10 total replicas gives 2 canary pods and 8 stable pods.
• With fewer total replicas the granularity is coarser. At 2 replicas, you can only achieve 0%, 50%, or 100%.

This is why the mesh requirement exists for services where a 5% canary is meaningful. At 2 replicas without a mesh, 5% rounds to 0 canary pods, which is no canary at all.

Argo CD Integration for GitOps Progressive Delivery

With Argo CD managing the application, the Rollout resource lives in Git alongside the AnalysisTemplates. Argo CD syncs the Rollout definition, and Argo Rollouts handles the runtime progression. The two controllers cooperate but operate at different layers.

One friction point: Argo CD’s health check for a Rollout needs to understand rollout-specific status fields. Argo CD 2.x ships a built-in health check for Rollout resources that marks the application degraded when a rollout is paused or has failed. Older Argo CD versions need a custom health check Lua script.

A second friction point: if Argo CD is configured with prune: true, it will delete the ReplicaSets that Argo Rollouts is managing during a canary. Disable pruning on ReplicaSets using an annotation:

# On each ReplicaSet or via Argo CD app config
metadata:
  annotations:
    argocd.argoproj.io/managed-by: argo-rollouts

The cleaner approach is to configure the Argo CD application to ignore ReplicaSet drift during active rollouts using ignoreDifferences on the application spec.

Automated Promotion Webhook in TypeScript

You can build a lightweight promotion controller that listens to rollout events and makes promotion decisions based on custom logic, for example checking a feature flag service or a business metric that Prometheus does not expose.

import { createServer, IncomingMessage, ServerResponse } from "http";

interface RolloutWebhookPayload {
  rolloutName: string;
  namespace: string;
  step: number;
  canaryWeight: number;
  stableRevision: string;
  canaryRevision: string;
}

interface AnalysisCheckResponse {
  status: "pass" | "fail" | "inconclusive";
  reason?: string;
}

async function checkBusinessMetric(
  payload: RolloutWebhookPayload
): Promise<AnalysisCheckResponse> {
  // Example: query an internal metric store for conversion rate
  const response = await fetch(
    `https://metrics.internal/api/conversion-rate?` +
      new URLSearchParams({
        service: payload.rolloutName,
        revision: payload.canaryRevision,
        window: "10m",
      }),
    {
      headers: { Authorization: `Bearer ${process.env.METRICS_TOKEN}` },
    }
  );

  if (!response.ok) {
    return { status: "inconclusive", reason: "metrics API unreachable" };
  }

  const data = (await response.json()) as { rate: number; sampleSize: number };

  // Require minimum sample size before making a call
  if (data.sampleSize < 100) {
    return { status: "inconclusive", reason: "insufficient sample size" };
  }

  // Canary conversion rate must be within 5% of stable baseline
  const baselineResponse = await fetch(
    `https://metrics.internal/api/conversion-rate?` +
      new URLSearchParams({
        service: payload.rolloutName,
        revision: payload.stableRevision,
        window: "10m",
      }),
    {
      headers: { Authorization: `Bearer ${process.env.METRICS_TOKEN}` },
    }
  );

  const baseline = (await baselineResponse.json()) as { rate: number };
  const delta = (data.rate - baseline.rate) / baseline.rate;

  if (delta < -0.05) {
    return {
      status: "fail",
      reason: `conversion rate degraded by ${(delta * 100).toFixed(1)}%`,
    };
  }

  return { status: "pass" };
}

const server = createServer(
  async (req: IncomingMessage, res: ServerResponse) => {
    if (req.method !== "POST" || req.url !== "/api/check") {
      res.writeHead(404);
      res.end();
      return;
    }

    const body = await new Promise<string>((resolve) => {
      let data = "";
      req.on("data", (chunk) => (data += chunk));
      req.on("end", () => resolve(data));
    });

    const payload = JSON.parse(body) as RolloutWebhookPayload;
    const result = await checkBusinessMetric(payload);

    res.writeHead(200, { "Content-Type": "application/json" });
    res.end(JSON.stringify(result));
  }
);

server.listen(3000, () => {
  console.log("Rollout analysis webhook listening on :3000");
});

This webhook is referenced in the AnalysisTemplate using the web provider. The jsonPath: "{$.status}" extracts the status field, and successCondition: result == "pass" controls promotion.

Tradeoffs

Factor	Canary (mesh)	Canary (pod-count)	Blue-Green
Traffic precision	Exact percentages	Coarse, replica-count dependent	Binary (0% or 100%)
Resource cost	1x replicas + small canary	1x replicas + small canary	2x replicas during rollout
Rollback speed	Reroute in seconds	New ReplicaSet takes time	Cutover in seconds
Blast radius control	Fine-grained	Coarse	None until cutover
Operational complexity	High (mesh required)	Low	Medium
Best for	Stateless APIs, high-traffic services	Low-traffic services, simple routing	Stateful services, strict isolation

Production Failure Cases

False-positive aborts from flapping metrics. A Prometheus query that measures a 2-minute rate window during a low-traffic period can produce high variance. One noisy data point triggers failureLimit: 1 and aborts a healthy rollout. Fix this by increasing count, using a longer rate window in the query, or raising failureLimit for low-traffic services.

Drift during long rollouts. A canary paused for 2 hours while an analysis runs is a version of your service running in production that has not been promoted. If a hotfix goes out to stable during that window, the canary is now based on a stale version. Track the stable revision SHA during long rollouts and abort automatically if stable is updated underneath the canary. The Rollout status exposes stableRS and canaryRS for this purpose.

Rollback behavior. When an analysis fails or a rollout is manually aborted, the controller sets weight back to 0 for the canary and scales down the canary ReplicaSet. Stable continues serving. In blue-green, the active service stays on the old ReplicaSet if post-promotion analysis fails and the scale-down delay has not elapsed. After the delay, the old set is gone and rollback requires a new rollout.

AnalysisRun timeout. If the metrics backend is unavailable during analysis, the run enters Error state. Whether that aborts or pauses the rollout depends on your analysis.errorLimit setting. Defaulting to abort on error is safer than defaulting to promote.

Production Considerations

Before deploying Argo Rollouts at scale:

• Install the Argo Rollouts controller with resource limits set. A controller managing hundreds of rollouts simultaneously needs real CPU and memory headroom.
• Pin the controller version in your cluster and version-control your AnalysisTemplates. Rolling back a bad AnalysisTemplate requires redeploying the template, which is a separate operation from rolling back the application.
• Use kubectl argo rollouts get rollout <name> --watch during early deployments. The status tree shows each step, the current analysis run result, and the current weights in real time.
• Namespace-scope your AnalysisTemplates when different teams own different services. A shared ClusterAnalysisTemplate is convenient but couples teams to each other’s analysis logic.
• Test your abort path explicitly. Run a canary with a deliberately broken image and verify that the analysis fires, the abort triggers, and stable continues serving. Do this in staging before you need it in production.

Closing

Argo Rollouts does not make deployment safer by default. It makes deployment safety programmable. An AnalysisTemplate that queries the wrong metric, or a failureLimit set too high, or a canary weight so small it gets no meaningful traffic, produces the illusion of progressive delivery without the actual protection.

The value is in the analysis layer. Get your Prometheus queries right for the service you are deploying. Test the failure path. Then set weights small and let the data do the promoting.

The rollout YAML is the easy part.