Automated TLS Certificate Management: ACME Protocol, Let's Encrypt, and Certificate Rotation in Production

Manual TLS certificate management is an operational liability. Someone forgets a renewal, a calendar reminder gets ignored, and at 2 AM on a Sunday you’re debugging why users can’t reach checkout. The infrastructure to automate this completely has existed for years. Most teams still haven’t fully adopted it, usually because the ACME protocol feels like a black box and cert-manager’s YAML surface area is intimidating.

This article walks through every layer: the protocol itself, the two challenge types and when each applies, how to configure cert-manager in Kubernetes, Caddy’s zero-config approach, Cloudflare’s edge certificate options, rotation strategies that don’t cause downtime, and the monitoring setup that gives you early warning before anything expires.

The ACME Protocol Flow

ACME (Automatic Certificate Management Environment, RFC 8555) is what Let’s Encrypt and other CAs use to automate the prove-you-own-this-domain step. Understanding the flow matters when things break.

Account registration. Your ACME client generates a key pair and registers with the CA directory endpoint. The private key signs all subsequent requests. If you lose this key, you lose the ability to revoke certificates issued under that account (though you can re-register with a new key and get new certs).

Order creation. The client submits an order for a certificate, listing the identifiers (domain names) it wants covered. The CA returns a list of authorization objects, one per identifier.

Challenges. For each authorization, the CA offers challenge options. The client picks one, provisions the required token in the specified location, and notifies the CA it’s ready. The CA fetches or queries that location to verify ownership.

Finalization. Once all authorizations are valid, the client submits a CSR. The CA issues the certificate. The client fetches and stores it.

Renewal. ACME clients typically renew at 60-70% of the certificate’s lifetime. Let’s Encrypt issues 90-day certificates, so renewal happens around day 60. This short lifetime is intentional: it bounds the damage window if a private key is compromised, and it forces automation because 90 days is short enough that manual processes reliably fail.

HTTP-01 vs DNS-01: The Real Tradeoffs

Both challenge types prove domain ownership, but they have meaningfully different operational profiles.

HTTP-01 requires the ACME client to serve a token at http://<domain>/.well-known/acme-challenge/<token>. The CA performs an HTTP GET to that URL. This works well for public-facing services where port 80 is reachable and you control the web server. It does not work for internal services, wildcard certificates, or environments where port 80 is blocked by a firewall or load balancer that doesn’t pass the challenge path through.

DNS-01 requires the ACME client to create a _acme-challenge.<domain> TXT record containing the token. The CA queries DNS to verify. This works for any domain you control, including internal ones, wildcard certificates (the only challenge type that supports *.example.com), and services not exposed on port 80. The tradeoff is that you need programmatic DNS API access, and DNS propagation delays (typically 30-300 seconds, occasionally longer) can cause challenges to fail if you don’t wait for propagation before notifying the CA.

Criterion	HTTP-01	DNS-01
Wildcard support	No	Yes
Internal/private domains	No	Yes
Port 80 requirement	Yes	No
DNS API access required	No	Yes
Propagation delay risk	None	Present
Multi-region complexity	Low	Medium
Revocation scope	Per domain	Per domain

For public-facing Kubernetes services where you control ingress, HTTP-01 is operationally simpler. For wildcard certs, internal clusters, or environments with restrictive network policies, DNS-01 is the only option.

cert-manager in Kubernetes

cert-manager is the standard approach for certificate automation in Kubernetes. It introduces three main custom resources: Issuer, ClusterIssuer, and Certificate.

Issuer vs ClusterIssuer. An Issuer is namespace-scoped. A ClusterIssuer is cluster-scoped and can issue certificates for any namespace. Use ClusterIssuer unless you have a specific reason to isolate certificate issuance per namespace.

A ClusterIssuer for Let’s Encrypt production using DNS-01 with Cloudflare:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - dns01:
          cloudflare:
            email: ops@example.com
            apiTokenSecretRef:
              name: cloudflare-api-token
              key: api-token

The cloudflare-api-token secret needs a Cloudflare API token with Zone:DNS:Edit permissions scoped to the relevant zones. Do not use your global API key.

A Certificate resource requesting a wildcard:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-example-com
  namespace: production
spec:
  secretName: wildcard-example-com-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  commonName: "*.example.com"
  dnsNames:
    - "*.example.com"
    - "example.com"
  duration: 2160h   # 90 days
  renewBefore: 720h # Renew 30 days before expiry

cert-manager stores the issued certificate and private key in the named Secret. Ingress resources reference this secret in their tls block. cert-manager monitors the certificate’s expiry and triggers renewal automatically when renewBefore threshold is hit.

For HTTP-01 with an nginx ingress controller:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-http01
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: ops@example.com
    privateKeySecretRef:
      name: letsencrypt-http01-account-key
    solvers:
      - http01:
          ingress:
            class: nginx

cert-manager creates a temporary Ingress resource to serve the challenge token, then cleans it up after validation.

Caddy’s Automatic HTTPS

Caddy handles TLS entirely without configuration when you give it a public domain. It registers an ACME account, obtains a certificate, stores it in its managed certificate storage (local disk or a configured storage backend), and renews before expiry. The Caddyfile:

example.com {
  reverse_proxy localhost:8080
}

That’s it. Caddy handles HTTP-01 by default. For DNS-01, you need a Caddy build with the DNS provider plugin compiled in, then configure the DNS challenge:

example.com {
  tls {
    dns cloudflare {env.CLOUDFLARE_API_TOKEN}
  }
  reverse_proxy localhost:8080
}

Caddy’s automatic renewal runs at 2/3 of the certificate lifetime. For multi-instance deployments, you need a shared storage backend (Redis, Consul, or an S3-compatible store via the caddy-storage-s3 module) to avoid every instance independently obtaining certificates and hitting rate limits.

Cloudflare Edge Certificates and Origin CA

If Cloudflare sits in front of your origin, you have two certificate boundaries to manage: the edge certificate (Cloudflare to visitors) and the origin certificate (your server to Cloudflare).

Edge certificates are managed entirely by Cloudflare. With Universal SSL enabled, Cloudflare automatically provisions and renews a certificate covering your domain and first-level subdomains. Advanced Certificate Manager extends this to wildcard coverage and custom certificate validity periods. You don’t manage edge certificate renewal at all.

Origin certificates are the certs your server presents to Cloudflare. You have two reasonable options:

1. Let’s Encrypt certificates on the origin, renewed by cert-manager or Certbot. Cloudflare validates these normally when SSL mode is Full (strict).

2. Cloudflare Origin CA certificates. These are issued by Cloudflare’s own CA, trusted only by Cloudflare’s edge (not by browsers directly). They can be valid for up to 15 years, which eliminates renewal automation concerns for origin certs if you’re comfortable with Cloudflare-only trust. Issue via the Cloudflare API:

const response = await fetch(
  `https://api.cloudflare.com/client/v4/certificates`,
  {
    method: "POST",
    headers: {
      "X-Auth-Email": process.env.CF_EMAIL!,
      "X-Auth-Key": process.env.CF_API_KEY!,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      hostnames: ["example.com", "*.example.com"],
      requested_validity: 5475, // 15 years in days
      request_type: "origin-rsa",
      csr: csrPem, // Generated locally, private key never leaves your server
    }),
  }
);

const { result } = await response.json();
// result.certificate is the issued PEM
// Store result.certificate alongside your locally generated private key

The CSR is generated locally; the private key never leaves your infrastructure. This is the right approach for origin certs when Cloudflare is your only ingress path.

Certificate Rotation Without Downtime

Rotation is where most teams create their own incidents. The goal is to replace a certificate before it expires without dropping in-flight TLS connections.

Graceful reload. Most servers support reloading TLS configuration without restarting. nginx uses nginx -s reload, which completes in-flight connections before workers restart. This is safe for certificate rotation. In Kubernetes, cert-manager can trigger pod restarts or use projected volumes, but the standard approach is mounting the TLS secret as a volume and using a sidecar or init container to reload nginx when the secret changes.

A simple TypeScript utility to check certificate expiry before triggering rotation:

import * as tls from "tls";

interface CertInfo {
  subject: string;
  validTo: Date;
  daysRemaining: number;
}

async function checkCertExpiry(hostname: string, port = 443): Promise<CertInfo> {
  return new Promise((resolve, reject) => {
    const socket = tls.connect({ host: hostname, port, servername: hostname }, () => {
      const cert = socket.getPeerCertificate();
      socket.destroy();

      if (!cert || !cert.valid_to) {
        return reject(new Error("No certificate returned"));
      }

      const validTo = new Date(cert.valid_to);
      const daysRemaining = Math.floor(
        (validTo.getTime() - Date.now()) / (1000 * 60 * 60 * 24)
      );

      resolve({
        subject: cert.subject?.CN ?? hostname,
        validTo,
        daysRemaining,
      });
    });

    socket.on("error", reject);
  });
}

// Usage in a monitoring script
const domains = ["api.example.com", "app.example.com", "example.com"];

for (const domain of domains) {
  const info = await checkCertExpiry(domain);
  if (info.daysRemaining < 14) {
    console.warn(`WARN: ${domain} expires in ${info.daysRemaining} days`);
  }
}

Dual-cert overlap windows. When rotating to a new CA or key type (e.g., migrating from RSA to ECDSA), configure your server to serve both certificates simultaneously. nginx supports this since 1.11.0:

ssl_certificate     /etc/ssl/certs/example.com.rsa.crt;
ssl_certificate_key /etc/ssl/private/example.com.rsa.key;
ssl_certificate     /etc/ssl/certs/example.com.ecdsa.crt;
ssl_certificate_key /etc/ssl/private/example.com.ecdsa.key;

nginx selects the appropriate certificate based on client capabilities. Run both for at least one full certificate validity period before removing the old one, or at minimum for long enough that all cached DNS resolutions and persistent connections have cycled.

Monitoring Certificate Expiry

Don’t rely solely on automation. Automation can fail, and you want independent alerting.

Prometheus with ssl_exporter. The ribbonmx/ssl_exporter scrapes TLS endpoints and exposes certificate metadata as metrics. Configure it to check your domains:

# ssl_exporter config
modules:
  https:
    prober: https
    timeout: 10s
    tls_config:
      insecure_skip_verify: false

# Prometheus scrape config
scrape_configs:
  - job_name: ssl_certificates
    metrics_path: /probe
    params:
      module: [https]
    static_configs:
      - targets:
          - example.com:443
          - api.example.com:443
          - "*.example.com:443"
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: ssl-exporter:9219

Key metric: ssl_cert_not_after (Unix timestamp of certificate expiry). Alert rules:

groups:
  - name: tls_certificates
    rules:
      - alert: CertificateExpiringSoon
        expr: (ssl_cert_not_after - time()) / 86400 < 14
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "Certificate expiring in {{ $value | humanizeDuration }}"
          description: "{{ $labels.instance }} certificate expires in less than 14 days"

      - alert: CertificateExpiryCritical
        expr: (ssl_cert_not_after - time()) / 86400 < 3
        for: 15m
        labels:
          severity: critical
        annotations:
          summary: "Certificate expiring in {{ $value | humanizeDuration }}"
          description: "{{ $labels.instance }} certificate expires in less than 3 days"

Also monitor cert-manager’s own metrics. The certmanager_certificate_expiration_timestamp_seconds gauge is exposed by cert-manager’s controller and reflects the certificate object’s expiry as cert-manager sees it (not necessarily what your server is currently serving, which matters if the TLS secret isn’t being mounted correctly).

Wildcard Certificates and Multi-Domain SANs

Wildcard certificates (*.example.com) cover one level of subdomains. api.example.com is covered; v2.api.example.com is not. If you need deeper nesting, you need either additional wildcard certs (*.api.example.com) or explicit SANs.

Multi-domain SANs let a single certificate cover unrelated domains on the same server. This is useful when a server hosts multiple sites and you want one cert rather than several. cert-manager handles this naturally:

spec:
  dnsNames:
    - "example.com"
    - "www.example.com"
    - "api.example.com"
    - "app.example.com"
    - "*.internal.example.com"

Let’s Encrypt limits SANs to 100 per certificate. For most use cases this is more than enough. Note that adding a new SAN to an existing certificate requires reissuance, not renewal.

Common Failure Modes

DNS propagation delays with DNS-01. The ACME client sets the TXT record and immediately notifies the CA, but the CA queries authoritative nameservers that may still be serving the old TTL. cert-manager has a dns01RecursiveNameservers field to specify resolvers and a dns01RecursiveNameserversOnly flag. Set TTLs on your _acme-challenge records to 60 seconds or less. Some DNS providers have higher minimum TTLs; check yours before assuming rapid propagation.

Let’s Encrypt rate limits. The main limits: 50 certificates per registered domain per week, 5 duplicate certificates per week, 5 failed validations per account per hostname per hour. The duplicate certificate limit catches teams who re-run provisioning scripts without checking whether a valid cert already exists. Always check before requesting. Use the staging endpoint (https://acme-staging-v02.api.letsencrypt.org/directory) during development; it has much higher limits and produces untrusted certs so you can test the full flow without burning production quota.

OCSP stapling failures. OCSP stapling improves performance by letting the server pre-fetch and cache the CA’s certificate status response. If your server is configured to require a valid OCSP response but can’t reach the OCSP responder (network policy, firewall egress rule, CA outage), it will fail to serve the staple and depending on your ssl_stapling_verify configuration either silently serve without stapling or refuse connections. Test OCSP stapling explicitly: openssl s_client -connect example.com:443 -status and check for OCSP Response Status: successful. Ensure your servers have outbound access to ocsp.int-x3.letsencrypt.org or whatever the OCSP URL is in the cert’s AIA extension.

Secret not updated in running pod. In Kubernetes, cert-manager updates the TLS Secret when it renews a certificate. If the Secret is mounted as a volume, Kubernetes eventually propagates the update to the pod’s filesystem (typically within 60-90 seconds for default kubelet sync). If the Secret is consumed as an environment variable, the pod must restart to pick up the new value. This is a common source of confusion: cert-manager renewed the cert, the Secret is current, but the pod is still serving the old cert because the env var path doesn’t support live updates.

Account key loss. If the ACME account private key is lost, you can’t revoke previously issued certificates through that account. You can still request new certificates with a new account. For critical infrastructure, back up the account key secret from Kubernetes and store it in your secrets manager.

Production Considerations

A few operational decisions that compound over time:

Keep staging and production ACME issuers separate in cert-manager. Accidentally pointing production workloads at the staging issuer produces untrusted certificates. Name them letsencrypt-staging and letsencrypt-prod, and require explicit selection.

For clusters with many services, cert-manager’s reconciliation loop can produce significant Let’s Encrypt API traffic at renewal time. Stagger renewBefore across certificate resources if you have dozens renewing simultaneously, or accept that rate limits may throttle renewals (cert-manager retries with backoff, so this usually resolves itself, but it can cause brief validity gaps if the cert expires before renewal completes).

When running cert-manager in a high-availability configuration (multiple replicas), only one instance processes a given Certificate resource at a time due to leader election. This is correct behavior, not a problem. The leader handles renewal; the others are standby. Don’t try to parallelize cert-manager replicas at the application level.

For services behind a CDN or DDoS mitigation layer that terminates TLS at the edge, the origin certificate expiry is less visible in end-to-end tests because the TLS session you observe is with the edge, not the origin. Your monitoring needs to check the origin cert directly (bypassing the CDN) in addition to the public endpoint.

Certificate transparency logs are public. Every certificate you obtain via public ACME CAs appears in CT logs within seconds. If your internal subdomain names are sensitive, this is a disclosure concern. Options: use a private CA (cert-manager supports this with the CA issuer type), use DNS-01 with a wildcard so individual subdomain names don’t appear in SANs, or accept the disclosure and treat subdomain names as non-secret.

The combination of short-lived certificates (90 days), automated renewal (cert-manager or Caddy), and independent monitoring (ssl_exporter with Prometheus alerts) is the standard. Manual certificates in production should be the exception and should have documented justification.