DNS Architecture for Production Systems: Record Types, Failover, CDN Routing, and Debugging Resolution

DNS is one of those layers most engineers only think about when something is broken. You add a record, wait for propagation, and forget about it until a user reports that your site is unreachable in Singapore. At scale, that approach stops working. DNS becomes a load-balancing layer, a failover mechanism, a security surface, and a performance bottleneck all at once.

This article goes deeper than “A records point to IPs.” It covers the resolution chain, every record type that matters in production, DNS-based failover with health checks, GeoDNS and latency-based routing, CDN integration patterns, DNSSEC and CAA security, common debugging scenarios, and practical commands you can run right now.

The Resolution Chain Most Engineers Skip

When a client resolves api.example.com, the chain is: local OS cache, then local resolver (usually your router or ISP), then a recursive resolver (like 8.8.8.8 or 1.1.1.1), then a root nameserver, then the TLD nameserver for .com, then your authoritative nameserver.

The recursive resolver does the heavy lifting. It walks the delegation chain and caches results at every step, respecting TTL values. This is why TTL is so important: it controls how long each layer in that chain can serve a cached answer before asking again.

# Trace the full resolution chain for a domain
dig +trace api.example.com

# Check which authoritative nameservers are responsible
dig NS example.com

# Query a specific nameserver directly (bypass your local cache)
dig @ns1.example.com api.example.com

# Check what your local resolver has cached vs. the authoritative answer
dig api.example.com            # local resolver
dig @8.8.8.8 api.example.com  # Google's resolver (different cache)
dig +short api.example.com     # just the answer, no headers

The +trace flag is the most useful thing most engineers have never run. It shows you exactly where delegation breaks down, which is the actual problem in the majority of “DNS is broken” incidents.

TTL Implications You Need to Understand

TTL is not a suggestion. It is a contract between you and every caching resolver on the internet. If you set a TTL of 3600 (one hour) and then need to change your IP quickly during an incident, you are stuck waiting up to an hour for caches to expire.

Recommended TTL strategy:

• Normal operations: 300-3600 seconds (5 minutes to 1 hour)
• Before a planned migration: lower TTL to 60 seconds 24 hours in advance
• Active incident requiring fast IP change: you need TTL to already be low

Low TTL comes with cost. Every resolver that cannot use a cached answer must query your authoritative nameserver. At high query volume, this matters. A TTL of 30 seconds on a domain that gets 1 million requests per hour means your nameserver handles far more load than if TTL were 300.

The other TTL trap is negative caching (NXDOMAIN responses). An authoritative nameserver can tell resolvers how long to cache a “this record does not exist” response via the SOA record’s minimum TTL field. If you create a record that was previously nonexistent, clients still get NXDOMAIN from cache until that negative TTL expires.

Record Types and When to Use Each

Record	Purpose	Key Behavior
A	Maps hostname to IPv4 address	Direct, no indirection
AAAA	Maps hostname to IPv6 address	Dual-stack requires both A and AAAA
CNAME	Alias one hostname to another	Cannot coexist with other records at the same name
ALIAS / ANAME	Alias at the zone apex	Resolves CNAME-like but returns A/AAAA (provider-specific)
MX	Mail exchange routing	Priority value controls order
TXT	Arbitrary text	Used for SPF, DKIM, DMARC, domain verification
SRV	Service discovery	Encodes host, port, priority, weight
CAA	Certificate Authority Authorization	Controls which CAs can issue certs for your domain
NS	Nameserver delegation	Points to authoritative nameservers for a zone
PTR	Reverse DNS	Maps IP to hostname (used in email reputation)
SOA	Start of Authority	Controls zone serial, TTL defaults, negative caching

A and AAAA

Use both for dual-stack. Clients follow RFC 8305 (Happy Eyeballs v2) to prefer IPv6 but fall back to IPv4 quickly. If you only have A records, IPv6-only clients cannot reach you. If you only have AAAA records, you break a larger segment of the internet.

# Check both address families
dig A api.example.com
dig AAAA api.example.com

# Test IPv6 connectivity specifically
curl -6 https://api.example.com/health

CNAME Restrictions

A CNAME cannot exist at the zone apex (example.com itself, not www.example.com). If you point your root domain to a CNAME, it breaks the SOA and NS records that must exist at the apex. This is why www.example.com can be a CNAME but example.com cannot, and why ALIAS/ANAME records were invented.

CNAME chains add latency. a.example.com -> b.cdn.com -> c.edge.com requires three lookups. Keep chains short.

SRV Records for Service Discovery

SRV records encode service endpoints with port and priority information. They are the backbone of protocols like SIP, XMPP, and older Kubernetes service discovery.

; Format: _service._proto.name TTL class SRV priority weight port target
_grpc._tcp.api.example.com. 300 IN SRV 10 50 50051 grpc1.example.com.
_grpc._tcp.api.example.com. 300 IN SRV 10 50 50051 grpc2.example.com.
_grpc._tcp.api.example.com. 300 IN SRV 20 0  50051 grpc3.example.com.

Priority 10 entries are tried before priority 20. Weight distributes load within a priority tier. This gives you weighted load balancing with automatic fallback, purely in DNS.

dig SRV _grpc._tcp.api.example.com

TXT Records for Email Authentication

SPF, DKIM, and DMARC all live in TXT records and are not optional if you send transactional email. Getting them wrong means your email lands in spam or gets rejected outright.

; SPF: authorize your mail servers
example.com. TXT "v=spf1 include:_spf.google.com ip4:203.0.113.0/24 ~all"

; DMARC: policy for handling SPF/DKIM failures
_dmarc.example.com. TXT "v=DMARC1; p=reject; rua=mailto:dmarc@example.com"

# Check SPF
dig TXT example.com | grep spf

# Check DMARC
dig TXT _dmarc.example.com

DNS-Based Failover and Health Checks

Route 53 and Cloudflare both support health-check-aware DNS. The pattern: you create multiple records for the same name, associate each with a health check, and the DNS provider removes failing endpoints from responses.

A TypeScript health check endpoint that these services will probe:

import { createServer, IncomingMessage, ServerResponse } from "http";

interface HealthStatus {
  status: "ok" | "degraded" | "down";
  checks: Record<string, boolean>;
  timestamp: string;
}

async function checkDependencies(): Promise<Record<string, boolean>> {
  const checks: Record<string, boolean> = {};

  // Database connectivity
  try {
    await db.query("SELECT 1");
    checks.database = true;
  } catch {
    checks.database = false;
  }

  // Cache connectivity
  try {
    await redis.ping();
    checks.cache = true;
  } catch {
    checks.cache = false;
  }

  return checks;
}

const server = createServer(async (req: IncomingMessage, res: ServerResponse) => {
  if (req.url !== "/health") {
    res.writeHead(404);
    res.end();
    return;
  }

  const checks = await checkDependencies();
  const allHealthy = Object.values(checks).every(Boolean);
  const status: HealthStatus["status"] = allHealthy ? "ok" : "down";

  // Return 200 only when healthy. DNS health checkers use the HTTP status code.
  res.writeHead(allHealthy ? 200 : 503, { "Content-Type": "application/json" });
  res.end(JSON.stringify({ status, checks, timestamp: new Date().toISOString() }));
});

server.listen(3000);

The key: return a non-2xx status when the instance is unhealthy. Route 53 and Cloudflare health checks look at the HTTP status code, not the response body. A 200 with {"status": "down"} in the body still marks the endpoint healthy.

Failover DNS Configuration (Route 53 Terraform)

resource "aws_route53_health_check" "primary" {
  fqdn              = "us-east-1.api.example.com"
  port              = 443
  type              = "HTTPS"
  resource_path     = "/health"
  failure_threshold = 3
  request_interval  = 10

  tags = { Name = "primary-health-check" }
}

resource "aws_route53_record" "api_primary" {
  zone_id = var.zone_id
  name    = "api.example.com"
  type    = "A"

  failover_routing_policy {
    type = "PRIMARY"
  }

  set_identifier  = "primary"
  health_check_id = aws_route53_health_check.primary.id
  ttl             = 60

  records = ["203.0.113.10"]
}

resource "aws_route53_record" "api_secondary" {
  zone_id = var.zone_id
  name    = "api.example.com"
  type    = "A"

  failover_routing_policy {
    type = "SECONDARY"
  }

  set_identifier = "secondary"
  ttl            = 60

  records = ["203.0.113.20"]
}

With failure_threshold = 3 and request_interval = 10, failover triggers after 30 seconds of consecutive failures. Your TTL must be low enough (60 seconds here) that clients pick up the secondary record within an acceptable window. The total worst-case failover time is health check timeout plus DNS TTL plus client retry.

GeoDNS and Latency-Based Routing

GeoDNS answers based on where the query originates. Latency-based routing answers based on measured round-trip time from each region to your endpoints. They are not the same thing.

GeoDNS: A client in Germany gets eu.api.example.com. A client in Australia gets ap.api.example.com. The routing is purely geographic. If your EU region goes down, German clients do not automatically get routed to US-East.

Latency-based routing (Route 53’s term): Responses are weighted by measured latency from AWS’s resolver locations to your endpoints. It adapts to real network conditions. If your EU endpoint has high latency for some reason, clients that would normally hit EU might be routed to US-East instead.

For most multi-region deployments, combine both: primary routing is geographic (to keep data locality and reduce default latency), with failover policies that route traffic to healthy regions when primary regions fail.

# See what different resolvers return (simulates geographic routing)
dig @1.1.1.1 api.example.com       # Cloudflare resolver
dig @8.8.8.8 api.example.com       # Google resolver
dig @208.67.222.222 api.example.com # OpenDNS resolver

# Use a VPN or proxy to actually test GeoDNS from specific regions
# Or use DNS lookup tools like dnschecker.org for a rough view

CDN Integration Patterns

CDNs need to be in the DNS path to intercept traffic. There are two main patterns:

Full proxy (orange cloud in Cloudflare terms): Your domain’s A/AAAA records point to Cloudflare’s IP ranges. Cloudflare proxies all traffic, applies WAF rules, caches, and routes to your origin. Your origin’s real IP is hidden. This is the most common pattern for web traffic.

DNS only (grey cloud): The record resolves directly to your origin IP. No proxy, no caching, no WAF. Use this for services that cannot go through a proxy (some SMTP configurations, custom TCP protocols, etc.).

CNAME delegation for CDN: You create a CNAME from your subdomain to a CDN-provided hostname. The CDN manages its own IP ranges and routing behind that hostname. Common with CloudFront:

# CloudFront CNAME pattern
assets.example.com. CNAME d1234abcdef.cloudfront.net.

# Cloudflare SaaS custom hostname pattern (for multi-tenant platforms)
customer.theirs.com. CNAME custom.yourplatform.com.

For zone apex with CDN, you need ALIAS/ANAME (Route 53 calls it an “Alias record”). Cloudflare’s proxied records handle this transparently because Cloudflare returns its own anycast IPs rather than your origin’s IP.

DNS Security: DNSSEC and CAA

CAA Records

CAA (Certification Authority Authorization) records specify which CAs are allowed to issue certificates for your domain. Without CAA records, any CA can issue a cert for your domain. With them, issuance outside your specified CAs is blocked.

; Allow Let's Encrypt and DigiCert only
example.com. CAA 0 issue "letsencrypt.org"
example.com. CAA 0 issue "digicert.com"
example.com. CAA 0 issuewild "digicert.com"  ; wildcard certs: DigiCert only
example.com. CAA 0 iodef "mailto:security@example.com"  ; report violations

dig CAA example.com

Add CAA records. They are one of the cheapest security wins available at the DNS layer and are almost never set by default.

DNSSEC

DNSSEC adds cryptographic signatures to DNS responses, preventing cache poisoning attacks where an attacker poisons a resolver’s cache with fake records. The chain of trust runs from the root (.) down through TLD nameservers to your authoritative nameserver.

The tradeoff: DNSSEC adds operational complexity. Key rotation must be managed carefully. A misconfigured DS record or expired RRSIG causes complete domain resolution failure for validating resolvers. Many organizations weigh this risk and skip DNSSEC for internal tooling while enabling it for customer-facing domains.

Checking DNSSEC status:

# Check if DNSSEC is enabled and valid
dig +dnssec example.com

# Check DS records at the parent zone
dig DS example.com @a.gtld-servers.net

# Validate the full chain (requires a DNSSEC-validating resolver)
dig +sigchase example.com  # older bind versions
# Or use: https://dnsviz.net for visual chain validation

Common Debugging Scenarios

Propagation Delays

“DNS hasn’t propagated yet” is almost always a caching problem, not a propagation problem. Modern authoritative DNS changes take seconds to replicate. What takes time is old TTLs expiring across recursive resolvers worldwide.

# Check what your authoritative nameserver currently says
dig @ns1.example.com api.example.com

# Check what your ISP's resolver has cached
dig api.example.com

# Check a remote resolver (different cache)
dig @1.1.1.1 api.example.com
dig @8.8.8.8 api.example.com

# See remaining TTL on a cached record
dig +ttlid api.example.com

If the authoritative nameserver returns the new value but resolvers still return the old value, you are waiting on TTL expiry. If the authoritative nameserver returns the old value, your DNS provider has not updated its records yet.

CNAME Chain Problems

Long CNAME chains increase resolution latency because each hop requires a separate lookup. Debugging them:

# Follow a CNAME chain step by step
dig CNAME www.example.com
dig CNAME target-from-above.cdn.com
# Continue until you hit an A record

# See the full chain in one query
dig +trace www.example.com

If your CNAME chain is longer than three hops, consider flattening it. Some DNS providers (Cloudflare) do CNAME flattening automatically, returning the final A record directly and eliminating intermediate lookups.

TTL Gotchas in Deployments

A real production scenario: you are migrating from one load balancer to another. You update the A record. Users still hit the old IP. Why?

1. Your TTL was 3600. You need to wait up to 3600 seconds for every resolver’s cache to expire.
2. Some ISP resolvers ignore TTL entirely (rare but documented).
3. Browsers and operating systems have their own DNS caches. Chrome’s DNS cache TTL ignores the record TTL and uses its own heuristics.

Mitigation: pre-flight TTL reduction. Two to three days before any planned IP migration, lower the TTL on the record to 60 seconds. Run the migration. Both IPs need to work for the full old-TTL duration. Raise TTL back to a normal value after the migration stabilizes.

Debugging with nslookup

# Interactive mode: useful for querying multiple records
nslookup
> server 8.8.8.8          # set resolver
> set type=MX             # query MX records
> example.com
> set type=TXT
> _dmarc.example.com
> exit

# Non-interactive
nslookup -type=SRV _grpc._tcp.api.example.com 8.8.8.8

dig is more reliable and scriptable than nslookup. Use nslookup when dig is not available (Windows environments by default).

DNS Provider Comparison

Feature	Route 53	Cloudflare	NS1
Latency-based routing	Yes	No (uses geo)	Yes
GeoDNS	Yes	Yes	Yes
Health check failover	Yes	Yes (Pro+)	Yes
Anycast network	Limited	Yes, global	Yes
DNSSEC	Yes	Yes	Yes
ALIAS/ANAME at apex	Yes (Alias)	Yes (flattening)	Yes
Traffic policies / filter chains	Yes	Yes (Workers)	Yes (filter chain)
API / IaC support	Terraform, CDK	Terraform, API	Terraform, API
Free tier	No	Yes (1 zone free)	No
Best for	AWS-native stacks	CDN + DNS together	Complex routing

The choice between Route 53 and Cloudflare is often determined by where your infrastructure lives. If you are already in AWS and use CloudFront, Route 53 integrates tightly and alias records to CloudFront distributions work without extra configuration. If you want edge caching and DNS together with minimal setup, Cloudflare is hard to beat on simplicity.

Production Checklist

Before you consider your DNS configuration production-ready:

• TTL on critical records is low enough to allow fast incident response (60-300 seconds)
• Both A and AAAA records exist for public-facing services
• CAA records are set and restrict certificate issuance to your CAs
• SPF, DKIM, and DMARC TXT records exist for all sending domains
• Health checks are configured for any failover or latency-based routing policy
• You have verified the authoritative nameserver returns correct records independently of any caching resolver
• You have a documented TTL reduction procedure before any planned IP migration
• DNSSEC is evaluated and either enabled with a tested rotation procedure or explicitly deferred with documented reasoning

DNS failures are quiet. A missing AAAA record does not throw an exception. A high TTL on a record you need to change during an incident does not warn you in advance. The cost of setting it up correctly once is far lower than debugging it under pressure at 2am.