Kubernetes Networking Explained: Services, Ingress, Network Policies, and DNS Resolution

Kubernetes networking trips people up because the abstractions compound. You add a Service on top of a pod network, an Ingress on top of a Service, and a NetworkPolicy on top of all of it. When something breaks, you have four layers to interrogate and most guides only explain one of them at a time.

This article covers all of it: the pod network model, every Service type, Ingress and IngressClass, NetworkPolicy for segmentation, CoreDNS for service discovery, and the debugging workflow you use when packets stop arriving.

The Pod Network Model

Every pod in a Kubernetes cluster gets its own IP address. Not every pod on a node shares a node IP through NAT. Every pod has a routable IP. This is the foundational assumption everything else builds on.

The consequence is that pods can communicate directly with each other across nodes without any NAT translation. A pod on node A with IP 10.244.1.5 can open a TCP connection to a pod on node B with IP 10.244.2.12 and the source address seen by the destination is the real pod IP, not the node IP.

Kubernetes does not implement this itself. It delegates to a Container Network Interface (CNI) plugin. Flannel uses VXLAN overlays. Calico uses BGP to distribute routes. Cilium uses eBPF to intercept and redirect packets in the kernel. The choice of CNI affects performance, NetworkPolicy support, and observability, but the flat IP model is the guarantee every CNI must satisfy.

Pods are ephemeral. Their IPs change on restart. You never hardcode a pod IP in application code. This is exactly why Services exist.

Service Types

A Service gives you a stable virtual IP (the ClusterIP) that load balances across a set of pods. The pods in scope are selected by label.

ClusterIP

ClusterIP is the default. It creates a virtual IP reachable only from inside the cluster. kube-proxy programs iptables (or IPVS) rules on every node so that traffic to the ClusterIP gets DNAT’d to one of the backing pod IPs.

apiVersion: v1
kind: Service
metadata:
  name: payments-api
  namespace: billing
spec:
  selector:
    app: payments-api
  ports:
    - name: http
      port: 80
      targetPort: 3000
  type: ClusterIP

The selector app: payments-api matches any pod with that label. Add a pod with the label and it immediately receives traffic. Remove the label or delete the pod and it stops receiving traffic within seconds, as the Endpoints controller reconciles.

ClusterIP is the right type for service-to-service communication inside the cluster. It is not reachable from outside the cluster.

NodePort

NodePort opens a port (30000-32767 by default) on every node in the cluster and forwards traffic from that port to the Service’s backing pods.

apiVersion: v1
kind: Service
metadata:
  name: payments-api
  namespace: billing
spec:
  selector:
    app: payments-api
  ports:
    - name: http
      port: 80
      targetPort: 3000
      nodePort: 31080
  type: NodePort

Traffic arriving at <any-node-ip>:31080 gets forwarded to the payments-api pods regardless of which node the traffic lands on. The forwarding is handled by kube-proxy on each node.

NodePort is rarely the right long-term solution. You expose node IPs to external traffic, which is a security surface. You also depend on knowing node IPs, which change when nodes are replaced. It is useful for local development with tools like minikube or kind, and sometimes for debugging production issues when you need to bypass an Ingress.

LoadBalancer

LoadBalancer extends NodePort by additionally provisioning an external load balancer through the cloud provider. On AWS this creates an ELB or NLB. On GCP it creates a Cloud Load Balancer. The load balancer gets a public IP and forwards traffic to the NodePort on the cluster nodes.

apiVersion: v1
kind: Service
metadata:
  name: payments-api
  namespace: billing
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
    service.beta.kubernetes.io/aws-load-balancer-internal: "false"
spec:
  selector:
    app: payments-api
  ports:
    - name: https
      port: 443
      targetPort: 3000
  type: LoadBalancer

The cloud provider provisions the load balancer asynchronously and populates status.loadBalancer.ingress with the external IP or hostname once it is ready.

The cost: one LoadBalancer Service typically maps to one cloud load balancer. At $15-20 per month per NLB on AWS, this adds up if you have many services that need external exposure. For HTTP traffic, Ingress is almost always the better answer because one load balancer can route to many services.

ExternalName

ExternalName maps a Service name to a DNS name outside the cluster. No proxying happens. The cluster DNS returns a CNAME pointing to the external name.

apiVersion: v1
kind: Service
metadata:
  name: legacy-payments
  namespace: billing
spec:
  type: ExternalName
  externalName: payments.legacy.internal.example.com

Pods in the cluster that resolve legacy-payments.billing.svc.cluster.local get back the CNAME for payments.legacy.internal.example.com. The actual connection goes directly to that address. This is useful during migrations when you want to route traffic to an external system using the same Service name your application already uses.

Ingress Controllers and IngressClass

An Ingress resource is a set of routing rules for HTTP and HTTPS traffic. It specifies which hostnames and paths should route to which Services. The Ingress resource itself is inert: you need an Ingress controller to read those rules and configure an actual reverse proxy.

Common controllers: nginx-ingress (the NGINX one from the community), ingress-nginx (the one maintained by the Kubernetes project itself, confusingly similar name), the AWS Load Balancer Controller, the GKE Ingress controller, Traefik, and Contour. They differ significantly in feature set and operational model.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: payments-ingress
  namespace: billing
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - payments.example.com
      secretName: payments-tls
  rules:
    - host: payments.example.com
      http:
        paths:
          - path: /api/v1
            pathType: Prefix
            backend:
              service:
                name: payments-api
                port:
                  number: 80
          - path: /webhooks
            pathType: Prefix
            backend:
              service:
                name: webhook-processor
                port:
                  number: 80

The ingressClassName field selects which controller handles this Ingress. This replaced the older kubernetes.io/ingress.class annotation in Kubernetes 1.18+. If you have multiple Ingress controllers in the cluster (nginx for internal traffic, the AWS LBC for public traffic), ingressClassName determines which one picks up each Ingress resource.

The tls block references a Secret of type kubernetes.io/tls. You can provision these with cert-manager, which integrates with Let’s Encrypt and cloud certificate managers to handle rotation automatically.

Path matching behavior varies by controller. With nginx-ingress, Prefix matching is exact prefix on the path component. Exact requires an exact match. ImplementationSpecific defers to the controller’s own rules. The path type matters when you have overlapping prefixes.

Network Policies

By default, every pod can talk to every other pod in the cluster across any namespace. Network Policies change that. A NetworkPolicy selects a set of pods and defines what ingress (inbound) and egress (outbound) traffic is allowed.

NetworkPolicy is enforced by the CNI plugin, not by kube-proxy or the API server. If your CNI does not support NetworkPolicy (Flannel without Canal, for instance), the resources will be accepted by the API server but silently ignored. Calico, Cilium, and Weave all support it.

A policy that allows the payments-api to receive traffic only from the API gateway and the billing worker, on port 3000:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: payments-api-ingress
  namespace: billing
spec:
  podSelector:
    matchLabels:
      app: payments-api
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: api-gateway
        - podSelector:
            matchLabels:
              app: billing-worker
      ports:
        - protocol: TCP
          port: 3000

Once any NetworkPolicy selects a pod, all traffic not explicitly allowed is denied. If you add a policy that only allows ingress, egress from that pod is still unrestricted unless you also add an egress policy with policyTypes: [Egress].

A common pattern is a default-deny policy for a namespace, then explicit allow policies per service:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: billing
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress

The empty podSelector matches all pods in the namespace. This denies all ingress and egress for every pod. Then you add explicit allow policies layered on top.

Cross-namespace traffic requires a namespaceSelector. Allow the monitoring namespace to scrape metrics from pods in the billing namespace:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-prometheus-scrape
  namespace: billing
spec:
  podSelector:
    matchLabels:
      app: payments-api
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
          podSelector:
            matchLabels:
              app: prometheus
      ports:
        - protocol: TCP
          port: 9090

Note the indentation: namespaceSelector and podSelector under the same from list item means “pods matching this podSelector AND in namespaces matching this namespaceSelector”. Two separate list items would mean “pods matching this selector OR pods in namespaces matching that selector”. The difference is significant for security.

CoreDNS and Service Discovery

CoreDNS is the cluster DNS server. Every pod’s /etc/resolv.conf points to the CoreDNS ClusterIP (typically 10.96.0.10) with a search domain list like billing.svc.cluster.local svc.cluster.local cluster.local.

A Service named payments-api in the billing namespace is reachable at:

payments-api.billing.svc.cluster.local

Because of the search domains, pods in the same namespace can use just payments-api. Pods in other namespaces need at least payments-api.billing.

The DNS record types:

• ClusterIP Services get an A record pointing to the virtual IP
• Headless Services (ClusterIP: None) get A records pointing to individual pod IPs, one per ready pod
• ExternalName Services get a CNAME record
• Services with named ports get SRV records: _http._tcp.payments-api.billing.svc.cluster.local

Headless Services are useful when your application manages its own load balancing or needs to discover all pod IPs directly. StatefulSets use them to give each pod a stable DNS name: payments-worker-0.payments-worker.billing.svc.cluster.local.

Service Discovery in Application Code

In TypeScript, a service client that resolves via cluster DNS and respects Kubernetes-style health:

import { createConnection } from "net";

interface ServiceEndpoint {
  host: string;
  port: number;
}

function resolveService(name: string, namespace: string): ServiceEndpoint {
  // In-cluster: use short name for same-namespace, full name across namespaces
  const host = `${name}.${namespace}.svc.cluster.local`;
  return { host, port: 80 };
}

async function fetchWithRetry(
  endpoint: ServiceEndpoint,
  path: string,
  maxRetries = 3
): Promise<Response> {
  const url = `http://${endpoint.host}:${endpoint.port}${path}`;

  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      const response = await fetch(url, {
        signal: AbortSignal.timeout(5000),
      });

      if (response.ok || response.status < 500) {
        return response;
      }

      // 5xx: retry
      if (attempt === maxRetries - 1) {
        throw new Error(`Service ${endpoint.host} returned ${response.status}`);
      }
    } catch (err) {
      if (attempt === maxRetries - 1) throw err;
      // Exponential backoff: 100ms, 200ms, 400ms
      await new Promise((r) => setTimeout(r, 100 * Math.pow(2, attempt)));
    }
  }

  throw new Error("unreachable");
}

// Usage
const paymentsEndpoint = resolveService("payments-api", "billing");
const result = await fetchWithRetry(paymentsEndpoint, "/api/v1/charge");

For services that use environment variable injection (the older Kubernetes mechanism before DNS was reliable), Kubernetes injects PAYMENTS_API_SERVICE_HOST and PAYMENTS_API_SERVICE_PORT automatically for any Service that exists when the pod starts. DNS is preferred: it handles Services created after the pod starts, and the variable names are not guaranteed to remain stable as naming conventions evolve.

Tradeoffs

Dimension	ClusterIP	NodePort	LoadBalancer	Ingress
External access	No	Yes (node IP)	Yes (dedicated IP)	Yes (shared IP)
Cost	Free	Free	One LB per Service	One LB for many Services
Protocol support	Any	Any	Any	HTTP/HTTPS only
TLS termination	Application	Application	LB or application	Controller or application
Path-based routing	No	No	No	Yes
Operational complexity	Low	Low	Medium	Medium to high

Debugging Techniques

Verify Endpoints

The first thing to check when a Service does not route traffic is whether it has any Endpoints. A Service with no Endpoints means either no pods match the selector, or no matching pods pass their readiness probe.

kubectl get endpoints payments-api -n billing
# NAME           ENDPOINTS            AGE
# payments-api   10.244.1.5:3000     5m

kubectl describe service payments-api -n billing
# Check Selector and Endpoints sections

If Endpoints is empty, check pod labels match the Service selector exactly and that pods are Ready.

DNS Resolution

From inside the cluster, use a debug pod to test DNS:

kubectl run debug --image=nicolaka/netshoot --rm -it --restart=Never -- \
  nslookup payments-api.billing.svc.cluster.local

If the name resolves but the Service is unreachable, the problem is in routing or NetworkPolicy, not DNS. If the name does not resolve, CoreDNS may be down or misconfigured. Check CoreDNS pod status in the kube-system namespace and look at its logs.

Connectivity Testing

Test TCP connectivity directly from a pod inside the cluster:

kubectl exec -it <pod-name> -n billing -- \
  curl -v http://payments-api:80/healthz

# Or test raw TCP
kubectl exec -it <pod-name> -n billing -- \
  nc -zv payments-api 80

For diagnosing NetworkPolicy issues, packet capture is the most reliable tool. Cilium and Calico have their own policy tracing commands. For raw packet inspection, tcpdump from inside the target pod’s namespace (using a privileged debug container):

kubectl debug -it <pod-name> -n billing \
  --image=nicolaka/netshoot \
  --target=payments-api \
  -- tcpdump -i eth0 port 3000

Then from another terminal, send a request to the Service and watch whether packets arrive at the pod. If the Service has Endpoints but packets never arrive at the pod, NetworkPolicy is blocking them. If packets arrive but the pod is not responding, the problem is in the application.

Ingress Debugging

Ingress problems usually fall into three categories: the controller cannot find the backend Service, TLS certificate issues, or path matching that does not work as expected.

# Check Ingress status
kubectl describe ingress payments-ingress -n billing

# Check controller logs (nginx-ingress example)
kubectl logs -n ingress-nginx \
  deployment/ingress-nginx-controller \
  --tail=100 | grep payments

# Verify the TLS secret exists and has the right keys
kubectl get secret payments-tls -n billing -o jsonpath='{.data}' | \
  python3 -c "import sys,json,base64; d=json.load(sys.stdin); print(list(d.keys()))"

For path matching issues, test directly against the controller’s pod IP bypassing DNS to isolate whether the problem is in routing rules or upstream DNS:

kubectl get pods -n ingress-nginx -o wide
# Note the pod IP

kubectl run debug --image=nicolaka/netshoot --rm -it --restart=Never -- \
  curl -H "Host: payments.example.com" http://<controller-pod-ip>/api/v1/test

Production Considerations

Readiness gates over readiness probes for zero-downtime deployments. Standard readiness probes only check whether the application is alive. Pod Readiness Gates let you gate readiness on external conditions, like whether the pod’s Endpoints entry has propagated to all nodes. Without this, traffic can arrive at a pod before kube-proxy has updated its iptables rules.

NetworkPolicy in CI, not just production. NetworkPolicy violations fail silently at the application layer: the pod just sees a connection timeout. Testing NetworkPolicy rules in a CI environment (using kind with Calico) catches misconfigurations before they cause production incidents.

CoreDNS cache TTL tuning. The default cache TTL in CoreDNS is 30 seconds. For rapidly changing endpoints, this can cause stale resolution. For most workloads it is fine. If you see DNS-related flakiness during deployments, check whether your application is caching DNS results longer than the cluster TTL.

IPVS mode for kube-proxy at scale. The default iptables mode creates one rule per Service-to-pod mapping. At 10,000 Services this becomes a performance problem: iptables rule traversal is O(n). IPVS mode uses kernel hash tables and is O(1). Enable it in the kube-proxy ConfigMap under mode: ipvs.

Egress NetworkPolicy for DNS. If you apply a default-deny-egress policy, pods cannot resolve DNS. You must explicitly allow egress to port 53 on the CoreDNS ClusterIP. This is the most common mistake when first applying network segmentation.

# Allow DNS egress for all pods in the namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-egress
  namespace: billing
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

The Kubernetes networking model is simple at its core: every pod gets an IP, all pod IPs are routable without NAT. Every abstraction above that, Services, Ingress, NetworkPolicy, DNS, is layered deliberately on top of that guarantee. When traffic breaks, you trace it layer by layer from pod IP to Endpoints to Service to Ingress controller to DNS, and the problem is almost always a missing label, a missing policy exception, or a controller that was never configured to read the resource.