Production Kubernetes Troubleshooting: Debugging CrashLoopBackOff, OOMKilled, and Networking Failures

Most Kubernetes production incidents fall into three categories: the application crashes, the application runs out of resources, or the application cannot reach something over the network. That mental model matters because it narrows your investigation immediately. You do not need to understand every corner of the Kubernetes API to fix a broken pod. You need to know which category you are in, which kubectl commands give you the right signal, and what the output actually means.

This guide walks through the three failure modes that account for the majority of on-call pages: CrashLoopBackOff, OOMKilled, and networking failures. Each section covers what is happening at the system level, the commands that surface the root cause, and the fixes that hold up in production.

CrashLoopBackOff: Reading the Signals

CrashLoopBackOff is not an error. It is a status. It means the kubelet started your container, the container exited, and the kubelet is waiting with exponential backoff before trying again. The backoff starts at 10 seconds, doubles each time, and caps at 5 minutes. The actual problem is whatever caused the container to exit.

Start here:

kubectl describe pod <pod-name> -n <namespace>

Look at the Last State section. The Reason field tells you whether the container was OOMKilled, Error, or Completed. The Exit Code field narrows things further:

• Exit code 1: Generic application error. Your code threw an unhandled exception or returned a non-zero exit.
• Exit code 137: The process received SIGKILL. This is almost always the OOM killer or a liveness probe failure triggering a restart.
• Exit code 139: Segmentation fault. Common with native bindings or misconfigured memory in JVM/Go applications.
• Exit code 143: The process received SIGTERM and did not handle it gracefully, or the terminationGracePeriodSeconds expired and the process was killed.
• Exit code 126/127: Permission denied or command not found. Usually a bad entrypoint or a missing binary in the container image.

Next, pull the logs from the previous crashed container:

kubectl logs <pod-name> -n <namespace> --previous

The --previous flag is critical. Without it, you get the logs from the current attempt, which may be a fresh container that has not failed yet. If the pod has multiple containers, specify which one with -c <container-name>.

Check init containers separately. A pod stuck in Init:CrashLoopBackOff means one of your init containers is failing. Init containers run sequentially before the main containers start. A common cause is an init container that waits for a database or service that is not available yet.

kubectl logs <pod-name> -n <namespace> -c <init-container-name>

Finally, check the events on the pod and the namespace:

kubectl get events -n <namespace> --sort-by='.lastTimestamp' | tail -20

Events tell you about image pull failures, failed scheduling, volume mount problems, and liveness/readiness probe failures. If you see Back-off restarting failed container alongside Liveness probe failed, your probe is killing the container before it finishes starting up. Increase initialDelaySeconds or switch to a startup probe.

Common CrashLoopBackOff Causes

1. Missing environment variables or secrets: The application tries to read a config value that does not exist and crashes on startup.
2. Database migrations that fail: An init container runs migrations against a locked or unreachable database.
3. Liveness probe misconfiguration: The probe path returns 404, or the timeout is too aggressive for a cold start.
4. Image tag mismatch: The latest tag pulled a broken image. This is why you pin image digests in production.

OOMKilled: Understanding the Memory Model

When a container exceeds its memory limit, the kernel’s OOM killer terminates the process. Kubernetes reports this as OOMKilled in the pod status. The container restarts, potentially hits the same limit, and you get a CrashLoopBackOff driven by memory exhaustion.

The distinction between requests and limits matters here. Requests are what the scheduler uses to place pods on nodes. Limits are what the kernel enforces. If your container requests 256Mi and limits at 512Mi, the scheduler reserves 256Mi on the node, but the container can use up to 512Mi before the OOM killer intervenes.

Check the current state:

kubectl describe pod <pod-name> -n <namespace> | grep -A 5 "Limits\|Requests\|Last State"

Then look at actual usage:

kubectl top pod <pod-name> -n <namespace>

kubectl top requires the metrics-server to be running in your cluster. If it is not installed, you will get an error. You can also check node-level pressure:

kubectl describe node <node-name> | grep -A 10 "Allocated resources"

This shows you how much of the node’s capacity is reserved by requests. If requests are overcommitted, pods compete for memory under pressure, and the OOM killer picks off the ones exceeding their requests first.

Check if resource quotas are constraining your namespace:

kubectl describe resourcequota -n <namespace>

Right-Sizing Memory

Setting memory limits is a tradeoff. Too low and your application gets killed during traffic spikes. Too high and you waste cluster capacity, or worse, you mask a memory leak that will eventually cause a node-level OOM event.

The Vertical Pod Autoscaler (VPA) in recommendation mode is useful here. It observes actual memory consumption over time and suggests limits:

kubectl get vpa <vpa-name> -n <namespace> -o jsonpath='{.status.recommendation.containerRecommendations[*]}'

If VPA is not an option, profile your application under realistic load. For JVM applications, the heap is only part of the picture. Metaspace, thread stacks, native memory, and off-heap buffers all contribute. A Java container with -Xmx256m can easily consume 400-500Mi total. Set your container limit to at least 1.5x the heap size and adjust from there.

For Go applications, the runtime returns memory to the OS lazily. A spike in allocations can push RSS high, and it may not come back down for minutes. Watch the container_memory_working_set_bytes metric over hours, not minutes.

A practical approach: set requests equal to the p95 memory usage during normal operation, and set limits to 1.3-1.5x that value. This gives headroom for spikes without overcommitting the node.

Networking Failures: Service Discovery and Beyond

Networking problems in Kubernetes range from straightforward DNS failures to subtle issues with network policies or misconfigured services. The symptoms are varied: connection timeouts, connection refused, name resolution failures, or intermittent packet loss.

DNS Resolution Failures

Every pod gets its DNS configuration from CoreDNS (or kube-dns in older clusters). When DNS breaks, everything breaks. Services cannot resolve each other, external API calls fail, and you get cascading timeouts.

Test DNS from inside a pod:

kubectl exec -it <pod-name> -n <namespace> -- nslookup <service-name>.<namespace>.svc.cluster.local

If that fails, check if CoreDNS is running:

kubectl get pods -n kube-system -l k8s-app=kube-dns

Look at CoreDNS logs for errors:

kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50

Common DNS issues include:

• CoreDNS pods in CrashLoopBackOff: Usually caused by a loop in DNS resolution. CoreDNS tries to resolve upstream, hits a node’s /etc/resolv.conf that points back to the cluster IP, and loops. The loop plugin in the Corefile detects this and crashes intentionally.
• DNS throttling under load: CoreDNS has a finite capacity. If your cluster has thousands of pods making frequent DNS queries, you may need to scale CoreDNS horizontally or enable NodeLocal DNSCache.
• ndots configuration: By default, Kubernetes sets ndots:5 in pod resolv.conf. This means a lookup for api.example.com first tries api.example.com.<namespace>.svc.cluster.local, then several other suffixes, before finally trying the bare name. This multiplies DNS queries. For pods that call many external services, set dnsConfig.options with ndots:2 in the pod spec.

Service Connectivity

If DNS resolves but connections fail, check the service and its endpoints:

kubectl get endpoints <service-name> -n <namespace>

An empty endpoints list means the service selector does not match any running pods, or the pods are not passing their readiness probes. Compare the service selector with the pod labels:

kubectl get svc <service-name> -n <namespace> -o wide
kubectl get pods -n <namespace> --show-labels

Test connectivity directly:

kubectl exec -it <debug-pod> -n <namespace> -- curl -v http://<service-name>:<port>/health

If you do not have a debug pod, create a temporary one:

kubectl run debug --image=busybox --rm -it --restart=Never -- sh

Network Policies

If services resolve and endpoints exist but connections still fail, network policies may be blocking traffic. Network policies are namespace-scoped and default-deny when any policy selects a pod.

List policies in the namespace:

kubectl get networkpolicies -n <namespace>

Inspect a specific policy:

kubectl describe networkpolicy <policy-name> -n <namespace>

A common mistake is applying an ingress policy that allows traffic from one namespace but forgetting to label the source namespace. NetworkPolicy namespaceSelector matches on namespace labels, not namespace names.

Quick-Reference Diagnostic Table

Symptom	First Command	Likely Cause	Fix
CrashLoopBackOff, exit code 1	kubectl logs <pod> --previous	Application error on startup	Fix app config, check env vars and secrets
CrashLoopBackOff, exit code 137	kubectl describe pod <pod>	OOMKilled or liveness probe kill	Increase memory limit or fix probe timing
CrashLoopBackOff, exit code 127	kubectl describe pod <pod>	Missing binary in image	Fix Dockerfile entrypoint or image tag
OOMKilled repeated	kubectl top pod <pod>	Memory limit too low or memory leak	Right-size limits, profile memory usage
Pod stuck in Pending	kubectl describe pod <pod>	Insufficient node resources	Scale nodes, reduce requests, check quotas
Connection refused to service	kubectl get endpoints <svc>	No ready endpoints	Check readiness probes, pod labels, selector
DNS resolution failure	kubectl exec <pod> -- nslookup <svc>	CoreDNS down or misconfigured	Check CoreDNS pods, review Corefile
Intermittent timeouts	kubectl get networkpolicies	Network policy blocking traffic	Audit policies, check namespace labels

Production Monitoring and Alerting

Debugging is reactive. For production clusters, set up alerts that catch these problems before they page you at 3 AM.

Alert on kube_pod_container_status_restarts_total increasing. A pod that restarts once is a blip. A pod that restarts five times in an hour is a problem. Use a rate-based alert rather than a threshold on the absolute counter, since the counter resets when pods are recreated.

Alert on container_memory_working_set_bytes approaching the limit. A container consistently running at 90% of its memory limit will eventually get OOM killed during a traffic spike. Give yourself time to react.

For networking, monitor CoreDNS latency and error rates. The coredns_dns_requests_total and coredns_dns_responses_total metrics, broken down by response code, tell you if DNS is degrading before applications start failing.

Build runbooks that map alert names to the diagnostic steps in this guide. When the PodCrashLooping alert fires, the on-call engineer should not need to remember which kubectl commands to run. The runbook should link directly to the relevant section and include cluster-specific context like namespace conventions and common failure modes for your applications.

Track pod restart counts in your dashboards alongside deployment events. Many CrashLoopBackOff incidents correlate with recent deployments. If you can see that a restart spike started two minutes after a deploy, you have your root cause and can roll back quickly.

Closing

The pattern across all three failure modes is the same: read the exit signal, get the logs from the right container at the right point in time, and verify assumptions about connectivity with actual probe commands rather than inference from application behavior. Most Kubernetes production incidents that take hours to resolve were solvable in minutes once the operator stopped guessing and started reading what the control plane was already reporting.