How Kubernetes Works Internally: The Control Loop, Scheduler, and Pod Lifecycle

Most teams adopt Kubernetes before they understand it. You write a Deployment manifest, run kubectl apply, and a pod appears somewhere in the cluster. It feels like magic, and for a while that’s fine. Then you hit a scheduling failure with a cryptic message, a pod gets stuck in Terminating for six minutes, or a service stops routing traffic during a node drain. At that point, “it’s declarative” stops being a useful mental model.

This article walks through what actually happens inside Kubernetes when you apply a manifest: how the API server handles the request, how etcd becomes the source of truth, how the scheduler filters and ranks nodes, how the kubelet drives the pod lifecycle through CRI and CNI, how kube-proxy translates Service objects into kernel-level packet routing, and how the entire system stays eventually consistent through the informer/watch pattern. The admission controller layer gets coverage too, because it’s where most policy and mutation logic silently runs before any of the above kicks in.

The API Server and etcd: Declarative State at Rest

The Kubernetes API server (kube-apiserver) is not a dispatcher. It does not tell nodes to do things. It is a validated, authenticated, versioned HTTP gateway to a single source of truth: etcd.

When you run kubectl apply -f deployment.yaml, the flow is:

1. kubectl serializes the manifest, computes a three-way diff against the live object (current state, last-applied annotation, new desired state), and sends a PATCH or PUT to the API server.
2. The API server authenticates the request (certificate, token, or webhook), runs authorization checks (RBAC), and then passes the object through admission controllers (more on that later).
3. If the object passes admission, the API server validates it against the OpenAPI schema for that resource version.
4. The object is written to etcd with a resource version (a monotonically increasing integer) as an optimistic concurrency token.
5. The API server returns 200 or 201 to the client. The pod does not exist yet on any node.

etcd stores all cluster state under a /registry/ prefix, organized by resource type and namespace. A Deployment lives at something like /registry/deployments/production/my-app. Every write to etcd triggers a watch event that propagates to any component watching that resource type.

The critical design choice here is that no controller, scheduler, or kubelet ever reads directly from etcd in a tight loop. They all watch the API server, which proxies etcd watch streams. This keeps etcd read pressure low and lets the API server apply rate limiting and caching through its watch cache (an in-memory store of recent resource versions used to serve list-watch requests without hitting etcd on every call).

The Informer and Watch Pattern: Event-Driven at Every Layer

Every Kubernetes controller (Deployment controller, ReplicaSet controller, the scheduler, the kubelet) uses the same client-go pattern: an Informer.

An Informer does three things:

1. Issues an initial LIST to get all existing objects of a type and populates a local in-memory store (the indexer).
2. Opens a long-lived HTTP/2 streaming WATCH connection to the API server, starting from the resource version returned by the LIST.
3. Feeds incoming watch events (ADDED, MODIFIED, DELETED) into a work queue that controllers drain.

// Conceptual TypeScript model of the Informer pattern
// (actual client-go is Go, but the structure maps cleanly)

type WatchEvent<T> =
  | { type: "ADDED"; object: T }
  | { type: "MODIFIED"; object: T }
  | { type: "DELETED"; object: T };

class Informer<T extends { metadata: { resourceVersion: string } }> {
  private store = new Map<string, T>();
  private workQueue: T[] = [];

  async start(listFn: () => Promise<T[]>, watchFn: (rv: string) => AsyncIterable<WatchEvent<T>>) {
    // Phase 1: initial list, populate local cache
    const items = await listFn();
    let lastRV = "";
    for (const item of items) {
      this.store.set(item.metadata.resourceVersion, item);
      lastRV = item.metadata.resourceVersion;
    }

    // Phase 2: stream events from last known resourceVersion
    for await (const event of watchFn(lastRV)) {
      if (event.type === "DELETED") {
        this.store.delete(event.object.metadata.resourceVersion);
      } else {
        this.store.set(event.object.metadata.resourceVersion, event.object);
        this.workQueue.push(event.object); // enqueue for reconciliation
      }
    }
  }

  dequeue(): T | undefined {
    return this.workQueue.shift();
  }
}

If the watch connection drops (network blip, API server restart), the informer re-lists and re-watches from the last seen resource version. If that version is no longer in etcd’s compacted history, it falls back to a full relist. This makes controllers resilient to transient failures without any explicit retry logic in the controller itself.

The work queue between the informer and the controller loop is a rate-limited queue that deduplicates by key. If a pod gets five rapid MODIFIED events while the controller is busy, the queue collapses them into a single reconcile call. This is why Kubernetes handles thundering herds reasonably well.

The Reconciliation Loop: Controllers as Reality Correctors

Every high-level object in Kubernetes is managed by a controller running the same basic loop:

desired state (from API server) - actual state (observed) = delta to fix

When you create a Deployment with replicas: 3, the Deployment controller creates a ReplicaSet. The ReplicaSet controller notices the ReplicaSet has 0 pods and 3 desired, so it creates 3 Pod objects (API server writes, not actual containers yet). The scheduler notices 3 unscheduled pods and binds them to nodes. The kubelet on each node notices a pod bound to it and starts pulling images and running containers.

No component tells any other component what to do. Each watches the API server and acts on the difference between what it sees and what it wants to see. This is the control loop in practice. The system is eventually consistent: if you kill the scheduler process mid-assignment, it comes back, re-lists unbound pods, and schedules them. Nothing is lost because the desired state is durable in etcd.

The Scheduler: Filtering and Scoring

The scheduler (kube-scheduler) watches for pods with spec.nodeName unset and spec.schedulerName matching its name. For each unscheduled pod, it runs a two-phase pipeline.

Phase 1: Filtering (Predicates)

The scheduler evaluates every node against a set of filter plugins. A node is eliminated from consideration if any filter returns false. Common built-in filters:

• NodeResourcesFit: does the node have enough CPU and memory headroom for the pod’s requests?
• NodeAffinity: does the node satisfy the pod’s nodeAffinity rules?
• TaintToleration: does the pod tolerate all taints on the node?
• PodTopologySpread: would scheduling here violate max skew constraints?
• VolumeBinding: can all required PersistentVolumeClaims be bound on this node?

After filtering, the scheduler has a candidate set. If the set is empty, the pod stays Pending and the scheduler logs a FailedScheduling event explaining which filter eliminated every node. This is almost always the first thing to check when a pod is stuck pending.

Phase 2: Scoring (Priorities)

Each candidate node is scored by a set of scoring plugins. Scores are weighted and summed. Common built-in scorers:

• LeastAllocated: prefer nodes with more remaining CPU and memory (spreads load).
• NodeAffinityPriority: give higher scores to nodes matching preferred affinity terms.
• InterPodAffinityPriority: prefer nodes that satisfy pod affinity preferences.
• ImageLocality: prefer nodes that already have the container image pulled (avoids pull latency for large images).

The node with the highest total score wins. The scheduler then writes a Binding object to the API server, which sets spec.nodeName on the pod. That write is the handoff point: now the kubelet on that node is responsible.

// Conceptual scoring pipeline
interface ScoringPlugin {
  name: string;
  weight: number;
  score(node: NodeInfo, pod: PodSpec): number; // 0-100
}

function selectNode(candidates: NodeInfo[], pod: PodSpec, plugins: ScoringPlugin[]): NodeInfo {
  let best: NodeInfo | null = null;
  let bestScore = -Infinity;

  for (const node of candidates) {
    const total = plugins.reduce((sum, plugin) => {
      return sum + plugin.score(node, pod) * plugin.weight;
    }, 0);

    if (total > bestScore) {
      bestScore = total;
      best = node;
    }
  }

  if (!best) throw new Error("No candidates after filtering");
  return best;
}

For clusters with hundreds of nodes, the scheduler does not score every candidate. It applies a configurable percentage threshold (percentageOfNodesToScore, defaulting to 50% with a floor of 100 nodes) to cap scoring work. Once enough candidates are found in the filter phase, scoring stops early. This trades optimal placement for scheduling throughput.

The Kubelet: Running Pods on a Node

The kubelet is the node agent. It watches the API server for pods scheduled to its node, then drives those pods through their full lifecycle using two main interfaces: CRI and CNI.

Container Runtime Interface (CRI)

CRI is a gRPC API between the kubelet and a container runtime (containerd, CRI-O). The kubelet does not know about Docker internals or OCI spec details directly: it calls CRI methods and the runtime handles the rest.

When a pod is scheduled to a node, the kubelet:

1. Calls RunPodSandbox to create the pod’s network namespace and pause container (the shared network namespace holder).
2. Pulls container images via PullImage for any not already present.
3. Calls CreateContainer and StartContainer for each container in the pod spec.
4. Begins probing: runs liveness probes (restart container if failing), readiness probes (gate traffic via the Endpoints object), and startup probes (suppress liveness during initialization).

Container Network Interface (CNI)

After the sandbox is created but before containers start, the kubelet calls the CNI plugin binary to set up networking for the pod’s network namespace. The CNI plugin (Calico, Cilium, Flannel, etc.) is responsible for:

• Assigning a cluster-scoped IP to the pod from the node’s CIDR block.
• Setting up the veth pair connecting the pod’s network namespace to the node’s root namespace.
• Programming routes or overlay tunnels so that pod-to-pod traffic across nodes works.

The pod IP is reported back to the kubelet, which writes it into the pod’s status.podIP field on the API server. After that, the Endpoints controller (watching pods and Services) adds the pod’s IP and port to the relevant Endpoints object, which is what kube-proxy translates into actual routing rules.

Pod Termination

Pod termination is more involved than creation. When you delete a pod, the API server sets deletionTimestamp on the object. The kubelet sees this and:

1. Stops the readiness probe immediately (pod is removed from Endpoints, traffic stops routing to it).
2. Sends SIGTERM to all containers.
3. Waits up to terminationGracePeriodSeconds (default 30) for containers to exit cleanly.
4. Sends SIGKILL to any container still running after the grace period.
5. Calls CNI to tear down the network namespace.
6. Calls RemovePodSandbox via CRI.
7. Removes the finalizer from the pod, which allows the API server to actually delete the object from etcd.

The six-minute Terminating state you see in production is almost always: a node is unreachable (kubelet cannot confirm teardown), or a finalizer is set and nothing is removing it, or a container is ignoring SIGTERM and the grace period is very long. The pod object stays in etcd until the kubelet confirms it is gone.

kube-proxy: Service Networking

A Kubernetes Service is not a running process. It is a virtual IP (the ClusterIP) that needs to be mapped to real pod IPs. kube-proxy does this mapping at the kernel level on every node.

kube-proxy watches the API server for Service and Endpoints objects. When either changes, it programs the node’s network stack to forward traffic destined for the ClusterIP to one of the pod IPs behind it.

iptables mode

In iptables mode (the long-standing default), kube-proxy writes DNAT rules into the PREROUTING and OUTPUT chains. Traffic hitting the ClusterIP on the correct port is redirected to a randomly selected pod IP using a chain of statistically weighted rules.

-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp --dport 443 -j KUBE-SVC-XYZABC
-A KUBE-SVC-XYZABC -m statistic --mode random --probability 0.33333 -j KUBE-SEP-POD1
-A KUBE-SVC-XYZABC -m statistic --mode random --probability 0.50000 -j KUBE-SEP-POD2
-A KUBE-SVC-XYZABC -j KUBE-SEP-POD3
-A KUBE-SEP-POD1 -p tcp -j DNAT --to-destination 10.244.1.5:8080

The problem with iptables mode at scale: rules are evaluated linearly, and a service with 1000 endpoints means the kernel walks up to 1000 rules per connection. Rule programming is also non-incremental: adding one endpoint rewrites the entire chain. At 10,000+ services this becomes measurably slow.

IPVS mode

IPVS (IP Virtual Server) uses hash tables in the kernel’s netfilter layer. Rule lookup is O(1) regardless of endpoint count. Programming is incremental. For clusters with more than a few hundred services, IPVS mode is the right choice.

kube-proxy in IPVS mode creates a dummy interface (kube-ipvs0), assigns all ClusterIPs to it, and programs IPVS virtual server entries pointing to real server (pod IP) backends. The kernel handles load balancing using one of several algorithms: round-robin, least-connection, or source hashing.

Enabling IPVS mode requires ipvs kernel modules on every node (ip_vs, ip_vs_rr, ip_vs_wrr, ip_vs_sh, nf_conntrack). If those modules are absent, kube-proxy falls back to iptables silently.

Admission Controllers: The Policy Enforcement Layer

Admission controllers run inside the API server process and intercept object writes before they are persisted to etcd. They run after authentication and authorization but before schema validation stores the final object.

There are two types:

Mutating admission controllers can modify the object. The MutatingAdmissionWebhook lets you point the API server at an external HTTPS endpoint. The API server sends an AdmissionReview request and your webhook can patch the object in the response. Common use cases: inject sidecar containers (service mesh proxies, log shippers), add default resource limits, attach labels or annotations automatically.

Validating admission controllers can only approve or reject. ValidatingAdmissionWebhook works the same way but cannot patch. Common use cases: block pods without resource limits set, enforce naming conventions, reject images from untrusted registries.

// Simplified AdmissionReview handler in TypeScript (Express + a webhook server)
import type { Request, Response } from "express";

interface AdmissionReview {
  apiVersion: string;
  kind: "AdmissionReview";
  request: {
    uid: string;
    object: Record<string, unknown>;
    operation: "CREATE" | "UPDATE" | "DELETE" | "CONNECT";
  };
}

interface AdmissionResponse {
  apiVersion: string;
  kind: "AdmissionReview";
  response: {
    uid: string;
    allowed: boolean;
    status?: { message: string };
    patch?: string; // base64-encoded JSON patch for mutating webhooks
    patchType?: "JSONPatch";
  };
}

function handleAdmission(req: Request, res: Response): void {
  const review: AdmissionReview = req.body;
  const pod = review.request.object as { spec?: { containers?: { resources?: unknown }[] } };

  const hasResourceLimits = pod.spec?.containers?.every(
    (c) => c.resources !== undefined
  );

  const response: AdmissionResponse = {
    apiVersion: "admission.k8s.io/v1",
    kind: "AdmissionReview",
    response: {
      uid: review.request.uid,
      allowed: hasResourceLimits ?? false,
      ...(!hasResourceLimits && {
        status: { message: "All containers must specify resource limits" },
      }),
    },
  };

  res.json(response);
}

Mutating webhooks run before validating webhooks, and both run before the object is written to etcd. Order within each phase is determined by the reinvocationPolicy and alphabetical ordering of webhook names. If a webhook is down and failurePolicy is Fail, the API request fails. If failurePolicy is Ignore, the webhook is skipped silently. For policy-critical webhooks, Fail is the correct setting: otherwise a degraded webhook server means no policy is enforced.

Built-in admission controllers (LimitRanger, ResourceQuota, PodSecurity, ServiceAccount, DefaultStorageClass) run as compiled-in code before external webhooks. PodSecurity replaced the deprecated PodSecurityPolicy in 1.25 and enforces the Pod Security Standards (Privileged, Baseline, Restricted) at the namespace level.

Production Considerations

etcd health is everything. A degraded etcd quorum slows all API server writes and can make the cluster read-only. Run etcd on dedicated nodes, monitor wal_fsync_duration_seconds (should be under 10ms on SSDs), and keep etcd cluster size odd (3 or 5 nodes). Do not run etcd on nodes shared with application workloads.

Scheduler throughput vs. placement quality. The default percentageOfNodesToScore of 50% means large clusters may not find the globally optimal node. For workloads where placement quality matters (GPU jobs, topology-aware streaming), consider custom schedulers or scheduler extenders. For most stateless services, the default is fine.

Admission webhook latency adds to every write. If your webhook takes 200ms to respond and you have five webhooks, every kubectl apply waits at least 1 second before the object is persisted. Keep webhooks fast. Cache expensive checks. Set a timeoutSeconds of 5 or less. Scope webhooks narrowly with namespaceSelector and objectSelector to skip resources they do not need to inspect.

kube-proxy is not the only option. Cilium with eBPF bypasses iptables and kube-proxy entirely, programming packet routing directly in the kernel using BPF maps. This reduces latency by eliminating conntrack overhead and scales better. It requires a kernel version of 4.9.17+ (practically, 5.10+ for full feature parity) and adds operational complexity. At medium scale the default kube-proxy in IPVS mode is good enough.

Resource requests are the scheduler’s only input. The scheduler places pods based on requests, not limits. A node where every pod’s limit is 4 CPU but requests are 0.5 CPU can be scheduled with 8x overcommit. When those pods actually consume resources, the node becomes CPU-pressured, triggering throttling or OOM kills. Set requests to realistic production p95 consumption, not minimums.

Tradeoffs: Kubernetes vs. Other Orchestrators

Dimension	Kubernetes	Nomad	Amazon ECS	Docker Swarm	Bare Metal (manual)
Scheduling model	Declarative, pluggable filter/score	Bin packing, constraint evaluation	Task placement constraints	Service spread/pack strategies	Manual
Learning curve	High (many concepts, large API surface)	Medium	Low (AWS-integrated)	Low	None for setup, high for ops
Multi-workload support	Pods, Jobs, CronJobs, DaemonSets, StatefulSets	Containers, VMs, raw executables, Java	Tasks and services (containers)	Services and stacks	Arbitrary
Networking model	CNI plugins, flexible, complex	Nomad CNI or host networking	VPC-native, ENI per task	Overlay (ingress mesh)	Direct, no abstraction
Service discovery	kube-dns, CoreDNS, Services	Consul integration (first-class)	Cloud Map or internal	Built-in DNS	Manual (Consul, etc.)
Autoscaling	HPA, VPA, KEDA, cluster-autoscaler	Nomad autoscaler (plugin)	Target tracking, step scaling	Limited	None
Stateful workloads	StatefulSets, PVCs (workable, complex)	Volumes with host or CSI	EFS/EBS mount (limited ordering)	Not well supported	Straightforward
Ecosystem / tooling	Very large (Helm, Argo, Flux, Karpenter)	Growing (Terraform-native)	AWS ecosystem only	Small, mostly stagnant	N/A
Operational overhead	High	Medium	Low (managed control plane)	Low	Low to start, high at scale
Multi-cloud portability	High	High	Low (AWS-native)	Medium	High

Kubernetes wins on ecosystem depth and flexibility. It loses on operational complexity, especially for small teams running fewer than a few dozen services. ECS is the practical choice when you are already deep in AWS and want managed control planes. Nomad is worth evaluating for polyglot workloads or when you find the Kubernetes API surface genuinely overwhelming for your use case.

Closing

Kubernetes is not magic. It is a collection of controllers, each watching a slice of API server state and reconciling differences with calm persistence. The API server is a validated write-ahead log. etcd is the durable storage. The scheduler is a filter-then-score pipeline that produces a single node binding. The kubelet is a state machine advancing pod lifecycle through CRI and CNI. kube-proxy is a kernel rule programmer. Admission controllers are synchronous middleware on every write. Once you see the boundaries between these components clearly, most cluster behavior that seems opaque becomes predictable. And the debugging process becomes a matter of finding which controller’s reconciliation loop is stuck, and why.