How Kubernetes Works Internally: The Control Loop, API Server, and Scheduler Behind Container Orchestration

When you run kubectl apply -f deployment.yaml, something moves. Within seconds a pod is scheduled, pulled, started, and passing health checks. The gap between “you pushed YAML” and “your container is running” spans five or six distinct subsystems, each with its own consistency model, failure behavior, and production failure mode.

Most engineers treat that gap as a black box. This article opens it.

Understanding Kubernetes internals matters in practice. When the scheduler is slow during a scale event, when a controller loop is stuck, when an admission webhook causes a 10-second delay on every deploy, you need a mental model of the moving parts to find the right lever. Reading dashboards is not enough. You need to understand what is normal and what is broken.

The Foundational Abstraction: Control Loops Everywhere

Before touching any specific component, the single most important concept is the control loop (also called a reconciliation loop). Every major Kubernetes component is, at its core, a process that:

1. Reads the current state of some resource from the API server.
2. Compares it to the desired state declared in the API server.
3. Takes an action to close the gap.
4. Repeats continuously.

This is declarative system design implemented as a feedback loop. You do not tell Kubernetes “create a pod on node-3.” You tell it “I want 3 replicas of this pod spec,” and the system figures out what actions close the gap between what is and what should be. The scheduler, the kubelet, the Deployment controller, the ReplicaSet controller, the endpoint slice controller — all of them are control loops watching for drift and reacting to it.

The power of this model is that every component is self-healing by construction. A controller that crashes and restarts simply re-reads state and resumes. Network partitions heal themselves when connectivity returns. You get eventual consistency without any component needing to track what it was “in the middle of doing.”

The limitation is that convergence is not instantaneous. Every hop through the system adds latency, and under high churn the loops queue up work faster than they can process it.

The API Server: The Single Source of Truth

The API server (kube-apiserver) is the only component that talks directly to etcd. Every other component in the control plane, including the scheduler, controllers, and kubelet, reads and writes exclusively through the API server’s REST interface. This is not an architectural nicety — it is the mechanism that keeps the cluster state consistent.

When a request arrives at the API server (from kubectl, a controller, or the kubelet), it passes through a staged pipeline:

1. Authentication. The API server validates the identity of the caller using one of several mechanisms: client certificates, bearer tokens (including OIDC tokens), service account tokens (JWT validated against the cluster’s signing key), or webhook token authenticators. Unauthenticated requests are rejected immediately.

2. Authorization. After authentication, the API server checks whether the authenticated identity has permission to perform the requested verb on the requested resource. RBAC is the dominant mode: the API server evaluates matching ClusterRole or Role bindings to determine permission. This check happens before any mutation.

3. Admission control. This is where most production surprises hide. Admission controllers run after auth but before the object is persisted. Kubernetes ships with built-in admission controllers (LimitRanger, ResourceQuota, DefaultStorageClass, PodSecurity), but the more interesting ones are the extension points: MutatingAdmissionWebhook and ValidatingAdmissionWebhook.

Mutating webhooks transform objects in-flight. A common use is injecting a sidecar container (like a service mesh proxy) into every Pod before it is written to etcd. Validating webhooks can reject objects that fail policy checks.

The catch is latency. Each webhook is an external HTTPS call that blocks the request pipeline. A webhook with a 200ms p99 response time adds 200ms to every kubectl apply. If the webhook server is unavailable and failurePolicy is set to Fail, every write to the cluster blocks until the webhook times out.

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: sidecar-injector
webhooks:
  - name: inject.sidecar.example.com
    failurePolicy: Ignore  # Fail vs Ignore: a real production decision
    timeoutSeconds: 5
    rules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        resources: ["pods"]
        operations: ["CREATE"]
    clientConfig:
      service:
        name: sidecar-injector
        namespace: kube-system
        path: /mutate

4. Validation and schema enforcement. The API server validates the object against its OpenAPI schema, checking required fields, type constraints, and field immutability rules. Custom Resource Definitions can include CEL validation expressions at this layer.

5. Persistence to etcd. After passing all the above, the object is serialized and written to etcd. The API server uses optimistic concurrency: every object has a resourceVersion field that maps to an etcd revision. Writes include the current resourceVersion, and etcd rejects writes where the version does not match, preventing lost updates.

6. Watch notifications. After a successful write, the API server fans out change events to every client that has a watch open on that resource type. This is the mechanism that drives all controller loops and the scheduler. Rather than polling, every component sets up a long-lived watch and reacts to events as they stream in.

The watch mechanism is efficient but not free. Each watch consumes a goroutine and memory on the API server. In clusters with thousands of nodes and hundreds of controllers, the API server’s watch fanout becomes a significant resource consumer.

etcd: Why Cluster State Lives Here

etcd is a distributed key-value store built on the Raft consensus protocol. It provides strong consistency: reads always return the most recently committed value, and writes are atomic across the cluster. Kubernetes uses etcd as its only persistent store.

The Raft model means etcd requires a quorum write (majority of nodes confirming the write) before acknowledging success. With a 3-node etcd cluster, any single node failure is tolerable. With 5 nodes, 2 can fail. Production control planes run etcd on dedicated nodes, separated from the API server, with fast local SSDs and low-latency networking between etcd members.

etcd stores Kubernetes objects as serialized protobuf under structured key paths: /registry/pods/default/my-pod, /registry/deployments/production/api-server, and so on. The key space is per-resource-type and per-namespace, which allows the API server to do efficient range scans (list all pods in a namespace) as a single etcd range query.

A critical operational constraint: etcd is not designed to be a high-throughput database. It is optimized for strong consistency and small key sizes, not bulk writes. Production clusters that generate high object churn (rapidly scaling jobs, high-frequency custom resource updates) can saturate etcd’s write throughput and cause API server latency spikes. Compaction and defragmentation are ongoing operational concerns.

The Scheduler: Filtering Then Scoring

When a Pod is created without a nodeName assigned, it lands in etcd with no scheduling decision made. The scheduler (kube-scheduler) watches for pods in this state and runs them through a two-phase process.

Phase 1: Filtering (Predicates)

The scheduler runs every node in the cluster through a set of filter plugins. A node that fails any filter is eliminated from consideration for this pod. Filters include:

• NodeUnschedulable: removes nodes marked as unschedulable.
• NodeResourcesFit: checks that the node has sufficient CPU and memory for the pod’s requests.
• NodeAffinity: enforces nodeSelector and nodeAffinity rules.
• PodTopologySpread: enforces topology spread constraints across zones or nodes.
• TaintToleration: eliminates nodes whose taints the pod does not tolerate.
• VolumeBinding: checks that persistent volume claims can be satisfied on the node.

The result of filtering is a feasible set: the nodes that can legally run this pod. If the feasible set is empty, the pod stays unscheduled and the scheduler emits an event. This is where kubectl describe pod shows “0/10 nodes are available: insufficient memory.”

// Conceptual model of the filtering pass
interface Node {
  name: string;
  allocatableMemoryMb: number;
  allocatableCpu: number; // millicores
  taints: { key: string; effect: string }[];
  labels: Record<string, string>;
}

interface PodSpec {
  requestedMemoryMb: number;
  requestedCpu: number; // millicores
  tolerations: { key: string; effect: string }[];
  nodeSelector: Record<string, string>;
}

function filterNodes(nodes: Node[], pod: PodSpec): Node[] {
  return nodes.filter((node) => {
    // Resource check
    if (node.allocatableMemoryMb < pod.requestedMemoryMb) return false;
    if (node.allocatableCpu < pod.requestedCpu) return false;

    // Taint check: every taint must have a matching toleration
    const untolerated = node.taints.filter(
      (taint) =>
        !pod.tolerations.some(
          (t) => t.key === taint.key && t.effect === taint.effect
        )
    );
    if (untolerated.length > 0) return false;

    // Node selector
    for (const [key, value] of Object.entries(pod.nodeSelector)) {
      if (node.labels[key] !== value) return false;
    }

    return true;
  });
}

Phase 2: Scoring (Priorities)

From the feasible set, the scheduler scores each node using a set of score plugins. Each plugin contributes a score in the range [0, 100], and the final score is a weighted sum across plugins. The node with the highest total score wins.

Key score plugins:

• NodeResourcesBalancedAllocation: favors nodes where the pod’s request fills CPU and memory proportionally, preventing fragmented nodes where CPU is exhausted but memory is wasted.
• LeastAllocated: favors nodes with the most remaining allocatable resources, spreading load.
• ImageLocality: gives a bonus to nodes that already have the pod’s container image cached.
• InterPodAffinity: adds scores for pods that request affinity to other pods (or anti-affinity).
• NodeAffinity: gives a soft preference score for nodes matching preferred nodeAffinity terms.

After scoring, the scheduler writes the selected nodeName back to the Pod object via a binding operation, a lightweight API call that patches only the spec.nodeName field. Once etcd persists this patch, the kubelet on that node picks it up via its watch and begins actually starting the pod.

The Kubelet: Where the Loop Meets the Linux Host

The kubelet is the agent running on every node. It watches for pods assigned to its node and drives those pods toward their desired state. This is the point where Kubernetes abstractions meet actual Linux primitives.

When the kubelet sees a newly assigned pod:

1. It calls the container runtime (via CRI, the Container Runtime Interface, using gRPC) to pull the image if not cached.
2. It calls the CNI plugin to set up the pod’s network namespace and attach it to the cluster network.
3. It calls the CSI plugin (if applicable) to mount persistent volumes.
4. It starts init containers in sequence, waiting for each to complete before proceeding.
5. It starts main containers and begins running liveness/readiness probes.

The kubelet continuously reconciles. If a container exits unexpectedly, the kubelet compares the pod’s restartPolicy against the current state and restarts or terminates accordingly. If a liveness probe fails, it kills the container and lets the restart policy drive the next action. If the node runs low on memory and evictionThreshold is crossed, the kubelet evicts pods ordered by their QoS class.

Kubelet health reporting is also part of the reconciliation: the kubelet updates the pod’s status.conditions fields and posts NodeCondition updates to the API server. This is how the control plane knows whether a node is Ready, MemoryPressure, or DiskPressure.

// Simplified kubelet reconciliation conceptual model
interface PodStatus {
  phase: "Pending" | "Running" | "Succeeded" | "Failed" | "Unknown";
  containerStatuses: {
    name: string;
    ready: boolean;
    restartCount: number;
    state: "running" | "terminated" | "waiting";
  }[];
}

interface DesiredPodSpec {
  containers: { name: string; image: string }[];
  restartPolicy: "Always" | "OnFailure" | "Never";
}

function reconcile(desired: DesiredPodSpec, current: PodStatus): string[] {
  const actions: string[] = [];

  for (const container of desired.containers) {
    const status = current.containerStatuses.find(
      (c) => c.name === container.name
    );

    if (!status) {
      actions.push(`start container: ${container.name}`);
      continue;
    }

    if (status.state === "terminated" && desired.restartPolicy === "Always") {
      actions.push(`restart container: ${container.name}`);
    }
  }

  return actions;
}

Controllers: The Declarative Layer Above the Kubelet

The kubelet handles a single pod on a single node. Controllers handle higher-level objects. A Deployment controller watches Deployment objects and ensures the correct number of ReplicaSets exist. The ReplicaSet controller watches ReplicaSets and ensures the correct number of Pod objects exist. The kubelet watches Pods and ensures the containers are actually running.

This layered model is intentional. Each controller is narrowly scoped, independently restartable, and handles exactly one level of abstraction. Adding a new resource type means writing a new controller, not modifying existing ones.

The controller manager (kube-controller-manager) runs the built-in controllers as goroutines in a single process. Custom controllers (Operators) run separately, typically deployed as pods in the cluster. Both use the same pattern: set up an informer (a cached watch with local indexing), register event handlers for Add/Update/Delete events, and push work items onto a rate-limited work queue for processing.

The work queue is critical for correctness under bursty conditions. When 50 pods are deleted simultaneously, the queue absorbs the spike. The controller processes items at a controlled rate, retrying with exponential backoff on failures, so transient errors do not cause tight retry loops that hammer the API server.

Scheduler Strategies: Tradeoffs

Strategy	Behavior	Strengths	Weaknesses
LeastAllocated	Spreads pods across nodes with most free resources	Even utilization, good for bursty workloads	Nodes near 100% capacity when cluster is full
MostAllocated	Packs pods onto fewer nodes	Enables bin-packing, lowers cost	Hot nodes, lower fault isolation
TopologySpread	Distributes across zones/racks	High availability across failure domains	Requires careful skew settings; can leave pods unschedulable
NodeAffinity (hard)	Restricts to nodes matching labels	Deterministic placement for specialized hardware (GPU, SSD)	Reduces feasible set, increases scheduling pressure
PodAntiAffinity (hard)	Keeps replicas on different nodes	High availability within a zone	Can prevent scheduling when node count is tight
Descheduler (post-schedule)	Re-evaluates placement and evicts for rebalancing	Corrects drift after cluster changes	Adds eviction events; not real-time

In practice, most production clusters combine topology spread constraints (for availability) with resource-based scoring (for utilization). Pure bin-packing (MostAllocated) makes sense for batch workloads where cost matters more than latency. Pure spreading makes sense for customer-facing services where a zone failure should not take out a disproportionate share of capacity.

Production Considerations

Control plane sizing. The API server, scheduler, controller manager, and etcd all have different scaling axes. etcd scales with write throughput and object count. The API server scales with concurrent requests and watch connections. A rule of thumb for production: size your etcd nodes to keep p99 write latency under 10ms; keep etcd on nodes with local NVMe, not network-attached storage. For the API server, monitor the apiserver_request_duration_seconds histogram and the etcd_request_duration_seconds histogram separately to distinguish API server CPU overhead from etcd I/O latency.

Watch pressure. Each controller, each node’s kubelet, and each kubectl get --watch client holds open watches. At thousands of nodes, the API server’s watch fanout goroutine count and memory pressure become real. Use apiserver_watch_cache_events_dispatched_total and apiserver_watch_events_total to monitor fan-out load. In very large clusters, the watch cache in the API server (an in-memory snapshot used to serve most list and watch requests without hitting etcd directly) must be sized appropriately via --watch-cache-sizes.

Admission webhook latency. Profile your admission webhooks with realistic load before rolling them to production. A webhook that is fast at 5 QPS may degrade at 500 QPS during a rolling deployment. Set timeoutSeconds conservatively (3-5s) and choose failurePolicy: Ignore only when the webhook is advisory (mutation). For validating webhooks enforcing security policy, failurePolicy: Fail is correct but requires high availability of the webhook server itself.

Scheduler throughput. The default scheduler processes pod scheduling serially per scheduling cycle. Under high pod creation rates (rapid HPA scale-up, batch job submission), the scheduler can become a bottleneck. Monitor scheduler_scheduling_attempt_duration_seconds and scheduler_pending_pods metrics. If pending pod queues grow and scheduling latency increases, the bottleneck is usually filter/score plugin performance or the scheduler’s single-threaded scheduling cycle rather than the API server.

etcd compaction. etcd retains the full revision history of all key writes for the duration of its compaction period. Without regular compaction, the database grows without bound and memory pressure builds. Set --auto-compaction-mode=periodic and --auto-compaction-retention=1h for most clusters. Monitor etcd_db_total_size_in_bytes and alert before it approaches the --quota-backend-bytes limit (default 2GB), which causes etcd to enter an alarm state and reject all writes.

Node lease heartbeats. Node health is determined by lease objects in the kube-node-lease namespace. Each kubelet renews its node lease every 10 seconds by default. If the lease is not renewed within --node-lease-duration-seconds (40 seconds default), the node controller marks the node NotReady. Network hiccups, API server overload, or kubelet CPU starvation can all trigger false NotReady events. Tune --node-monitor-grace-period and --node-monitor-period carefully, because premature NotReady marking triggers pod eviction from the node.

Closing

Kubernetes is a distributed system built from control loops operating on a consistent store. Each component does exactly one thing: the API server serializes access to etcd, the scheduler assigns pods to nodes, controllers maintain the declared replica count and rollout state, and the kubelet drives containers into existence on the actual Linux host. The elegance is in how these loops compose: each one handles its own abstraction level, retries independently, and resumes correctly after any failure.

The failure modes, when they appear, almost always trace back to one of three root causes: the API server or etcd under write pressure, an admission webhook with latency or availability issues, or a control loop consuming API server resources faster than intended. All three are observable through the control plane metrics surfaces. Once you know where each loop lives and what it watches, debugging becomes finding which loop stopped making progress and why.