Kubernetes Architecture: How Container Orchestration Really Works

1. Why Kubernetes exists

Kubernetes is an open source platform for running containerized workloads and services. It is portable and extensible, and it supports both declarative configuration and automation . That definition is accurate but abstract. The easier way in is to look at the jobs someone has to do when containers run on more than one machine.

The jobs nobody wants to do by hand

Picture three servers and a dozen containers placed on them by hand. Very quickly you need answers to the same questions every day:

• Placement. Which server has room for the next container? Kubernetes asks you to say how much CPU and memory each container needs, then fits containers onto the nodes you gave it .
• Failures. A process crashes at 3 a.m. Kubernetes restarts failed containers, replaces them, kills containers that fail health checks, and holds traffic back until they are ready .
• Scaling. Traffic doubles. Kubernetes supports horizontal scaling, including automatic scaling on CPU usage .
• Discovery. Clients need to find copies that keep moving. Kubernetes can expose containers through a DNS name or its own IP address and spread traffic across instances .
• Rollouts. You describe the new desired state and Kubernetes moves the actual state towards it at a controlled rate, with rollback available .
• Configuration. Secrets and configuration can be updated without rebuilding images .
• Utilization. Packing containers onto nodes by their stated needs is how a cluster avoids one idle server next to an overloaded one.

The common idea is that you stop issuing step-by-step instructions. Kubernetes is built from independent control processes, each continuously driving the current state towards the desired state, rather than a fixed "do A, then B, then C" workflow .

Try it

The lab below puts the two approaches side by side. On the left, three servers run hand-placed containers; on the right, a cluster runs a Deployment with a set number of replicas. Raise the traffic or crash a server and compare: the left side grows a to-do list for a human, while the right side shows controllers noticing that the desired count no longer matches what is observed and creating replacement Pods on the surviving nodes. The simulation compresses time, and note what it does not do: it never adds a machine.

What Kubernetes is not

Kubernetes is not a traditional all-inclusive PaaS. It offers PaaS-like features such as deployment, scaling and load balancing, but those defaults are optional and pluggable . It does not build your application or deploy source code; CI/CD is left to you . It has no built-in middleware, databases, caches or cluster storage systems, and it does not dictate logging, monitoring or alerting, though all of those can run on it as ordinary workloads .

Most importantly, Kubernetes does not configure, maintain or heal machines. Its self-healing covers containers and Pods, not the hosts underneath them .

A realistic scenario

A team runs a web service as five replicas across three nodes. One node's power supply fails. The control plane eventually decides the node is unreachable, and the Deployment's controllers create replacement Pods that are scheduled onto the two remaining nodes, if they have room. Nobody gets a new server out of this. If the remaining nodes are full, the replacements wait as Pending until capacity appears, and that capacity has to come from a person or a separate node autoscaler.

Self-healing is not provisioning

The single most common misunderstanding about Kubernetes is that it "adds servers when it needs them". It does not create nodes itself. A Node object gets into the API either because a kubelet registered itself or because someone added the object by hand .

Adding and removing machines automatically is the job of separate node autoscalers. They create and delete the cloud resources, usually VMs, behind Nodes, so each one has to be integrated with each cloud provider . The two that SIG Autoscaling sponsors are Cluster Autoscaler and Karpenter, and they decide using Pod resource requests, not actual usage .

Pod-level scaling is a different layer again. The HorizontalPodAutoscaler changes a workload's replica count through its scale subresource . More replicas only help if there is room to schedule them; the HPA itself never adds a node.

Layer — What changes — Who does it; Container restart — A failed container is restarted in the same Pod — The kubelet on that node; Pod replacement — A new Pod with a new UID replaces a lost one — Workload controllers through the API server; Replica count — spec.replicas goes up or down — A person, or the HPA; Machines — Nodes are created or removed — People, or a node autoscaler outside core Kubernetes

Constant change is normal

Kubernetes expects constant change. A cluster may never settle into a stable state, and that is fine as long as the controllers keep running and making useful changes . This is why "the cluster is converging" is a more accurate description than "the deploy finished" for most of what happens.

Where it came from

The name comes from the Greek word for helmsman or pilot, and "K8s" counts the eight letters between the K and the s . Google open sourced Kubernetes in 2014, drawing on more than 15 years of running production workloads . The Overview page does not mention Borg, but the 2015 Kubernetes Blog describes Kubernetes as a direct descendant of Borg, Google's internal cluster manager, and notes that many early Kubernetes developers had worked on it . The Pod idea came from Borg's "alloc", and Borg's Jobs and Tasks were the starting point for Kubernetes' more flexible label-based grouping .

This study describes the current docs. On the access date, the project maintained release branches for the three latest minor releases: 1.37, 1.36 and 1.35 .

Investigate it

``bash # Client and server versions kubectl version

# The machines the cluster knows about (it did not create them) kubectl get nodes -o wide

# Every resource type this API server serves kubectl api-resources

# A Deployment's desired and observed replica counts kubectl get deployment web ``

Whether a cluster grows when Pods are Pending depends entirely on whether a node autoscaler is installed and how it is configured for your provider. Managed offerings differ here; check yours rather than assuming.

2. The Kubernetes mental model

Kubernetes has two kinds of thing, and keeping them apart makes the rest of this study easier.

The first kind is physical and running: a cluster is a control plane plus worker nodes, and it needs at least one worker node to run Pods . A node is a virtual or physical machine . A Pod is the smallest deployable unit: one or more containers that share storage and network and are always scheduled together .

The second kind is records in the API. Kubernetes objects are persistent entities that represent the state of the cluster. Creating one records an intent, and the system keeps working to make it true . A Deployment, a ReplicaSet, a Service and a ConfigMap are all objects. So, confusingly, are Pods and Nodes: each running thing also has an API object that describes it.

Spec and status

Most objects have a spec, the desired state that you set, and a status, the current state written by Kubernetes components . The control plane keeps reconciling the two: if one of a Deployment's three replicas fails, it starts a replacement to close the gap . Controllers do this as control loops, like a thermostat that compares the temperature you set with the room temperature and switches equipment to close the gap .

Every manifest needs apiVersion, kind, metadata and spec, and kubectl turns the YAML into JSON API requests :

``yaml apiVersion: apps/v1 kind: Deployment metadata: name: web namespace: default spec: replicas: 3 selector: matchLabels: app: web template: metadata: labels: app: web spec: containers: - name: web image: nginx:1.27 resources: requests: cpu: 100m memory: 128Mi `

Labels and selectors

Labels are key/value pairs used to organise and select objects. They mean nothing to the core system, but selecting by label is the basic way Kubernetes groups objects . Services, Deployments and NetworkPolicies all find their Pods this way . Selectors can be equality-based or set-based; comma-separated requirements are ANDed and there is no OR operator .

Explore the tree

The lab below is a clickable map of a small cluster. Click the cluster, the control plane, a node, a Pod or a container, then the API objects alongside them: Deployment, ReplicaSet, Service, EndpointSlice, ConfigMap and Namespace. Each panel gives a definition, what the thing is responsible for, how long it lives, what it is related to, a common misconception and the kubectl commands to look at it.

A realistic scenario

You apply the manifest above. Nothing about it says "start three containers". It records that a Deployment named web should exist with three replicas. The Deployment controller manages that through a ReplicaSet it owns ; the ReplicaSet creates or deletes Pods from its template until the count matches . A Service with the selector app: web` would pick up those Pods by label and keep tracking them as they come and go . You rarely create Pods yourself: workload resources do it, because Pods are ephemeral and disposable .

Names, UIDs and generations

A name is unique per resource type, and per namespace for namespaced types; a UID is unique across the whole cluster . UIDs are generated by the system and never reused, which is how a re-created object with the same name is told apart from the old one . Most names must be valid DNS subdomain names: at most 253 characters of lowercase alphanumerics, - or . . With generateName the server adds a random suffix, and since v1.31 it makes up to 8 attempts before returning HTTP 409 .

metadata.generation is a system-set, read-only sequence number for the spec . A Deployment's status.observedGeneration records the generation the controller last observed . Comparing the two is a common way to tell whether the status reflects your latest change, though the pages read do not spell that rule out .

Ownership

Objects point at their owners through metadata.ownerReferences. A ReplicaSet finds its Pods this way, and it also adopts any Pod that matches its selector and has no controller owner . That adoption rule is why a stray Pod with the right labels can be counted as one of a ReplicaSet's replicas . Cross-namespace owner references are not allowed , and an object is garbage collected once every owner listed has been deleted .

Namespaces

Namespaces isolate groups of resources inside one cluster and scope names. They apply only to namespaced objects such as Deployments and Services, not to cluster-wide ones such as Nodes, StorageClasses and PersistentVolumes . A cluster starts with four: default, kube-node-lease, kube-public and kube-system .

Declarative versus imperative

kubectl has three ways to manage objects: imperative commands, imperative object configuration and declarative object configuration. Pick one per object, because mixing them gives undefined behaviour .

Style — Example — Notes; Imperative command — kubectl create deployment web --image=nginx:1.27 — Acts on live objects and keeps no history of earlier configurations; Declarative — kubectl apply -f web.yaml — Works out create, update or delete per object and patches only the differences, so other writers' changes survive

Client-side kubectl apply computes its patch from the file, the live object and the last-applied-configuration annotation . Server-Side Apply is meant to replace it; you opt in with --server-side , and the API server then tracks which field manager owns each field .

"Declarative" does not mean "instant" or "guaranteed". Applying a manifest stores intent. Whether three Pods actually run depends on scheduling, images, quotas and more, which later chapters cover.

Investigate it

``bash # Spec and status side by side kubectl get deployment web -o yaml

# Select by label, equality-based and set-based kubectl get pods -l app=web --show-labels kubectl get pods -l 'app in (web,api)'

# Who owns this Pod? kubectl get pod <pod-name> -o jsonpath='{.metadata.ownerReferences[*].kind}'

# Preview a declarative change, then apply it server-side kubectl diff -f web.yaml kubectl apply --server-side -f web.yaml

# Cluster-scoped resource types kubectl api-resources --namespaced=false ``

3. Inside a Kubernetes cluster

A cluster has a control plane and worker nodes. In production the control plane usually spans several machines .

The pieces

• Control plane: kube-apiserver, etcd, kube-scheduler, kube-controller-manager and an optional cloud-controller-manager .
• Each node: the kubelet, a container runtime and an optional kube-proxy .
• Addons: built from ordinary Kubernetes resources such as DaemonSets and Deployments, with their namespaced parts in kube-system. Examples are DNS, the Dashboard, resource monitoring, cluster-level logging and network plugins . Cluster DNS is technically an addon, but the docs say every cluster should run it .

Kubernetes does not ship a network plugin in core. A CNI plugin assigns Pod IP addresses and lets Pods talk to each other .

Two kinds of traffic

It helps to split everything that moves through a cluster into two kinds.

Control traffic is API calls. Communication is hub-and-spoke: all API use from nodes and Pods ends at the API server, and no other control plane component exposes remote services . The scheduler, the controllers and every kubelet are API clients. They watch for changes and write results back. The API server is the front end to the cluster's shared state, which every other component goes through , and it stores that state in etcd .

Application traffic is runtime networking. When one Pod calls another, packets go across the Pod network directly, with no proxy and no NAT, unless someone has segmented the network on purpose . Service traffic is redirected by rules on each node; there is no central proxy . None of this passes through the control plane.

That split explains a lot of behaviour. If the API server is down, nothing can change, but traffic between running Pods keeps flowing over the network the plugin built.

Explore the diagram

The diagram below can be panned and switched between views: control-plane communication, workload lifecycle, networking, status reporting and persistent state. Click a component to see its inputs, outputs and what happens when it fails. The journey button replays a Deployment from kubectl apply to running Pods, highlighting each hop.

Optional pieces

kube-proxy is optional: some network plugins do Service packet forwarding themselves, and with those, nodes do not need kube-proxy . Where it runs, it uses the operating system's packet filtering layer when one is available and otherwise forwards traffic itself . cloud-controller-manager only exists where there is a cloud to talk to .

A realistic scenario

A developer runs kubectl logs against a Pod. The request goes to the API server like any other. The API server then opens a connection to the kubelet on the Pod's node to fetch the logs . Meanwhile the Pod's own clients reach it over the Pod network without involving the control plane at all. Two different paths, both "talking to the Pod".

Node to control plane

Nodes and Pods reach the API server on a secure HTTPS port, typically 443, with client authentication. Pods find it through the kubernetes Service in the default namespace, using an injected service account token and root certificate . Because every component goes through the API server, they coordinate by watching: a watch lets a client read the current state and then subscribe to later changes without missing events .

Control plane to node

The reverse direction is narrower and less protected by default:

• The API server opens connections to the kubelet for logs, attach and port-forward. By default it does not verify the kubelet's serving certificate, so those connections are open to man-in-the-middle attacks unless you set --kubelet-certificate-authority .
• API server proxy connections to nodes, Pods and Services default to plain HTTP, neither authenticated nor encrypted .
• SSH tunnels from the control plane to nodes are deprecated. The replacement is the Konnectivity service, where agents on the nodes dial out to a Konnectivity server and all control-plane-to-node traffic goes through those connections .

Who assigns which address

Pod, Service and Node IPs come from separate, non-overlapping ranges. The network plugin assigns Pod IPs, kube-apiserver assigns Service IPs, and the kubelet or cloud-controller-manager assigns node IPs . The container runtime, not the kubelet, loads and calls the CNI plugins . Kubernetes defines the network APIs but does not build the Pod network, so networking behaviour depends on the plugin you install .

How the control plane itself runs

The control plane can run as systemd services on dedicated machines, as static Pods run by the kubelet (common with kubeadm), self-hosted inside the cluster, or hidden behind a managed cloud service . With static Pods, a node's kubelet runs the components directly from local manifests, without the API server controlling them . On a managed service you usually cannot see control plane Pods at all.

Failure boundaries

If the API server's backing storage is lost, the API server cannot start, but kubelets keep running the same Pods and Service proxying keeps going . The cluster freezes rather than stops: no scaling, no rollouts, no replacements, until the control plane is back.

The networking half of the diagram is a model. Which component programs Service rules (kube-proxy in iptables, nftables or IPVS mode, or a plugin's own dataplane), and how cross-node packets travel, varies by network plugin and environment .

Investigate it

``bash # Where the API server is kubectl cluster-info

# System components and addons, with their nodes kubectl get pods -n kube-system -o wide kubectl get daemonsets,deployments -n kube-system

# The in-cluster Service that fronts the API server kubectl get service kubernetes -n default

# API server readiness checks kubectl get --raw='/readyz?verbose' ``

4. Control plane components

The control plane is made up of kube-apiserver, etcd, kube-scheduler, kube-controller-manager and an optional cloud-controller-manager . Its components can run on any machine; simple setup scripts usually put all of them on one machine and run no user containers there . Each has a narrow job.

Component — Reads — Writes — Job; kube-apiserver — Requests from every client — Objects to etcd — Validates and configures API objects and serves REST operations; etcd — Writes from the API server — Durable key-value data — Consistent, highly available store for all cluster data; kube-scheduler — Pods with no node assigned — A binding to a node — Picks a node for each new Pod; kube-controller-manager — Shared state, through the API server — Changes, through the API server — Runs many controllers in one process; cloud-controller-manager — Nodes and Services — Cloud resources and state — Cloud-specific controllers only

kube-apiserver

The API server is the core of the control plane; its HTTP API is how users, cluster parts and external components communicate . It scales horizontally: you run several instances and balance traffic across them . It stores the serialized state of objects in etcd , and the docs recommend that ideally only the API server can reach etcd, because access to etcd amounts to root access to the cluster .

etcd

etcd is leader-based, and a majority of members must agree on every update. Quorum for an n-member cluster is (n/2)+1 with integer division , so three members tolerate one failure and five tolerate two . It is sensitive to network and disk I/O; without an elected leader, Kubernetes cannot change cluster state and no new Pods can be scheduled .

kube-scheduler

The scheduler watches for new Pods with no node assigned and picks one, weighing resource requirements, constraints, affinity and anti-affinity, data locality, interference and deadlines . It does not start containers: it tells the API server its decision, which is called binding, and the kubelet on the chosen node runs the Pod .

kube-controller-manager

Logically each built-in controller is its own process, but they are compiled into one binary and run in one process . Examples include the Node controller, the Job controller, the EndpointSlice controller and the ServiceAccount controller, and the docs say that list is not exhaustive . Controllers usually act by sending requests to the API server rather than acting directly .

cloud-controller-manager

It runs only cloud-specific controllers: Node (was the VM deleted?), Route, and Service (cloud load balancers). On-premises and local learning clusters do not have one .

Follow a journey

The diagram below shows only the control plane plus a kubelet. Pick the journey "create a Deployment" or "a node stops heartbeating"; only the components involved light up, in order. Notice that every arrow starts or ends at the API server.

A realistic scenario

You create a Deployment. The API server stores it. The Deployment controller sees it and asks for a ReplicaSet; the ReplicaSet controller sees that and asks for Pods; the scheduler sees unassigned Pods and binds them; kubelets see Pods bound to their nodes and start them. No component calls another directly. Each one reacts to state it reads from the API server.

Leader election

You can run several copies of kube-scheduler and kube-controller-manager, but only one of each should act at a time. In HA setups they use Lease-based leader election so that one instance is active and the others wait on standby . Leader election is on by default (--leader-elect=true), and a replica must win leadership before it runs its main loop .

Setting — Default — Meaning; --leader-elect-lease-duration — 15s — In effect, the longest a stopped leader can go before another candidate replaces it; --leader-elect-renew-deadline — 10s — Must be less than the lease duration; --leader-elect-retry-period — 2s — How often candidates retry

For kube-controller-manager, the lock is a Lease named kube-controller-manager in kube-system . Behind the ControllerManagerReleaseLeaderElectionLockOnExit feature gate, it releases the lock during a leader transition instead of waiting for it to expire . The production guide says the scheduler should be fault tolerant but not necessarily highly available: if the primary goes away, another instance takes over through leader election .

The API server is scaled differently: you add instances and load-balance traffic across them .

What each failure looks like

• API server down. Nothing can be read or changed through the API. Kubelets keep running the Pods they already have .
• etcd loses quorum. If most members fail permanently, the cluster has failed: running Pods may keep running, but Kubernetes cannot change any state or schedule new Pods . If quorum is lost to a transient network failure, etcd resumes safely on its own once the network recovers .
• Scheduler down. New Pods are not bound to nodes. An unschedulable Pod normally stays Pending until it can be placed, rather than failing ; without a scheduler, every new Pod is in that position.
• Controller manager down. Nothing reconciles. Deleted Pods are not replaced and Deployments do not progress, because controllers are what close the gap between spec and status . In a resilient control plane another instance takes over .

etcd sizing

Run etcd with an odd number of members . Growing to an even count adds no fault tolerance, and an odd count guarantees that a majority side exists during a partition . For production the docs strongly recommend a static five-member cluster; a single member is for testing only . More members improve availability at the cost of performance, never add capacity, and etcd should not be autoscaled .

The API server stores resources under a key prefix that defaults to /registry ; the Secret secret1 in namespace default, for example, is stored at /registry/secrets/default/secret1 .

No docs page read states word for word that the API server is the only component that talks to etcd. The docs give it as a security recommendation and describe the hub-and-spoke pattern .

Investigate it

``bash # Leader election locks for control plane components kubectl get leases -n kube-system kubectl get lease kube-controller-manager -n kube-system -o yaml

# Control plane Pods, if your cluster runs them as Pods kubectl get pods -n kube-system -o wide

# API server health endpoints kubectl get --raw='/livez?verbose' kubectl get --raw='/readyz?verbose' ``

On a managed service the control plane is usually hidden, so the Pods and some Leases may not be visible to you.

5. Worker node anatomy

A node is a virtual or physical machine managed by the control plane. It runs the kubelet, a container runtime and usually kube-proxy . The node is where API objects finally become processes.

The kubelet

The kubelet is the primary node agent and runs on every node . It works from PodSpecs, mostly received from the API server, and makes sure the containers they describe are running and healthy; it ignores containers that Kubernetes did not create . Once a Pod's spec.nodeName is set, that node's kubelet is responsible for it .

The container runtime and the CRI

The kubelet does not run containers itself. It talks to the container runtime over the Container Runtime Interface, a gRPC protocol in which the kubelet is the client and the runtime serves a runtime service and an image service . Because the CRI is a plugin interface, runtimes can be swapped without recompiling Kubernetes; it has been stable since v1.23 . Supported runtimes include containerd, CRI-O and any other CRI implementation .

Starting a Pod on a node

• The kubelet sees a Pod bound to its node.
• It uses the CRI to create a Pod sandbox and set up its networking; when that is done, the PodReadyToStartContainers condition is set . The runtime calls the CNI plugin, which provides the Pod's network .
• Images are pulled according to imagePullPolicy .
• Containers are created and started, with each container's requests and limits passed to the runtime, which on Linux usually turns them into cgroup settings .
• Probes and conditions decide when the Pod is Ready , and the kubelet reports status back to the API server.

The lab below lets you select a Pod on one simulated node and step through exactly that sequence: kubelet watch, sandbox (CRI RunPodSandbox and a CNI-assigned IP), pull, create, start, readiness and status report. The event log on the right is generated from the simulation's state, and timing is compressed.

Joining the cluster

Self-registration is the default (--register-node is true), and most distributions use it . When a Node object exists, Kubernetes checks that a kubelet with a matching name has registered; if not, the Node is kept but ignored until it becomes healthy . Kubernetes assumes a node with the same name is the same machine, so a significantly changed host should be removed and re-added .

What a node reports

Node status has Addresses, Conditions, Capacity and Allocatable, Info and Declared Features, shown by kubectl describe node . Conditions include Ready, DiskPressure, MemoryPressure, PIDPressure and NetworkUnavailable . Capacity is the node's total resources; Allocatable is the part normal Pods can use .

A realistic scenario

A new VM boots with a kubelet configured to join. The kubelet registers a Node, which starts reporting status and heartbeats. The scheduler can now consider it for Pods. If the runtime does not support the v1 CRI API, the kubelet will not register the node at all , which is a common reason a "new node" never shows up.

Heartbeats and Leases

Nodes send two kinds of heartbeat: updates to the Node's .status, and a Lease object in kube-node-lease, one per Node. Leases are lightweight, which helps large clusters . Every heartbeat updates the Lease's spec.renewTime, and the control plane uses that timestamp to judge availability .

The kubelet renews its Lease every 10 seconds by default, retrying failed renewals with exponential backoff from 200 ms capped at 7 s . A node's Ready condition becomes Unknown when the node controller has not heard from it within node-monitor-grace-period, which defaults to 50 seconds .

The Node Status page disagrees with itself. One paragraph still says 40 seconds for the unreachable timeout . Before v1.32 the default was 40 seconds; v1.32 raised it to 50 . Use 50 s for current versions.

On full status updates, the docs read in two ways. The Node page says the kubelet updates .status when something changes, or otherwise every 5 minutes . The kubelet reference gives --node-status-update-frequency a 10 s default and says it must stay compatible with the node controller's grace period . Treat the Lease as the frequent heartbeat and .status as the detailed report.

Capacity, allocatable and what the scheduler counts

The scheduler admits Pods by summing container requests against what the node offers. Containers started directly by the runtime, and processes outside the kubelet, are not counted . By default a kubelet runs at most 110 Pods (--max-pods) .

Labels and cordoning

Node labels are only set at registration; restarting the kubelet with new --node-labels under the same name has no effect . kubectl cordon marks a node unschedulable: new Pods stay off it but existing Pods are untouched, and DaemonSet Pods tolerate it . "SchedulingDisabled" in kubectl output is not a real condition; it means spec.unschedulable is set .

Static Pods

The kubelet can also run Pods from local manifests with no API server involved. kubeadm uses this to run the control plane . For each static Pod the kubelet creates a read-only mirror Pod in the API; deleting the mirror does not stop the static Pod .

Runtimes and the dockershim history

The kubelet and runtime must use the same cgroup driver. cgroupfs is the kubelet default, but systemd is recommended when systemd is init or with cgroup v2 .

Earlier versions worked differently. Before v1.24, Kubernetes had a built-in Docker Engine integration called dockershim; its removal was announced with v1.20 and it was removed in v1.24 . It existed because Docker Engine does not implement CRI . Docker Engine can still be used through the external cri-dockerd adapter , and images built with docker build run on every CRI runtime .

Investigate it

``bash kubectl get nodes -o wide kubectl describe node <node-name>

# Per-node heartbeat Leases kubectl get leases -n kube-node-lease

# Capacity versus allocatable kubectl get node <node-name> -o jsonpath='{.status.capacity}{"\n"}{.status.allocatable}{"\n"}'

# Keep new Pods off a node, then allow them again kubectl cordon <node-name> kubectl uncordon <node-name> ``

What sits between the kubelet and the kernel (runtime, low-level runtime, CNI plugin, cgroup layout) varies by distribution and provider. The lab shows one plausible arrangement.

6. Pods and containers

A Pod is not a container. It is the smallest deployable unit in Kubernetes: one or more containers that share storage and network and are always co-located and co-scheduled . One container per Pod is the most common case, and Kubernetes manages the Pod rather than the containers directly. Several containers in one Pod is an advanced pattern, only for tightly coupled containers, and never a way to replicate .

What the containers share

• Network. Each Pod gets a unique IP per address family, and its containers share one network namespace, so they share ports and can talk over localhost .
• Volumes. All containers in a Pod can mount its shared volumes, and volume data survives a container restart .
• IPC. Containers in the same Pod can use SystemV semaphores or POSIX shared memory .

Under the hood each Pod gets its own Linux namespaces and cgroups, the same mechanisms that isolate a single container, so a Pod is like a set of containers with shared namespaces and shared volumes .

Init containers and native sidecars

Init containers run one at a time, in order, before app containers start, and each must succeed before the next . A native sidecar is an init container with restartPolicy: Always: it starts before the app and keeps running for the Pod's whole life. It is stable since v1.33 .

``yaml apiVersion: v1 kind: Pod metadata: name: web labels: app: web spec: volumes: - name: logs emptyDir: {} initContainers: - name: migrate image: busybox:1.36 command: ["sh", "-c", "echo schema ok"] - name: log-shipper image: busybox:1.36 restartPolicy: Always command: ["sh", "-c", "touch /var/log/app/app.log; tail -F /var/log/app/app.log"] volumeMounts: - name: logs mountPath: /var/log/app containers: - name: app image: nginx:1.27 resources: requests: cpu: 100m memory: 128Mi limits: memory: 256Mi volumeMounts: - name: logs mountPath: /var/log/app `

Step through it

The lab below runs a Pod like this one: an init container, a native sidecar log shipper and the app, sharing a volume and localhost. Step forward and watch two separate things change: the Pod's phase and each container's state. Then break the init container and watch the Pod show Init:Error and restart attempts while the app never starts. The simulation compresses back-off delays.

Phase is a summary

The Pod phase is a high-level summary, not a full rollup of container state and not a state machine . The phases are Pending, Running, Succeeded, Failed and Unknown . Each container separately is Waiting, Running or Terminated . Values like "CrashLoopBackOff" and "Terminating" in kubectl's STATUS column are display values, not phases .

A realistic scenario

The node running web fails. The Pod is not moved. A Pod with a given UID is never moved to another node; a controller creates a near-identical replacement with a new UID, possibly the same name, possibly on a different node . The emptyDir` contents are gone, because anything tied to the Pod's lifetime is destroyed with it, even if an identical replacement is created . That is why bare Pods are rare: workload resources create them because Pods are disposable .

Phases, states and conditions

Three different status layers describe a Pod, and they are easy to mix up:

Layer — Values — Scope; Phase — Pending, Running, Succeeded, Failed, Unknown — Whole Pod, summary only; Container state — Waiting, Running, Terminated — Each container; Conditions — PodScheduled, PodReadyToStartContainers, Initialized, ContainersReady, Ready — Specific facts about the Pod

Running means the Pod is bound to a node and all its containers have been created, with at least one running, starting or restarting . Pending includes time waiting to be scheduled and pulling images . Ready means the Pod should be added to matching Services' load balancing . PodReadyToStartContainers was once called PodHasNetwork, and means the sandbox exists, networking is configured and volumes are mounted .

Restarts are not reschedules

Restarting a container is not restarting the Pod. A Pod is an environment, not a process, and it exists until it is deleted . restartPolicy can be Always, OnFailure or Never, defaults to Always, and applies to app containers and regular init containers; sidecars ignore it . Restarts back off exponentially (10 s, 20 s, 40 s and so on) up to 300 seconds, and the timer resets after 10 minutes of clean running .

A Pod stays on its node until it finishes, is deleted, is evicted for lack of resources, or the node fails . It is scheduled only once . If the chosen node crashes before the Pod starts, that Pod never starts .

Init and sidecar details

A failing init container is retried, unless restartPolicy is Never, in which case the Pod fails . Regular init containers do not support lifecycle, liveness, readiness or startup probes . On termination, sidecars stay up until the main containers have stopped, then shut down in reverse order of their position in the spec .

Graceful termination defaults to 30 seconds: the kubelet asks the runtime to send TERM (or the image's STOPSIGNAL), then KILL when the period ends .

Requests and limits

The scheduler uses requests to pick a node; the kubelet enforces limits and reserves at least the requested amount . A node is refused if the sum of requests would exceed capacity, even when actual usage is low . CPU limits are enforced by throttling; memory limits by OOM kills, but only when the kernel detects memory pressure, so a container can go over its memory limit for a while . If you set only a limit and nothing defaults a request, the limit is copied as the request .

Units bite. 0.1 CPU equals 100m , but a lowercase m on memory means millibytes, so 400m of memory is 0.4 bytes . Write memory as Mi or Gi.

Investigate it

``bash # Node, Pod IP and display status kubectl get pod web -o wide

# Phase, conditions, per-container states and events kubectl describe pod web kubectl get pod web -o jsonpath='{.status.phase}{"\n"}' kubectl get pod web -o jsonpath='{range .status.conditions[*]}{.type}={.status}{"\n"}{end}'

# Logs from a specific container, and from its previous run kubectl logs web -c log-shipper kubectl logs web -c app --previous

# Events for this Pod only kubectl get events --field-selector involvedObject.name=web ``

7. kubectl and the API server: what happens after kubectl apply

Part II follows one change through the cluster. It starts with a familiar command, kubectl apply -f web.yaml, and ends with Pods running on nodes. This chapter walks the whole chain at overview level. Chapters 8 to 12 then zoom in on each stage: the request itself, etcd, the controllers, the scheduler and the kubelet.

The API server is the core of the control plane. Its HTTP API is how end users, the parts of the cluster and external components talk to one another . kubectl and tools such as kubeadm do their work by calling that API ; kubectl turns your YAML manifest into a JSON API request . All API traffic from nodes and their Pods ends at the API server, in a hub-and-spoke pattern .

``yaml apiVersion: apps/v1 kind: Deployment metadata: name: web spec: replicas: 3 selector: matchLabels: app: web template: metadata: labels: app: web spec: containers: - name: web image: nginx:1.27 resources: requests: cpu: 250m memory: 128Mi `

One command, many independent actors

Kubernetes is not a "do A, then B, then C" workflow. It is a set of independent control processes, each pushing current state towards desired state . After kubectl apply:

• The API server accepts the object. It authenticates, authorizes and admits the request, validates the object and writes it to etcd .
• The Deployment controller sees the new Deployment and creates a ReplicaSet that it owns .
• The ReplicaSet controller creates Pods from the template until the count matches replicas . Each new Pod is Pending: accepted, but not yet scheduled .
• The scheduler watches for Pods with no node, picks one for each and records the choice through the API server; this is binding .
• The kubelet on the chosen node creates a Pod sandbox and networking through the container runtime, then pulls images and starts containers .
• Readiness. When the Pod is Ready it should be added to the load balancing of matching Services , and the EndpointSlice controller updates the Service's EndpointSlices .
• Rollout complete. Once every replica is updated and available, the Deployment's Progressing condition reads NewReplicaSetAvailable .

None of these components calls the next. Each one watches the API for the objects it cares about, and every write goes back through the API server.

Try it below: change replicas, press Deploy, and step through the playback. The event log, the object inspector and the node resource bars show each actor's step as it happens. The simulation compresses time and runs one controller step per tick.

A realistic scenario

You apply the manifest above and kubectl prints a confirmation such as deployment.apps/web created (illustrative output) almost at once. A moment later kubectl get pods shows three Pods, all Pending. That is not an error: the API server has done its part, and the scheduler and kubelets have not finished theirs. If the Pods stay Pending, the cause lies further down the chain, and kubectl describe pod` shows the scheduler's reasons .

Resources, verbs and groups

The Kubernetes API is a resource-based HTTP interface using POST, PUT, PATCH, DELETE and GET . It has its own verb names: a GET on a collection is a list, on one object a get, and with ?watch a watch . For authorization, POST is create, PUT is update, PATCH is patch, and DELETE is delete or deletecollection .

The core API group lives at /api/v1 and is written as apiVersion: v1; named groups live at /apis/GROUP/VERSION, which is why a Deployment says apps/v1 . Versions apply to the whole API, and the server converts between them because every version is a view of the same stored data . Built-in beta APIs are off by default, except those introduced before 1.22 .

Watch and resourceVersion

Every object carries a resourceVersion, and lists return one too, so a client can list and then watch from that point . A watch delivers every later change without gaps, and a dropped watch resumes from the last version it saw . The server keeps history for a limited window, by default the last 5 minutes with etcd 3 . Ask for older history and the watch fails with 410 Gone; the client must list again and re-watch, which client-go's Reflector does . Controllers read the API this way through informers .

From 1.35, conformance requires resource versions to be orderable as increasing integers within a type , yet the ObjectMeta reference still tells clients to treat them as opaque . Either way, pass the value back unmodified.

409 Conflict

On a PUT the client sends the resourceVersion it read. If the object has changed since, the server answers 409 Conflict: optimistic concurrency that prevents lost updates .

Client-side and server-side apply

Classic kubectl apply computes a patch on the client from the file, the live object and the last-applied-configuration annotation . Server-Side Apply, opted into with --server-side, is meant to replace it . The server tracks which field manager owns each field, and applying a different value to a field another manager owns is a conflict . kubectl's manager name is kubectl by default ; --force-conflicts takes the field over . Ownership is stored in metadata.managedFields, hidden unless you pass --show-managed-fields .

Investigate it

``bash kubectl api-resources --api-group=apps kubectl get --raw /apis/apps/v1 | head -c 400 kubectl get deployment web -o jsonpath='{.metadata.resourceVersion}{"\n"}' kubectl get pods -l app=web --watch kubectl diff -f web.yaml kubectl apply --server-side -f web.yaml kubectl get deployment web -o yaml --show-managed-fields | head -40 ``

Caveats

The lab runs the chain in strict order for clarity. In a real cluster the steps overlap: the scheduler may bind one Pod while the ReplicaSet controller is still creating the next, and the order of log lines varies.

8. The life of an API request

Every API request, whether from kubectl, a controller or a kubelet, passes through the same fixed stages: transport security, authentication, authorization, admission control, and then validation and storage . Each stage can stop the request, and each failure has a recognisable signature.

The stages

• TLS. By default the API server listens on port 6443 . With a private CA, the client needs that CA certificate in its kubeconfig, and it may present a client certificate here .
• Authentication: who are you? Authenticators are tried in turn until one succeeds; if none does, the answer is HTTP 401 . Kubernetes stores no users at all. The username comes from the authenticator , whether that is a client certificate, a bearer token or an authenticating proxy .
• Authorization: may you do this? Authorization denies by default. The first authorizer that allows or denies decides, and if all have no opinion the request is denied with HTTP 403 .
• Mutating admission. Admission runs only for create, update, delete and connect; reads skip it . Mutating controllers run first and may change the object .
• Validation. The object is checked by its type's validation routines .
• Validating admission. Validating controllers can only accept or reject, and any rejection fails the request at once .
• Persistence. The object is written to etcd .

Pick a scenario below and send it. The lab highlights the stage that decides the outcome: a bad token stops at authentication, a missing permission at authorization, a negative replica count at validation, an exhausted quota in admission, and a server-side dry run passes every check but writes nothing. The responses are simulated.

401 versus anonymous

An invalid bearer token gets 401. A request with no credentials is different: it becomes system:anonymous in the group system:unauthenticated, and authorization then judges it . So an unauthenticated request often fails with 403, not 401.

A realistic scenario

A CI job that deployed yesterday now fails with Forbidden. The token still authenticates, or the error would be 401. Something changed in authorization, such as a removed RoleBinding. kubectl auth can-i asks the authorization layer directly whether an action is allowed , and --as lets an administrator check on the job's behalf .

The docs we checked do not show the exact text of a modern 403 message for a denied resource request; they only say the result is HTTP 403 Forbidden .

Authorization modes

The modes are AlwaysAllow, AlwaysDeny, ABAC, Webhook, RBAC and Node. With neither --authorization-mode nor --authorization-config, kube-apiserver defaults to AlwaysAllow; installers normally set Node,RBAC . Node authorization gives kubelets permissions based on the Pods scheduled to them . RBAC has no deny rules, so AlwaysAllow,RBAC behaves exactly like AlwaysAllow . A Webhook authorizer blocks the request while it makes a synchronous HTTP call .

Admission in detail

Mutating webhooks are called one after another, and each may modify the object . Validating webhooks are called in parallel, and any rejection fails the request . They run only after all mutation is done and the object has been validated . A webhook's timeout defaults to 10 seconds, settable from 1 to 30 . failurePolicy defaults to Fail: if the webhook is unreachable, errors or times out, the request is rejected . A broken webhook can therefore block every write it matches.

ValidatingAdmissionPolicy is the in-process alternative: rules written in CEL, which take effect only through a ValidatingAdmissionPolicyBinding whose actions are Deny, Warn or Audit . The admission page still describes a feature gate and a v1alpha1 API for it, but that was only true in the alpha era, in earlier versions .

Built-in controllers matter too. NamespaceLifecycle stops new objects in a terminating namespace . ResourceQuota enforces quota, and because incrementing usage is a side effect it needs a matching reconciliation process .

The list of default admission plugins differs between the admission controllers page and the kube-apiserver flag help . For a specific build, read the --enable-admission-plugins help from kube-apiserver -h.

Dry run

?dryRun=All runs admission, validation and merge-conflict checks, stops before storage and returns the object that would have been saved . Dry-run requests are guaranteed not to be stored or have side effects, and are authorized exactly like the real request . kubectl apply --dry-run=server therefore catches errors the published OpenAPI schemas alone may miss .

Investigate it

``bash kubectl auth can-i create deployments --namespace default kubectl auth can-i --list --namespace default kubectl auth can-i delete deployments --as=system:serviceaccount:ci:ci-bot -n default kubectl apply --dry-run=server -f web.yaml kubectl get mutatingwebhookconfigurations,validatingwebhookconfigurations kubectl get validatingadmissionpolicies,validatingadmissionpolicybindings kubectl get pods -v=8 2>&1 | grep -E 'GET|Response Status' ``

Caveats

Which authenticators, authorizers and webhooks a cluster runs depends on how it was installed; managed services configure much of this for you. The lab shows one fixed chain.

9. etcd and desired state

Every chapter so far has said that objects are "stored". This is where they go. etcd is a consistent, highly available key-value store that holds all of Kubernetes' cluster data . Kubernetes stores the serialized state of its objects there , and access to etcd amounts to root access to the cluster, so ideally only the API server can reach it .

No docs page we read states word for word that the API server is the only component that talks to etcd. The docs give it as a security recommendation and describe the hub-and-spoke pattern .

Keys

kube-apiserver stores every resource under a key prefix that defaults to /registry . The Secret secret1 in namespace default lives at /registry/secrets/default/secret1 .

Leaders, followers and quorum

etcd is leader-based: the leader must send heartbeats to all followers on time to keep the cluster stable . The leader handles every request that needs consensus, and followers forward such requests to it . A majority must agree on every update. Quorum for n members is (n/2)+1, using integer division .

Members — Quorum — Failures tolerated; 1 — 1 — 0; 3 — 2 — 1; 4 — 3 — 1; 5 — 3 — 2; 7 — 4 — 3

The failure counts come from etcd's own table . A fourth member adds no fault tolerance, and an odd count guarantees a majority side exists during a network partition , which is why etcd should run with an odd number of members . For production the Kubernetes docs strongly recommend five members; a single member is for testing only .

Stop members one at a time below and watch the quorum line. With three members, stopping one leaves writes working; stopping two leaves the cluster unable to accept writes, while workloads already running are untouched. The lab models quorum only, not Raft's timing.

When quorum is lost

With no elected leader, Kubernetes cannot change cluster state and no new Pods can be scheduled . Running Pods may keep running : the kubelets already have their PodSpecs. If quorum was lost to a transient network failure, etcd resumes safely once the network recovers . If most members fail permanently, the cluster has failed, and you restore from a backup .

A realistic scenario

A three-node control plane with stacked etcd loses one node to a hardware fault. Writes continue, because two of three is still a majority. While that node is down, a second failure would stop all changes. Adding members improves availability at the cost of performance, never capacity, and etcd should not be autoscaled .

Versions and compaction

etcd keeps multiple versions of each key: a write keeps the previous version . That history must be compacted, and kube-apiserver asks etcd to compact every 5 minutes by default (--etcd-compaction-interval) . This is also why the API's watch window is limited . Without compaction etcd runs out of space, raises a space quota alarm and refuses writes with mvcc: database space exceeded until history is compacted, defragmented and the alarm disarmed . Defragmentation is expensive, so run it rarely but often enough that no member exceeds its quota .

The default etcd storage quota (--quota-backend-bytes) is not stated on any page read .

Performance

etcd is sensitive to network and disk I/O, and starvation can cause heartbeat timeouts . Larger clusters write more slowly because each write is copied to more machines; five members, tolerating two failures, is usually enough, and seven is a sensible ceiling . Run production etcd on dedicated machines or isolated environments, on 3.4.29+ or 3.5.11+ .

Secrets are not encrypted by default

By default the API server writes resources to etcd in plain text , and Secrets are stored unencrypted, readable by anyone with etcd access . Encryption at rest is enabled with --encryption-provider-config . The first provider in the EncryptionConfiguration encrypts new writes; on reads each matching provider is tried in order; the default identity provider gives no confidentiality . Enabling encryption does not rewrite existing data, so each Secret must be rewritten; encrypted values start with a prefix such as k8s:enc:aescbc:v1: .

Backups

Take a snapshot from a live member with etcdctl snapshot save, and encrypt the file, because it holds all cluster state . Never restore while API servers are running: stop them, restore every member, restart them, and preferably restart the scheduler, controller manager and kubelets too .

Investigate it

On a kubeadm control-plane node the etcd certificates usually sit under /etc/kubernetes/pki/etcd; other installers differ, and managed services typically give you no etcd access.

``bash export ETCDCTL_API=3 FLAGS="--endpoints=https://127.0.0.1:2379 --cacert=/etc/kubernetes/pki/etcd/ca.crt --cert=/etc/kubernetes/pki/etcd/server.crt --key=/etc/kubernetes/pki/etcd/server.key" etcdctl $FLAGS member list --write-out=table etcdctl $FLAGS endpoint status --cluster --write-out=table etcdctl $FLAGS get /registry/secrets/default/secret1 --print-value-only | hexdump -C | head etcdctl $FLAGS snapshot save /var/backups/etcd-snapshot.db ``

Caveats

Recovery time after losing quorum depends on the failure, the hardware and the installer; the docs give no figure, and neither does this chapter.

10. Controllers and reconciliation

A control loop is a loop that never ends and keeps regulating the state of a system . The docs' analogy is a thermostat: the temperature you set is the desired state, the room temperature is the current state, and the thermostat switches equipment on or off to close the gap .

Kubernetes controllers are control loops. Each watches cluster state and makes or requests changes to move the current state towards the desired state . A controller tracks at least one resource type, whose spec is the desired state ; components write the current state into status .

Controllers act through the API

A controller usually works by sending requests to the API server rather than acting directly . The Job controller never runs a Pod itself: it asks the API server to create or delete Pods, and the scheduler and kubelet act on them . Controllers also write back to the objects that configure them, as when the Job controller marks a Job finished . Some controllers change things outside the cluster, then report current state back .

This is the chain from chapter 7 seen from the inside. The Deployment controller owns ReplicaSets . A ReplicaSet reaches its count by creating and deleting Pods from its template , and finds its Pods through their ownerReferences .

Many small loops

Kubernetes prefers many simple controllers, each managing one part of the state, over one monolithic set of loops, and the design allows for controllers failing . Logically each built-in controller is its own process; to reduce complexity they are compiled into one binary, kube-controller-manager, and run in one process . Kubernetes expects constant change, and the cluster may never settle. That is fine as long as the controllers keep making useful changes .

The lab starts with a desired count of 5 and only 3 Pods observed. Watch it converge, then delete a Pod, change replicas, or pause the simulated controller manager and see that nothing moves until you resume. The loop runs on a simulated clock.

A realistic scenario

Someone runs kubectl delete pod web-7d4b9c-abcde on a Deployment's Pod. Within moments a new Pod appears with a different name. The ReplicaSet controller saw fewer Pods than desired and created a replacement. It is a new object with a new UID, not the old Pod coming back . Deleting Pods by hand does not shrink a Deployment; changing replicas does.

A ReplicaSet also adopts any Pod that matches its selector and has no controller owner . A stray Pod with the right labels counts towards its replicas, and the ReplicaSet may delete one of its own Pods to compensate.

kube-controller-manager

Its controllers watch shared state through the API server, and --controllers defaults to `, every on-by-default controller . Examples include the Node, Job, EndpointSlice and ServiceAccount controllers . Each has tuning knobs: --concurrent-replicaset-syncs defaults to 5 .

Leader election

Leader election is on by default; a replica must win leadership before running its main loop . The lock is a Lease named kube-controller-manager in kube-system , and Leases ensure only one instance is active while the others stand by . The lease duration defaults to 15 s, effectively the longest a stopped leader can go before another takes over ; the renew deadline is 10 s and the retry period 2 s . A feature gate lets the controller manager release its lock on exit instead of waiting out the lease ; the HA docs describe it as alpha in 1.36 and off by default . So in a resilient control plane, if a controller fails, another part of the control plane takes over . The lab's pause button stands for the case where no replica holds the lease.

Owner references and garbage collection

Owner references tell the control plane which objects depend on which, and cannot cross namespaces . An object is garbage collected once every owner has been deleted, and it can have at most one managing controller .

--cascade — What happens; background (default) — The owner goes at once; the garbage collector removes dependents afterwards; foreground — The owner gets a deletionTimestamp and the foregroundDeletion finalizer and stays visible until dependents with blockOwnerDeletion=true are gone; orphan — Dependents are left in place

generation and observedGeneration

metadata.generation is a system-set sequence number for the spec . A Deployment's status.observedGeneration is the generation its controller last observed . If it lags generation, the status describes an older spec; the docs read do not spell this rule out . The status counts replicas, updatedReplicas, readyReplicas and availableReplicas .

Operators and CRDs

A CustomResourceDefinition adds a resource the API server serves and stores like a built-in . On its own it only stores data; paired with a controller it becomes a declarative API . An operator is that controller, an ordinary API client following the control loop principle .

The controller and API concepts pages do not call controllers "level-triggered" or "edge-triggered". That label is a community description, not something these docs state .

Investigate it

`bash kubectl get lease kube-controller-manager -n kube-system -o yaml kubectl get rs -l app=web -o jsonpath='{.items[].metadata.ownerReferences[*].name}{"\n"}' kubectl get deployment web -o jsonpath='{.metadata.generation} {.status.observedGeneration}{"\n"}' kubectl delete pod -l app=web --wait=false && kubectl get pods -l app=web --watch kubectl delete deployment web --cascade=orphan kubectl get crd ``

11. Scheduler internals

kube-scheduler watches for newly created Pods that have no node assigned, and for each one it finds it becomes responsible for choosing the best node . It is the default scheduler, and it is designed so you can replace it . The scheduler does not create Pods; the ReplicaSet controller already did that. It does not start containers either. It tells the API server its decision, a step called binding, and the kubelet on the chosen node runs the Pod .

Filter, score, bind

Node selection has two steps. Filtering finds the feasible nodes, for example by checking free resources against the Pod's requests. Scoring ranks the nodes that survived . The Pod goes to the highest-scoring node, and ties are broken at random . If no node is feasible, the Pod stays Pending until the scheduler can place it; nothing fails permanently .

Binding is recorded through a Binding object, and the result shows up in the Pod's spec.nodeName; from then on that node's kubelet owns the Pod's lifecycle . A Pod is scheduled once. It stays on its node until it stops or is terminated, and is never moved to another node .

Requests, not usage

The scheduler checks the sum of requests, not actual usage, and limits play no part in the check . A node with very low real CPU use still rejects a Pod whose requests do not fit, which protects against later spikes . Containers started directly by the runtime and processes outside the kubelet are not counted .

Configure the three nodes and the Pod below, then schedule it. Each node shows whether it passed filtering and, if not, why; survivors get a score and the winner is bound. The scores are illustrative, not the real plugin arithmetic. Try the failure presets to see a Pod stay Pending with a message.

A realistic scenario

A team deploys a Pod requesting 2 CPU to a three-node cluster whose dashboards show the nodes nearly idle. The Pod stays Pending. The existing Pods' requests already fill each node's allocatable CPU, so filtering rejects every node. When it cannot place a Pod, the scheduler emits a FailedScheduling Warning event each time it fails, and kubectl describe pod shows it . The docs' example reads "0/42 nodes available: insufficient cpu" . The fix is lower requests, fewer Pods, or more nodes, and adding nodes is the job of a separate node autoscaler or a person, not the scheduler .

The exact wording of modern FailedScheduling messages, such as "0/3 nodes are available: 3 Insufficient cpu.", is not shown on the debugging pages checked .

The scheduling framework

Each attempt has a scheduling cycle, which picks a node, and a binding cycle, which applies the choice. Scheduling cycles run one at a time; binding cycles can run in parallel . Either can be aborted, sending the Pod back to the queue .

Extension point — Role; QueueSort — Orders the single pending queue; default PrioritySort; Filter — Per node, in order; the first plugin to reject ends checks for that node; PostFilter — Only when nothing is feasible; typically preemption; Score, NormalizeScore — Rank, rescale, combine by plugin weight; Reserve / Unreserve — Claim the node's resources before binding; undo on failure; Permit — Approve, deny or wait with a timeout; PreBind, Bind, PostBind — Pre-work such as volumes, then bind; PostBind is informational

Default plugins include NodeResourcesFit, NodeAffinity, TaintToleration, PodTopologySpread, InterPodAffinity, NodeUnschedulable, VolumeBinding and DefaultBinder . NodeUnschedulable filters out cordoned nodes .

NodeResourcesFit scoring

NodeResourcesFit checks the node has the requested resources. Its default scoring strategy, LeastAllocated, favours emptier nodes, with CPU and memory weighted equally . MostAllocated and RequestedToCapacityRatio support bin packing instead . Real scores combine many plugins, so the lab's numbers are only a sketch.

Large clusters

percentageOfNodesToScore lets the scheduler stop searching once it has enough feasible nodes, trading placement quality for speed . Unset, it follows a linear formula: 50% at 100 nodes, 10% at 5,000, with a 5% floor . With fewer than 100 feasible nodes, every node is checked . The node list is walked round-robin, interleaving zones , and algorithms run with a parallelism of 16 by default .

Retries

Unschedulable Pods are retried with backoff: 1 s initially, 10 s at most by default . If a Pod sets no schedulerName, the API server sets default-scheduler . Scheduling is recorded as the PodScheduled condition . Like the controller manager, kube-scheduler uses a Lease for leader election in HA setups .

The scheduler concept page still mentions Scheduling Policies (Predicates and Priorities) alongside profiles; the configuration reference describes only profiles and plugins .

Investigate it

``bash kubectl get pods -o wide kubectl describe pod web-7d4b9c-abcde | sed -n '/Conditions:/,$p' kubectl get events --field-selector reason=FailedScheduling kubectl describe node worker-1 | grep -A 10 'Allocated resources' kubectl get pod web-7d4b9c-abcde -o jsonpath='{.spec.nodeName} {.spec.schedulerName}{"\n"}' kubectl get lease kube-scheduler -n kube-system ``

12. Kubelet and the container runtime

The scheduler's work ends with a node name on a Pod. Here the Pod becomes processes. The kubelet is the primary node agent and runs on every node . It works from PodSpecs, mostly received from the API server, and makes sure the containers they describe are running and healthy; it ignores containers Kubernetes did not create .

Talking to the runtime

The kubelet does not run containers itself. It talks to the container runtime over the Container Runtime Interface (CRI), a gRPC protocol: the kubelet is the client, and the runtime serves a runtime service and an image service . Because CRI is a plugin interface, runtimes can be swapped without recompiling Kubernetes ; containerd and CRI-O are common choices . By default the kubelet connects to containerd's socket . Since 1.26 it requires the v1 CRI API and will not register the node otherwise .

Starting a Pod

• Sandbox and network. The kubelet asks the runtime to create a Pod sandbox and set up its networking; when that is done, PodReadyToStartContainers is set . The runtime, not the kubelet, loads and calls the CNI plugins that give the Pod its IP .
• Pull. Images are pulled according to imagePullPolicy .
• Create and start. The kubelet passes each container's requests and limits to the runtime, which on Linux usually turns them into cgroup settings . A healthy start shows the events Scheduled, Pulling, Pulled, Created and Started .
• Report. The kubelet writes Pod status back to the API server and keeps the node's heartbeat going (see the deep dive).

When it goes wrong

If a container keeps exiting, the kubelet restarts it with exponential back-off, 10 s, 20 s, 40 s and so on, capped at 300 s, and the timer resets after 10 minutes of clean running . kubectl shows CrashLoopBackOff, a display value, not a Pod phase . If the image cannot be pulled, the status is ImagePullBackOff and retries continue with a growing delay, capped at 300 s .

The lab runs one node's kubelet with two broken workloads: an image that crashes on start and an image that does not exist. Watch the restart count and the growing back-off gaps on the simulated clock, compressed so the 300 s cap arrives quickly.

A realistic scenario

A release goes out with the tag v2.1 instead of v2.10. The Pods are scheduled, but they never start. kubectl get pods shows ImagePullBackOff, and kubectl describe pod shows the pull failure. A Pod stuck in Waiting has been scheduled but cannot run, most often because of an image pull failure . The fix is to correct the tag; the kubelet keeps retrying on its own.

Image pull policy

With IfNotPresent, an image is pulled only if it is not on the node; with Always, the kubelet asks the runtime to pull every time it starts a container, and cached layers are reused; with Never, it never pulls . If omitted, the policy is Always for :latest or untagged images and IfNotPresent otherwise, set once at creation . By default the kubelet pulls one image at a time (serializeImagePulls: true), and it never pulls one Pod's images in parallel . One slow pull can therefore delay unrelated Pods on the same node.

Static Pods

The kubelet can also read Pod manifests from a local path or an HTTP endpoint, checked every 20 s by default . Static Pods run from these manifests are not controlled by the API server, and kubeadm uses them to run the control plane . The kubelet creates a read-only mirror Pod for each; deleting the mirror does not stop the static Pod . It reads every file in staticPodPath not starting with a dot, so a stray backup copy can override the real manifest .

Lifecycle hooks

postStart runs right after a container is created, concurrently with its entrypoint, with no ordering guarantee . preStop must finish before TERM is sent, and its time counts against the grace period; if either hook fails, the container is killed . Delivery is at-least-once, so handlers should be idempotent .

Back-off variations

On current versions, a node can lower the 300 s cap with crashLoopBackOff.maxContainerRestartPeriod , and an alpha gate shortens the whole series . The probes page's summary says the first crash restarts immediately, which is less precise than the 10 s-first series .

Status reporting

The kubelet posts node status every 10 s by default and also renews a per-node Lease in kube-node-lease , every 10 s by default . It resyncs running containers at least every minute . At most 110 Pods run per kubelet by default .

In earlier versions the kubelet could manage CNI itself through --network-plugin and --cni-bin-dir; those flags were removed in 1.24 . Docker Engine's built-in dockershim was removed in the same release .

Investigate it

``bash kubectl describe pod web-7d4b9c-abcde | sed -n '/Containers:/,/Conditions:/p' kubectl get pod web-7d4b9c-abcde -o jsonpath='{.status.containerStatuses[*].restartCount}{"\n"}' kubectl logs web-7d4b9c-abcde --previous kubectl get pods -n kube-system -o wide # on the node itself sudo crictl pods sudo crictl ps -a sudo crictl images journalctl -u kubelet --since "10 min ago" ``

Caveats

Sandbox setup, image pulls and networking are done by the runtime and CNI plugin you installed, and their behaviour and timing vary between implementations.

13. Deployments and ReplicaSets

A Deployment is declarative: you state how many copies of a Pod you want and what they should look like, and the Deployment controller moves the actual state towards that at a controlled rate, using ReplicaSets that it owns . It is the object most stateless applications are run with, and the docs recommend it over using a bare ReplicaSet unless you need custom update orchestration or no updates at all .

The ownership chain

There are three layers. The Deployment owns one or more ReplicaSets, and each ReplicaSet owns Pods. A ReplicaSet has one job: keep a fixed number of identical Pods running, creating or deleting Pods from its template until the count matches . It tracks its Pods through each Pod's metadata.ownerReferences . Those references are also how cleanup works: an object is garbage collected once every owner it lists has been deleted .

To keep ReplicaSets from claiming each other's Pods, the Deployment controller adds a pod-template-hash label to every ReplicaSet it creates or adopts. The value is a hash of the Pod template, and it is added to the ReplicaSet's selector, its template labels and its existing Pods .

Scaling is not a rollout

Two kinds of change look similar in YAML but behave very differently. A rollout, meaning a new ReplicaSet and a new revision, starts only when .spec.template changes. Changing .spec.replicas does not start a rollout and does not create a revision . The two are kept apart on purpose, so that manual scaling and autoscaling can happen during or between rollouts . If you leave .spec.replicas out it defaults to 1 .

In the lab below, scale a Deployment from 2 to 8 replicas. Watch which objects change: the same ReplicaSet raises its count, only new Pods are created, the existing two are untouched, and the replicas, ready and available counts climb at different moments. The simulation compresses time, and the spreading you see is a simplified model of the scheduler.

A realistic scenario

A checkout service runs 2 replicas and a sale is about to start. An operator runs kubectl scale deployment checkout --replicas=8. The Deployment's desired count changes, the ReplicaSet controller notices it owns 2 Pods but wants 8 and creates 6 new ones, and the scheduler places each of them. Unless the cluster defines its own defaults, kube-scheduler applies a built-in soft spreading preference across nodes and zones , so the new Pods tend to land on different nodes, though that is a preference rather than a guarantee. Nothing about the Pod template changed, so no revision is recorded and kubectl rollout history looks exactly as it did before.

The ReplicaSet creates the Pods; the scheduler only chooses a node for each one. If no node has room, the new Pods stay Pending, and the Deployment simply reports fewer available replicas than it wants.

How a ReplicaSet counts

A ReplicaSet adopts at once any Pod that matches its selector and has no controller owner, so a stray Pod with matching labels is counted towards its replicas . In apps/v1 a Deployment's selector cannot be changed after creation, and if two controllers have overlapping selectors they fight over the same Pods; Kubernetes does not stop you setting that up . The docs also warn against managing a Deployment's ReplicaSets by hand .

When a ReplicaSet scales down it chooses victims in a fixed order: Pending or unschedulable Pods first, then Pods with a lower pod-deletion-cost, then Pods on nodes running more replicas, then newer Pods before older ones (creation times grouped on a log scale), and at random if all else is equal . Spreading is only enforced at scheduling time, so scaling down can leave Pods unevenly placed; the docs point to the Descheduler for rebalancing .

Status fields

Field — Counts; replicas — Non-terminating Pods that match the selector; updatedReplicas — Non-terminating Pods with the current template; readyReplicas — Non-terminating Pods with a Ready condition; availableReplicas — Ready Pods that have stayed ready for at least minReadySeconds; unavailableReplicas — Pods still needed for 100% available capacity, including ones not yet created

minReadySeconds defaults to 0, so a Pod is available as soon as it is Ready . Terminating Pods are counted separately in terminatingReplicas, behind a feature gate, and while they shut down the total number of Pods can briefly exceed .spec.replicas . If Pod creation fails, for example on an exceeded quota, the Deployment shows a ReplicaFailure condition with reason FailedCreate ; the Deployment itself still exists, which is why the docs say to check its status .

Revision history

Revision history lives in the old ReplicaSets, which are scaled to 0 rather than deleted. revisionHistoryLimit keeps 10 by default; with 0, old ReplicaSets are cleaned up and a rollout cannot be undone . The CHANGE-CAUSE column of kubectl rollout history comes from the kubernetes.io/change-cause annotation .

Proportional scaling

If an autoscaler or a user scales a RollingUpdate Deployment mid-rollout, the extra replicas are spread across all active ReplicaSets in proportion : larger ReplicaSets get a larger share, leftovers go to the largest, and ReplicaSets at zero are not scaled up . The documented example has an old ReplicaSet at 8 and a new one at 5; scaling to 15 adds 3 to the old one and 2 to the new one .

Investigate it

``bash kubectl scale deployment web --replicas=8 kubectl get deployment web -o wide kubectl get replicasets -l app=web --show-labels kubectl get pods -l app=web -o wide kubectl get pod <pod-name> -o jsonpath='{.metadata.ownerReferences[0].name}' kubectl rollout history deployment web `

Deleting a Deployment uses background cascading deletion by default: the owner goes at once and the garbage collector removes ReplicaSets and Pods afterwards . kubectl delete --cascade=orphan` leaves them in place instead .

14. Service discovery and networking

Kubernetes networking starts with a simple model. Every Pod gets its own IP address, unique across the cluster , and all containers in a Pod share one network namespace, so they can talk over localhost . Pods can reach every other Pod, on the same node or another, directly, with no proxy and no NAT, unless someone has segmented the network on purpose .

Kubernetes defines this model but does not build the network itself. The container runtime sets up each Pod's network namespace, and a separate implementation (a CNI plugin on Linux) provides the actual network, so behaviour depends on which plugin is installed . The runtime, not the kubelet, loads and calls the CNI plugins .

Why Services exist

Pod IPs are not stable. A Deployment creates and destroys Pods, so their addresses change; a Service gives clients a stable IP or DNS name for a changing set of backends . It usually picks those backends with a label selector, and its controller keeps the Service's EndpointSlices up to date as matching Pods come and go .

Type — What it gives you; ClusterIP (default) — An IP reachable only from inside the cluster; NodePort — A cluster IP plus the same port on every node; LoadBalancer — Usually a node port plus an external load balancer, which needs a cloud provider or one you supply; ExternalName — A DNS CNAME to an outside hostname, with no proxying; Headless (clusterIP: None) — No cluster IP and no load balancing; clients get Pod IPs through DNS

No central proxy

A cluster IP does not belong to any host. It is a virtual IP that packet rules on each node redirect . kube-proxy, unless it has been replaced, runs on every node and watches Services and EndpointSlices . It programs that node's kernel to catch traffic to a Service's IP and port and redirect it to an endpoint, and a control loop keeps the rules in sync . Traffic therefore never flows through a central box: the decision is made on the node where the client Pod runs.

In the lab, a client Pod calls the Service web. Toggle a backend's readiness and watch it leave the EndpointSlice, then compare a same-node and a cross-node path and switch between the dataplane descriptions. The simulation simplifies the rules each dataplane actually installs.

A realistic scenario

One of three web Pods starts failing its readiness probe after a bad cache warm-up. The kubelet keeps the container running and marks the Pod not Ready ; its IP is removed from the EndpointSlices of every matching Service , and because only ready endpoints get traffic , every node's rules stop sending requests to it. Nobody edits the Service, and no Pod is restarted.

EndpointSlices

The control plane creates EndpointSlices for every Service with a selector , at most 100 endpoints per slice by default , and kube-proxy treats them as its source of truth . Each endpoint has three conditions: serving follows the Pod's Ready condition, terminating is set once the Pod has a deletion timestamp, and ready means serving and not terminating . Proxies skip terminating endpoints, but if every endpoint is terminating they may still use ones that are serving, so rolling updates do not drop traffic . The older Endpoints API is deprecated in favour of EndpointSlice .

Dataplanes

On Linux kube-proxy has three modes: iptables, ipvs and nftables. In 1.37 the default is iptables, a future version will make nftables the default, and the docs advise setting the mode explicitly . In iptables mode, per-Service chains DNAT to a randomly chosen endpoint by default ; very large clusters can accumulate tens of thousands of rules and slow updates . nftables needs kernel 5.13 or later and handles changes and packets more efficiently . ipvs mode is deprecated, to be disabled by default from v1.40 . Some network implementations replace kube-proxy entirely with their own proxy , so what a node actually does varies by plugin.

The docs explain why Kubernetes uses virtual IPs rather than round-robin DNS: many DNS clients ignore TTLs and cache results too long . Other routing knobs: internalTrafficPolicy: Local drops traffic on a node with no local endpoint rather than falling back ; trafficDistribution: PreferSameZone prefers local endpoints with cluster-wide fallback ; sessionAffinity: ClientIP pins clients, with a 3-hour default timeout .

NetworkPolicy

NetworkPolicy controls layer 3/4 traffic, and the network plugin enforces it, so the plugin must support it . Without such a plugin a policy does nothing, yet the API still accepts it . By default Pods are not isolated ; once a policy selecting a Pod includes Ingress, only allowed connections get in . Policies are additive, never conflicting: the result is the union of all that apply , and a connection needs both the source's egress and the destination's ingress to allow it . This is a namespace-wide ingress default deny :

``yaml apiVersion: networking.k8s.io/v1 kind: NetworkPolicy metadata: name: default-deny-ingress spec: podSelector: {} policyTypes: - Ingress `

An egress default deny also blocks DNS, so name resolution must be allowed explicitly . A frequent YAML mistake: namespaceSelector and podSelector in one from entry are ANDed, in two list items ORed . There is no explicit deny rule, TLS or logging , and whether a change affects open connections depends on the implementation .

Investigate it

`bash kubectl get pods -l app=web -o wide kubectl get service web -o yaml kubectl get endpointslices -l kubernetes.io/service-name=web -o yaml kubectl get networkpolicy -A kubectl -n kube-system get pods -o wide ``

Pod, Service and node IPs come from separate ranges, assigned by the network plugin, kube-apiserver and the kubelet or cloud-controller-manager respectively .

15. CoreDNS and Service names

Services give stable IPs, but nobody wants to hard-code IPs. Kubernetes creates DNS records for Services and Pods, and the kubelet sets up each Pod's DNS so containers can find Services by name . Cluster DNS is technically an addon, but the docs say every cluster should run it . CoreDNS is the default implementation , and its Service is still called kube-dns for compatibility with workloads that expect the old name .

The records

A normal Service gets an A and/or AAAA record named my-svc.my-namespace.svc.cluster-domain.example, which resolves to its cluster IP . A headless Service's name resolves instead to the IPs of all its selected Pods, and clients use the whole set or round-robin through it . Because the namespace is part of the name, a short name such as data resolves only within the caller's own namespace; to reach another you add it, as in data.prod .

The search list and ndots

Short names work because of the resolver configuration. The kubelet writes each Pod's /etc/resolv.conf with a search list of <namespace>.svc.cluster.local, svc.cluster.local and cluster.local, and options ndots:5. Any name with fewer than five dots is therefore tried against the search domains first . For a Pod in default, a file like this (illustrative; the nameserver address depends on the cluster):

``text search default.svc.cluster.local svc.cluster.local cluster.local nameserver 10.96.0.10 options ndots:5 `

In the lab, resolve web, web.default, web.default.svc.cluster.local and web.shop from a Pod in default. You will see each candidate name the resolver builds from the search list, and whether CoreDNS answers with a cluster IP, a set of Pod IPs for a headless Service, or NXDOMAIN. The simulation shows one query at a time and does not model caching on the client.

A realistic scenario

A developer deploys an API in namespace default that calls orders, and it works. In production the orders Service lives in namespace shop, and the same code fails: orders expands to orders.default.svc.cluster.local, which does not exist. The fix is to call orders.shop, which the search list completes to orders.shop.svc.cluster.local. The flip side is useful: because short names stay inside the caller's namespace, the same configuration can be deployed unchanged to dev, staging and prod namespaces .

The ndots rule also applies to external names. api.example.com` has two dots, fewer than five, so it is tried against the cluster search domains before being looked up as written .

dnsPolicy

dnsPolicy — Behaviour; ClusterFirst (default) — Cluster DNS; names outside the cluster domain are forwarded upstream by the cluster DNS server; Default — Copies the node's resolver configuration; despite the name it is not the default; ClusterFirstWithHostNet — Needed for hostNetwork Pods; without it, ClusterFirst quietly behaves like Default; None — Ignores cluster settings; everything comes from the Pod's dnsConfig

The kubelet passes the DNS server address through --cluster-dns and the cluster domain through --cluster-domain , so cluster.local is a common setting, not a constant.

Inside CoreDNS

CoreDNS reads a Corefile from the coredns ConfigMap in kube-system. The default Corefile includes the kubernetes plugin with ttl 30, forward to /etc/resolv.conf, cache 30, loop, reload and loadbalance . The kubernetes plugin answers from Service and Pod IPs; on its own its TTL defaults to 5 seconds, but the default Corefile sets 30 . After you edit the ConfigMap, the reload plugin applies the change, and the docs say to allow two minutes . Readiness is served on port 8181 and health on 8080, with lameduck 5s keeping the process alive briefly after it reports unhealthy .

Other record shapes

Named Service ports get SRV records such as _http._tcp.web.default.svc.cluster.local, with one answer per Pod for a headless Service . A StatefulSet's governing headless Service gives each Pod a stable name of the form podname.service.namespace.svc.cluster.local . An ExternalName Service is just a CNAME .

Misconception: DNS does the load balancing

For a normal Service, DNS returns one cluster IP and the node's proxy rules choose the backend. Kubernetes deliberately avoids round-robin DNS here because many clients ignore TTLs and cache too long . Only headless Services hand Pod IPs to the client .

Investigate it

``bash kubectl exec -it <pod-name> -- cat /etc/resolv.conf kubectl exec -it <pod-name> -- nslookup web.default.svc.cluster.local kubectl -n kube-system get service kube-dns kubectl -n kube-system get configmap coredns -o yaml kubectl -n kube-system logs -l k8s-app=kube-dns `

When debugging a Service, first check that its full name <service>.<namespace>.svc.cluster.local resolves . The nslookup step needs an image that ships it; the k8s-app=kube-dns` label is the one typically used by CoreDNS deployments, but installers can differ.

16. Ingress, Gateway API and external traffic

A ClusterIP Service is reachable only from inside the cluster . Getting traffic in from the outside world takes one of a few building blocks, and they stack on each other.

Node ports and load balancers

The lowest layer is a NodePort Service: the control plane allocates a port, from 30000-32767 by default, and every node forwards that port to the Service . A LoadBalancer Service usually builds on that node port, and the cloud-controller-manager points an external load balancer at it . Kubernetes has no load balancer of its own, so this only works if you provide one or connect a cloud provider . It is the simplest, least configurable way in : one Service, one external address, no HTTP routing.

Ingress

An Ingress is an API object for HTTP and HTTPS access, offering load balancing, TLS termination and name-based virtual hosting . On its own it does nothing: an Ingress controller must be running, and controllers differ in behaviour and in the annotations they support . Ingress handles only HTTP and HTTPS; other protocols usually go through a NodePort or LoadBalancer Service .

The project now recommends Gateway over Ingress. Ingress is frozen: it is still GA and will not be removed, but it will not get new changes .

Gateway API

Gateway API is an add-on made of custom resources for infrastructure provisioning and advanced routing, not a built-in type the way Ingress is . Neither its CRDs nor a controller ship with a plain cluster; a controller often installs the CRDs for you . Its design splits ownership by role: infrastructure provider, cluster operator and application developer .

• GatewayClass: a set of Gateways managed by one controller .
• Gateway: an instance of traffic-handling infrastructure, such as a cloud load balancer or an in-cluster proxy , belonging to exactly one GatewayClass .
• HTTPRoute: maps requests from a Gateway listener to backends, usually Services, by host, path and headers .

The diagram traces these paths side by side. Start the journey to follow one request from a client through a Gateway and HTTPRoute to a Service and a Pod, then click the LoadBalancer, NodePort and Ingress controller paths to compare where each decision is made.

A realistic scenario

A team runs a shop with / served by web and /api by api. With only LoadBalancer Services they would need two external addresses. Instead a platform team installs a Gateway API controller and creates a Gateway, and the shop team attaches an HTTPRoute with two path rules in its own namespace. The controller turns that into configuration for whatever proxy or cloud load balancer it manages, and that part is entirely implementation-specific.

Ingress classes

Each Ingress should name an IngressClass, which names the controller that implements it and can point to extra parameters . If more than one IngressClass is marked as default, admission rejects new Ingresses that have no ingressClassName .

Gateway and HTTPRoute in YAML

``yaml apiVersion: gateway.networking.k8s.io/v1 kind: Gateway metadata: name: shop namespace: shop spec: gatewayClassName: example listeners: - name: http protocol: HTTP port: 80 --- apiVersion: gateway.networking.k8s.io/v1 kind: HTTPRoute metadata: name: shop-routes namespace: shop spec: parentRefs: - name: shop rules: - matches: - path: type: PathPrefix value: /api backendRefs: - name: api port: 80 - backendRefs: - name: web port: 80 `

example stands for whatever GatewayClass your controller provides. The Gateway sets no addresses, so the controller assigns one . By default a Gateway only accepts Routes from its own namespace; Routes from elsewhere need allowedRoutes . The implementation may send traffic to the Service IP or directly to its EndpointSlices .

Failure behaviour and variations

• No controller. An Ingress with no controller is accepted and ignored . A Gateway API object cannot even be created until the CRDs exist .
• Node health. With externalTrafficPolicy: Cluster, load balancer health checks hit kube-proxy on port 10256, which returns 503 while the node is being deleted . How fast a given cloud load balancer reacts is provider-specific.
• Deprecated paths. Service externalIPs are deprecated; the docs point to a load balancer controller or Gateway API .
• Mesh. The same route resources cover east-west (mesh) as well as north-south traffic .

Migration

Gateway API succeeds Ingress but does not include an Ingress kind, so moving existing Ingresses is a one-time conversion . Ingress will not be removed, so there is no deadline, but it is no longer being developed .

Investigate it

`bash kubectl get service -A --field-selector spec.type=LoadBalancer kubectl get ingressclass kubectl get ingress -A kubectl get crd | grep gateway.networking.k8s.io kubectl get gatewayclass,gateway,httproute -A kubectl describe httproute shop-routes -n shop ``

Annotations, TLS defaults, timeouts and header handling all vary between Ingress controllers and Gateway implementations. Check your controller's documentation rather than assuming another controller's behaviour.

17. Persistent storage

Containers are disposable, and so is their filesystem. When a container crashes or stops, the files it created or changed are lost and the kubelet restarts it clean . Anything that must outlive that needs a volume. Ephemeral volumes live as long as their Pod; persistent volumes outlive it; both keep data across container restarts .

emptyDir: scratch space for one Pod

An emptyDir is created when the Pod is assigned to a node and deleted permanently when the Pod leaves that node. A container crash does not remove the Pod, so the data survives crashes . It is tied to that exact Pod UID: an identical replacement Pod gets a fresh, empty one .

PV, PVC and StorageClass

Durable storage is split between two objects. A PersistentVolume (PV) is a piece of cluster storage, created by an admin or provisioned dynamically, with a lifecycle independent of any Pod; a PersistentVolumeClaim (PVC) is a user's request for storage . A control loop binds each new PVC to a matching PV, one-to-one, and a PVC with no match stays Pending until one appears .

Most clusters do not pre-create PVs. Dynamic provisioning creates storage on demand when a PVC is created , following a StorageClass, which names a provisioner, parameters and a reclaim policy. Kubernetes itself does not say what classes mean; "fast" or "standard" is up to the cluster .

``yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi `

With no storageClassName, this claim uses the cluster's default StorageClass . ReadWriteOnce` means a single node, not a single Pod .

In the lab, a Pod writes data to its container filesystem, an emptyDir, a zonal RWO PVC and an RWX PVC. Restart the container, delete the Pod, and send the replacement to the same node, another node in the same zone, or another zone, and see what survives and when the replacement stays Pending. The simulation treats attach and detach as instant, which real storage does not.

A realistic scenario

A single-replica database uses an RWO PVC backed by a zonal cloud disk. The PV carries node affinity for that zone, so Pods using it are scheduled only onto matching nodes . When its node fails, a replacement Pod can start on another node in the same zone. If the only nodes with spare capacity are in a different zone, the replacement stays Pending: the data is fine, but it cannot follow the Pod across zones.

Access modes

Mode — Meaning; ReadWriteOnce (RWO) — Read-write by a single node; several Pods on that node can use it; ReadOnlyMany (ROX) — Read-only by many nodes; ReadWriteMany (RWX) — Read-write by many nodes; ReadWriteOncePod (RWOP) — Only one Pod in the whole cluster; CSI volumes only

A volume is mounted with one access mode at a time, even if it supports several . Which modes are available depends on the storage backend and driver.

Binding modes and topology

volumeBindingMode defaults to Immediate: the PVC is bound and provisioned as soon as it is created . For topology-limited storage that can put the disk in the wrong zone, before anyone knows where the Pod will run, leaving the Pod unschedulable . WaitForFirstConsumer waits until a Pod uses the claim and then provisions to match that Pod's scheduling constraints .

Reclaim and protection

Dynamically provisioned PVs inherit their StorageClass's reclaim policy, which defaults to Delete, so deleting the PVC deletes the disk . With Retain, the PV becomes Released with its data and must be cleaned up by hand before reuse . Recycle is deprecated . Storage Object in Use Protection delays deleting a PVC while a Pod uses it, and a PV while it is bound, showing Terminating with a protection finalizer . PV phases are Available, Bound, Released and Failed .

CSI and other volume types

CSI is the standard interface that lets Kubernetes use any storage system; once a driver is installed, Pods use it through a PVC, a generic ephemeral volume or a CSI ephemeral volume . Drivers are out-of-tree, shipped by vendors rather than in the Kubernetes repository . For a generic ephemeral volume, a controller creates a real PVC and deletes it with the Pod . An emptyDir with medium: Memory is a tmpfs whose files count against the container's memory limit . StatefulSets go further, giving each Pod its own PVC that a rescheduled Pod mounts again .

Misconceptions

• "RWO means one Pod." It means one node ; use RWOP for a single Pod .
• "Unbound claims will fall back." A 100Gi claim will not bind to 50Gi PVs, and waits indefinitely .
• "Setting no class opts out." Omitting the class uses the default; you opt out with storageClassName: "" .

A common real-world failure is a "Multi-Attach" error while an RWO volume is still attached to the old node and the new Pod starts elsewhere. The Volumes, Persistent Volumes and Storage Classes pages do not describe it . What the docs do say is that when a Pod deletion has not succeeded within 6 minutes and the node is unhealthy, Kubernetes force-detaches the volumes .

Investigate it

``bash kubectl get storageclass kubectl get pvc -n <namespace> kubectl describe pvc data -n <namespace> kubectl get pv kubectl describe pv <pv-name> kubectl get volumeattachments `

kubectl describe pv` shows which PVC a volume is bound to . Attach times, zone handling and resize support all vary by CSI driver and cloud provider.

18. ConfigMaps, Secrets and Namespaces

Container images should not change between environments, so configuration is kept outside them. Kubernetes has two objects for that, ConfigMaps and Secrets, and uses Namespaces to group and scope everything.

ConfigMaps

A ConfigMap stores non-confidential key-value data, which Pods consume as environment variables, command-line arguments or files in a volume . It provides no secrecy or encryption; confidential data belongs in a Secret . Its data cannot exceed 1 MiB .

How you consume a ConfigMap decides what happens when it changes:

Consumed as — After the ConfigMap changes; Environment variable — Never updated; the Pod must be restarted; Mounted volume — Updated eventually, after up to the kubelet sync period plus cache propagation delay; Volume with subPath — Never updated

In the lab, change a ConfigMap value and watch the same key in three places: an environment variable that keeps its old value until the Pod restarts, a mounted file that updates after a delay, and a subPath mount that never does. The simulation compresses the propagation delay.

Secrets

Secrets hold confidential data, but the protection is thinner than the name suggests. Values in data must be base64-encoded, and base64 is an encoding, not encryption :

``bash printf 's3cr3t' | base64 # prints czNjcjN0 echo 'czNjcjN0' | base64 -d # prints s3cr3t `

By default Secrets are stored unencrypted in etcd. Anyone with API or etcd access can read them, and anyone who can create a Pod in a namespace can read that namespace's Secrets .

Namespaces

Namespaces group resources within one cluster: names must be unique within a namespace but can repeat across them . A cluster starts with four: default, kube-node-lease for node heartbeat Leases, kube-public, readable by everyone, and kube-system for system objects .

A realistic scenario

An operator raises a log level by editing the web-config ConfigMap from info to debug. The app reads LOG_LEVEL from an environment variable, so nothing happens. Running Pods keep info until they are replaced . Once the Pods are replaced, for example with kubectl rollout restart deployment web, the new ones read debug`. Had the app read a mounted file and watched it, the change would have arrived on its own, after a delay.

Immutable configuration

ConfigMaps and Secrets can be marked immutable: true. That prevents accidental changes and lowers kube-apiserver load, because the watches on them are closed . To change one, you create a new object and point the Pod template at it, which also gives you a rollout.

How Secrets travel

Each Secret is limited to 1 MiB, so very large Secrets cannot exhaust API server and kubelet memory . Built-in types include Opaque, kubernetes.io/tls, kubernetes.io/dockerconfigjson and kubernetes.io/service-account-token; an empty type means Opaque . A Secret is sent only to nodes running a Pod that needs it, and the kubelet keeps it in tmpfs rather than on durable storage, deleting it when the Pod goes away . Mounted Secret volumes update eventually; subPath mounts do not .

Without --encryption-provider-config, kube-apiserver writes resources to etcd as plain text, and listing identity as the first provider still gives no confidentiality . Turning encryption on does not rewrite existing data; each Secret must be rewritten before it is encrypted at rest .

The minimum steps the docs list are: enable encryption at rest, use least-privilege RBAC, restrict which containers get Secrets, and consider external secret stores . RBAC needs care. Granting list or watch on Secrets lets the holder read every Secret in the namespace, not only those its Pods reference . The built-in view role cannot read Secrets for that reason , while edit can .

Namespace scoping

Not everything is namespaced. Nodes, PersistentVolumes, StorageClasses and Namespaces themselves are cluster-scoped . Namespaces cannot be nested, each resource is in exactly one, and the kube- prefix is reserved . kube-node-lease holds one Lease per node, which the kubelet renews as a heartbeat . Because a Service's DNS name includes its namespace, the same config works in dev, staging and prod namespaces, and reaching across needs the full name .

Quotas

A ResourceQuota limits a namespace's total resource use and object counts, and a request that would exceed it is rejected with 403 Forbidden . Once a namespace has a cpu or memory quota, Pods must declare requests or limits, and a LimitRange can supply defaults . The trap: a Deployment over quota is still created, but its Pods may never appear , with the failure visible as a ReplicaFailure condition .

A Namespace is a naming and policy boundary, not a network or security boundary on its own. Pods in different namespaces can still reach each other unless a NetworkPolicy, enforced by a plugin that supports it, says otherwise .

Investigate it

``bash kubectl get configmap web-config -o yaml kubectl get secret db-credentials -o jsonpath='{.data.password}' | base64 -d kubectl exec <pod-name> -- printenv LOG_LEVEL kubectl get namespaces kubectl api-resources --namespaced=false kubectl describe resourcequota -n <namespace> kubectl auth can-i list secrets -n <namespace> ``

The second command shows how little stands between API access and a Secret's value. How quickly a mounted update reaches a file varies with kubelet configuration .

19. Horizontal Pod Autoscaling

The HorizontalPodAutoscaler (HPA) scales a workload such as a Deployment or StatefulSet by changing its replica count, working through the target's scale subresource; it does not apply to objects that cannot be scaled, such as DaemonSets . It is part of the core Kubernetes API . It exists because the right number of replicas changes during the day, and editing replicas by hand does not keep up.

A control loop on a timer

The HPA is a control loop that runs at intervals rather than continuously; the kube-controller-manager flag --horizontal-pod-autoscaler-sync-period sets the interval, which defaults to 15 seconds . On each pass it reads metrics, computes a desired count and writes it back. The core formula is:

``text desiredReplicas = ceil[currentReplicas × (currentMetricValue / desiredMetricValue)] `

So 200m of CPU against a 100m target doubles the replicas, and 50m halves them . For a utilization target, utilization is a percentage of the containers' resource requests; if a container has no request for that resource, utilization is undefined and the HPA takes no action for that metric . To avoid flapping, no scaling happens if the ratio is within 0.1 of 1.0 by default . With several metrics, the HPA computes a desired count for each and uses the largest .

Scaling up is fast and scaling down is deliberately slow. By default scale-up has no stabilization window , while scale-down uses the highest recommendation from the last 5 minutes .

The lab below lets you move a traffic slider, set the target utilization and the min/max bounds, and switch metrics or the HPA itself off. Watch the chart of desired, current and ready replicas, and the formula as it is evaluated at each sync. The simulation compresses time, so a 5-minute stabilization window passes in seconds.

Where the numbers come from

Per-Pod resource metrics come from the resource metrics API (metrics.k8s.io), usually served by the Metrics Server add-on, which has to be installed separately . cAdvisor inside the kubelet collects container usage, metrics-server pulls it from every kubelet, and the API server serves it to the HPA, the VPA and kubectl top; this pipeline covers only CPU and memory . Other metrics come from the custom or external metrics APIs .

What the HPA does not do

The HPA writes replica counts. It never adds nodes. If the new Pods do not fit on existing nodes, they stay Pending until something else adds capacity: node autoscaling provisions new Nodes when Pods cannot be scheduled on the existing ones , and that is a separate component (covered in the deep dive).

A realistic scenario

A shop's checkout Deployment runs 4 replicas with a 60% CPU target. A promotion pushes average utilization to 120%. At the next sync the HPA computes ceil(4 × 120/60) = 8 and writes replicas: 8`; the default scale-up policies allow that jump . If the cluster has room, four new Pods start; if not, they wait in Pending. When traffic drops, the replica count falls only after the 5-minute window has passed .

Not-ready Pods and missing metrics

Before computing the ratio, the HPA ignores Pods that are being deleted and discards failed Pods . Pods with missing metrics are set aside, and when scaling on CPU so are Pods that are not yet ready or whose latest sample predates readiness . The average is then recomputed with the set-aside Pods counted at 100% of target when scaling down and 0% when scaling up ; on a scale-up, not-yet-ready Pods are also counted at 0% . If the adjusted ratio flips direction or falls within tolerance, nothing is scaled, though the status still reports the original average . A Pod counts as not yet ready for CPU if it turned ready within --horizontal-pod-autoscaler-initial-readiness-delay (default 30s) of starting , and during the 5-minute CPU initialization period its usage is ignored unless the sample was taken entirely while it was Ready .

If some metrics cannot be fetched and the rest suggest a scale-down, scaling is skipped; a scale-up still happens if any available metric asks for more .

The HPA concept page does not say what happens when no metric at all can be fetched, for example when Metrics Server is down. Only the partial-failure rule is documented . The resource metrics pipeline page also does not give the interval at which metrics-server scrapes kubelets .

Behaviour policies

The behavior field limits the rate of change. When several policies apply, the one allowing the largest change wins by default (selectPolicy: Max); Min picks the smallest and Disabled turns that direction off . A policy's periodSeconds can be at most 1800 , and percent policies round up . The default scale-down behaviour is a 300 s window plus a policy allowing 100% of current replicas to be removed every 15 s, so it can go straight to minReplicas once the window allows . Since v1.33 each HPA can also set its own tolerance per direction .

``yaml apiVersion: autoscaling/v2 kind: HorizontalPodAutoscaler metadata: name: checkout spec: scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: checkout minReplicas: 2 maxReplicas: 20 metrics: - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 60 behavior: scaleDown: stabilizationWindowSeconds: 300 policies: - type: Pods value: 4 periodSeconds: 60 `

With an HPA active, remove spec.replicas from the Deployment manifest, or every kubectl apply resets the count . During a rollout the Deployment controller splits the HPA's number across ReplicaSets . Scale to zero is beta in v1.37 and works only with object or external metrics .

Node autoscaling and the VPA

Node autoscalers create or delete cloud provider resources, usually VMs . Cluster Autoscaler (pre-configured node groups) and Karpenter (provisioning from NodePools) are the two sponsored by SIG Autoscaling; both are separate components integrated with each cloud provider, not part of the control plane . They look only at Pod requests, not real usage, which is why they are paired with the HPA . Removing a non-empty Node terminates its Pods and recreates them elsewhere . How quickly a new node appears depends on the provider.

The VerticalPodAutoscaler is a CRD-based add-on, installed separately, that adjusts requests and limits through a recommender, an updater and an admission webhook .

Investigate it

`bash kubectl autoscale deployment checkout --cpu-percent=60 --min=2 --max=20 kubectl get hpa kubectl describe hpa checkout kubectl top pods `

kubectl get hpa and kubectl describe hpa show current and target metrics, conditions and recent scaling events . kubectl top` fails if the Metrics API is not served .

20. Rolling updates and rollbacks

A Deployment is declarative: you state the desired state, and the Deployment controller moves the actual state towards it at a controlled rate using ReplicaSets it owns . A rollout, meaning a new ReplicaSet and a new revision, starts only when .spec.template changes; scaling does not start one . The strategy decides how old Pods give way to new ones.

Two strategies

.spec.strategy.type is either RollingUpdate (the default) or Recreate . With Recreate every old Pod is killed, and their removal awaited, before any new Pod is created, which causes downtime . RollingUpdate replaces Pods gradually within two limits:

• maxUnavailable (default 25%) is the most Pods that may be unavailable during the update; a percentage rounds down .
• maxSurge (default 25%) is the most Pods that may exist above the desired count; a percentage rounds up .

They cannot both be 0 . With the defaults, at least 75% of the desired Pods stay available and at most 125% exist .

The documentation works through 3 replicas with the defaults: maxUnavailable rounds down to 0 and maxSurge rounds up to 1. The new ReplicaSet goes to 1, then the old one to 2 and the new one to 2, and so on, never fewer than 3 available or more than 4 in total, until the new ReplicaSet has 3 and the old one 0 .

Readiness is the gate

Each step waits for new Pods to become ready, or old ones to be killed . A new Pod counts as available once it is Ready, or once it has stayed ready for minReadySeconds if that is set above 0 . A release whose Pods never pass their readiness probe therefore stops the rollout instead of replacing every healthy Pod.

In the lab below, roll v1 to v2 and change maxSurge and maxUnavailable to see how many Pods exist and how many are available at each step. Pause and resume, inject a readiness failure, or ship a release that is never ready and watch the Deployment report ProgressDeadlineExceeded, then roll back. The simulation compresses the 10-minute deadline.

When it stalls

.spec.progressDeadlineSeconds defaults to 600. If a rollout makes no progress for that long, the Progressing condition becomes False with reason ProgressDeadlineExceeded . That is only a report: Kubernetes does not roll back by itself, and higher-level tools must act on the condition . kubectl rollout status exits non-zero when the deadline is exceeded, so it can gate a CI pipeline .

A realistic scenario

A team ships v2 of an API with a typo in its database URL. The first surge Pod starts but never passes readiness, and with three replicas maxUnavailable rounds down to 0, so the old Pods keep serving. Ten minutes later the pipeline's kubectl rollout status fails, and someone runs kubectl rollout undo. Users saw no errors in this case, but only because the readiness probe actually exercised the database connection.

A rolling update is not a zero-downtime guarantee. Availability depends on the strategy settings, on readiness probes that reflect real health, on spare capacity for surge Pods, and on the application handling shutdown and version skew.

Counting during the update

Terminating Pods are not counted in availableReplicas, so the real Pod count and resource use can go above replicas plus maxSurge until terminating Pods finish their grace period . Surge Pods need room: if they cannot be scheduled, they stay Pending and the rollout makes no progress. The docs list the usual causes of stuck rollouts as insufficient quota, readiness probe failures, image pull errors, insufficient permissions, limit ranges and application misconfiguration . Pod creation errors such as exceeded quota appear as a ReplicaFailure condition with reason FailedCreate .

A rollout is complete when every replica is updated and available and no old replicas run; Progressing then reads True with reason NewReplicaSetAvailable .

``yaml apiVersion: apps/v1 kind: Deployment metadata: name: api spec: replicas: 3 minReadySeconds: 10 progressDeadlineSeconds: 600 strategy: type: RollingUpdate rollingUpdate: maxSurge: 1 maxUnavailable: 0 selector: matchLabels: app: api template: metadata: labels: app: api spec: containers: - name: api image: registry.example.com/api:2.0.0 readinessProbe: httpGet: path: /ready port: 8080 `

Rollover and proportional scaling

If the template changes again mid-rollout, the controller creates another ReplicaSet and scales down the one it was scaling up, without waiting for the earlier rollout to finish . If an autoscaler or a user scales the Deployment mid-rollout, the extra replicas are spread across active ReplicaSets in proportion .

Pause, resume and rollback

kubectl rollout pause stops template changes from starting rollouts, so you can batch several edits and resume once . A paused rollout is not checked against the progress deadline , and a paused Deployment cannot be rolled back .

kubectl rollout undo returns to the previous revision, and --to-revision picks a specific one. Only the Pod template is rolled back . A rollback is a forward move to a new revision number, not a rewind of the history . History lives in old ReplicaSets; revisionHistoryLimit keeps 10 by default, and with 0 a rollout cannot be undone .

Rolling back the template does not roll back anything outside it: database migrations, ConfigMap contents or external state stay as v2 left them.

Investigate it

`bash kubectl set image deployment/api api=registry.example.com/api:2.0.1 kubectl rollout status deployment/api kubectl get rs -l app=api kubectl rollout history deployment/api kubectl rollout undo deployment/api --to-revision=3 kubectl get deployment api -o jsonpath='{.status.conditions}' `

The CHANGE-CAUSE column in rollout history comes from the kubernetes.io/change-cause annotation; the old --record` flag is deprecated .

21. Health probes and recovery

Probes are health checks that the kubelet, not the control plane, runs regularly on each container, either by running code inside it or by making a network request . They exist because a running process is not the same as a working service: an app can be deadlocked, still warming a cache, or unable to reach its database while its process looks fine.

Three probes, three consequences

Probe — On failure — Restarts the container?; Readiness — The Pod's IP is removed from the EndpointSlices of every matching Service — No; it keeps running and keeps being probed for its whole life; Liveness — After failureThreshold failures the kubelet restarts the container; the Pod stays on the same node — Yes; Startup — Holds off liveness and readiness until it succeeds; if it fails, the container is killed and the restart policy applies — Yes

Each probe uses exactly one mechanism: exec passes on exit code 0, httpGet on status 200 to 399, tcpSocket if the port is open, and grpc if the health status is SERVING . A probe that is not configured always counts as Success .

Defaults

initialDelaySeconds defaults to 0 and periodSeconds to 10 ; timeoutSeconds to 1, and a timeout counts as a failure ; successThreshold to 1 ; failureThreshold to 3. So with the defaults a liveness restart comes after about 30 seconds of failures .

In the lab below, compare a readiness failure, a liveness failure and a crash. Watch the traffic: a readiness failure removes the Pod from the Service without restarting it, a liveness failure restarts the container after three failed checks, and a crash restarts it according to the restart policy. The simulation uses strict 10-second ticks, which is an approximation .

Restarts and back-off

A Pod's restartPolicy is Always (the default), OnFailure or Never, and Deployments allow only Always . The kubelet restarts exited containers with an exponential back-off of 10 s, 20 s, 40 s and so on, capped at 300 seconds , and resets the timer once a container has run for 10 minutes without problems . A restart always happens in the same Pod on the same node; moving work elsewhere needs a new Pod from a controller .

CrashLoopBackOff is not a Pod phase. It is a kubectl status showing that the back-off is in effect for a container that keeps crashing .

A realistic scenario

A Java service takes 90 seconds to start. Its liveness probe uses the defaults, so the kubelet kills it after about 30 seconds, before it ever serves; it restarts, is killed again, and ends in CrashLoopBackOff. Nothing is wrong with the app. The fix is a startup probe on the same endpoint with a high failureThreshold .

Misconfigured liveness probes

Liveness probes do not wait for readiness probes, and a careless one can cause cascading failures under load: slow responses fail the probe, containers restart, requests fail and the remaining Pods take more load . A liveness check that calls a shared dependency turns a database blip into a fleet-wide restart. Keep liveness checks local to the process, and use readiness for "can I serve right now".

``yaml apiVersion: v1 kind: Pod metadata: name: api spec: containers: - name: api image: registry.example.com/api:2.0.0 startupProbe: httpGet: path: /healthz port: 8080 failureThreshold: 30 periodSeconds: 10 livenessProbe: httpGet: path: /healthz port: 8080 readinessProbe: httpGet: path: /ready port: 8080 `

A probe result can also be Unknown, which means no action and another check; a readiness probe counts as Failure until its initial delay has passed . While a container is not Ready, the kubelet may probe readiness off schedule so the Pod becomes ready sooner . A probe-triggered restart respects terminationGracePeriodSeconds, the Pod's value (30 s if unset) unless a probe-level value is set; that is not allowed on readiness probes .

The first restart: the docs disagree

The restart-policy section gives the delay series as 10 s, 20 s, 40 s, capped at 300 s . The same page's summary list says the first crash gets an immediate restart and back-off covers later ones .

The Pod Lifecycle page does not state the exact delay before the very first restart; its two descriptions differ. The lab treats the series as the delay before each restart after the first crash.

Two feature gates change the numbers. ReduceDefaultCrashLoopBackOffDecay (alpha since v1.33, off by default) starts at 1 s and caps at 60 s . KubeletCrashLoopBackOffMax (beta since v1.35, on by default) lets each node lower the cap through crashLoopBackOff.maxContainerRestartPeriod .

OOMKilled

Memory limits are enforced reactively by kernel OOM kills, while CPU limits are enforced by throttling . A container killed for exceeding its memory limit shows reason OOMKilled, with exit code 137 in the docs' example, and if its restart policy allows, the kubelet restarts it again and again . That loop also shows as CrashLoopBackOff, whose causes include app errors, bad configuration, resource limits and failing liveness or startup probes .

Investigate it

`bash kubectl get pods kubectl describe pod api kubectl logs api --previous kubectl get pod api -o jsonpath='{.status.containerStatuses[0].lastState}' kubectl get endpointslices -l kubernetes.io/service-name=api `

kubectl logs shows the container's output and kubectl describe pod its events . A crash-looping container shows State: Waiting with Reason: CrashLoopBackOff and a Last State of Terminated with an exit code . By default the kubelet keeps one terminated container and its logs after a restart, which is what --previous` reads .

22. Node failures and Pod replacement

Machines fail, lose power or drop off the network. Kubernetes does not see a failure directly; it sees a node that has stopped reporting, and it reacts on timers. Understanding those timers explains why replacement Pods take minutes, not seconds.

Heartbeats

Nodes send two kinds of heartbeat: updates to the Node's .status, and a Lease object in the kube-node-lease namespace, one per Node . The kubelet renews its Lease every 10 seconds by default . The node controller checks every node every 5 seconds .

The timeline

• Silence. If a running Node goes quiet for --node-monitor-grace-period, 50 seconds by default, the node controller sets its Ready condition to Unknown . Before v1.32 that default was 40 seconds .
• Taint. Ready=Unknown maps to the node.kubernetes.io/unreachable taint, and Ready=False to node.kubernetes.io/not-ready; both carry the NoExecute effect . The scheduler now places no new Pods there .
• Wait. Every Pod gets automatic tolerations for both taints with tolerationSeconds: 300, so Pods stay bound to the failed node for 5 minutes .
• Evict. When the toleration expires, API-initiated eviction of the node's Pods begins .
• Replace. Controllers notice too few Pods and create new ones, which the scheduler places on surviving nodes, if they have room.

So for a Deployment with default settings, replacement starts roughly 50 s + 300 s after the node goes silent, and the Pods then still have to be scheduled, pull images and pass readiness.

In the lab below, try deleting a Pod, crashing a container and failing probes, then mark a node unreachable. Watch the simulated clock: Unknown after the 50-second grace period, eviction after the 300-second toleration, replacements on other nodes, and the old Pods stuck in Terminating. The simulation compresses time; the clock shows what would have elapsed.

Pods that do not go away

Pods on an unreachable node are not deleted automatically. They show as Terminating or Unknown and leave the API only when the Node object is deleted, the kubelet comes back and confirms, or someone force-deletes them . During a network partition the kubelet never hears that its Pods were deleted, so they may keep running, and replacements elsewhere can run at the same time as the originals .

A realistic scenario

A worker in a three-node cluster loses its network link. For about a minute nothing changes. Then the node shows NotReady, and five minutes later its Deployment Pods show Terminating while replacements start on the other two nodes. If those nodes lack room for the extra Pods, the replacements wait in Pending. Kubernetes does not provision a machine; only a node autoscaler or an operator adds capacity .

Deployments versus StatefulSets

A Deployment's ReplicaSet can create a replacement as soon as a Pod is marked for deletion. A StatefulSet guarantees at most one Pod per identity, so it waits until the old Pod is really gone and its name is free . With the node unreachable, that does not happen on its own, so a StatefulSet replica on a dead node is not replaced until someone deletes the Node or the Pod.

Force deletion (kubectl delete pod --grace-period=0 --force) frees the name without the kubelet's confirmation. For StatefulSets it can leave two Pods with the same identity running and cause split-brain .

Volumes and non-graceful shutdown

If a node shuts down without the kubelet noticing, StatefulSet Pods stay stuck in Terminating and their volumes stay attached to the dead node . If a Pod deletion has not succeeded within 6 minutes and the node is unhealthy, Kubernetes force-detaches the volumes . For faster recovery, an admin can add the node.kubernetes.io/out-of-service taint, which force-deletes Pods that do not tolerate it and detaches their volumes at once; the admin must remove the taint afterwards . What detaching actually involves depends on the CSI driver and storage system.

Graceful node shutdown on Linux uses systemd inhibitor locks, but it stays inactive unless shutdownGracePeriod and shutdownGracePeriodCriticalPods are set, because both default to 0 .

Eviction is rate-limited

Eviction runs at --node-eviction-rate, 0.1 nodes per second by default . A zone is unhealthy when at least 55% of its nodes (minimum 3) are not Ready ; then small clusters stop evictions and larger ones slow to 0.01 per second . If every zone is fully unhealthy, the controller assumes the control plane has lost connectivity and evicts nothing . These rules stop a network fault from emptying the whole cluster.

On a cloud provider, the node controller asks whether an unhealthy node's VM still exists, and deletes the Node if not, which lets bound Pods be cleaned up . PodGC later removes orphan Pods bound to Nodes that no longer exist .

Tuning the wait

You can shorten the 5-minute wait per workload with your own tolerations :

``yaml tolerations: - key: node.kubernetes.io/unreachable operator: Exists effect: NoExecute tolerationSeconds: 60 - key: node.kubernetes.io/not-ready operator: Exists effect: NoExecute tolerationSeconds: 60 `

Shorter is not free: a brief network blip then evicts Pods that would have recovered in place.

The Nodes page still says 40 seconds in one sentence about the unreachable timeout; that is a stale older default . Use 50 seconds for current versions .

Investigate it

`bash kubectl get nodes kubectl get leases -n kube-node-lease kubectl describe node worker-2 kubectl get pods -o wide --field-selector spec.nodeName=worker-2 kubectl get events --field-selector involvedObject.kind=Node `

Start with kubectl get nodes` to check that every node is registered and Ready . Since v1.29 taint-based eviction runs in its own taint-eviction-controller inside kube-controller-manager .

23. Scheduling constraints and resource pressure

By default the scheduler places a Pod on any node with room for its requests. Constraints narrow or bias that choice: run on GPU nodes, keep replicas in different zones, keep ordinary workloads off control-plane nodes. Resource pressure works from the other direction, deciding what leaves a node that is running out of memory or disk.

Constraints on nodes

• nodeSelector is the simplest: the Pod lands only on nodes carrying every listed label .
• Node affinity has a hard form, requiredDuringSchedulingIgnoredDuringExecution (no match, no scheduling), and a soft form, preferred... (the scheduler tries but schedules anyway) . "IgnoredDuringExecution" means a running Pod is not evicted if labels later change . Each preferred rule has a weight from 1 to 100, and a node's satisfied weights are added to its score .
• Inter-pod affinity and anti-affinity use the labels of Pods already running in a topology domain, such as a node or zone . They are expensive, and the docs advise against them in clusters larger than several hundred nodes .
• Topology spread constraints limit how unevenly matching Pods spread across domains; maxSkew is required , and whenUnsatisfiable defaults to the hard DoNotSchedule .

Taints and tolerations

Taints sit on nodes and repel Pods; tolerations on Pods let them past matching taints. A toleration permits scheduling but does not attract the Pod . NoSchedule blocks new Pods without a toleration, PreferNoSchedule is a soft version , and NoExecute also evicts running Pods . kubectl cordon marks a node unschedulable, which keeps new Pods off but leaves existing ones alone .

In the lab below, add a taint and see Pods without a toleration filtered out, switch between required and preferred node affinity, cordon a node, and shrink a node's free resources. Each node shows why it was rejected; the scores are simplified and illustrative, not kube-scheduler's real numbers.

Requests decide, not usage

The scheduler checks the sum of requests against capacity, not actual usage, and limits play no part . Limits may add up to more than the node's capacity, which is overcommit . Every Pod gets a QoS class: Guaranteed (every container has CPU and memory requests equal to limits) , Burstable (at least one request or limit) , or BestEffort (none) .

A realistic scenario

A team adds a required node affinity for disktype=ssd but the label exists on only one node, which is full. The new Pods stay Pending, and kubectl describe pod shows a FailedScheduling event . Nothing fails permanently; the Pods wait until a node becomes feasible . Changing the rule to preferred lets them run elsewhere while still favouring SSD nodes.

Priority and preemption

If a Pod cannot be scheduled, the scheduler may preempt lower-priority Pods to make room; priority comes from a PriorityClass named in priorityClassName . Without a globalDefault class, Pods get priority zero . The preemptor's status.nominatedNodeName is set, but it is not guaranteed to land there , and victims get their normal graceful termination period . Preemption respects PodDisruptionBudgets only on a best-effort basis and ignores QoS class .

Spreading replicas

``yaml topologySpreadConstraints: - maxSkew: 1 topologyKey: topology.kubernetes.io/zone whenUnsatisfiable: DoNotSchedule labelSelector: matchLabels: app: web `

Without cluster-level defaults, kube-scheduler already applies soft spreading across hostnames and zones . Spread is enforced only at scheduling time, so scaling down can leave Pods imbalanced .

Node-pressure eviction

The kubelet evicts Pods when a node runs short. It marks them Failed and terminates them, ignoring PDBs and terminationGracePeriodSeconds . Default hard thresholds on Linux are memory.available<100Mi, nodefs.available<10%, imagefs.available<15% and inode thresholds of 5% . Overriding one hard threshold sets the others to zero unless MergeDefaultEvictionSettings is true . The kubelet first tries to reclaim node-level resources, such as unused images .

The docs disagree on eviction order. The QoS page says BestEffort, then Burstable, then Guaranteed . The Node-pressure Eviction page says QoS class is not used directly: Pods using more than their requests go first, ordered by Priority and then by how far over they are .

If memory runs out before the kubelet reacts, the kernel OOM killer acts on scores set by QoS class; an OOM-killed container can be restarted in place, unlike an evicted Pod .

Voluntary disruptions and PDBs

A PodDisruptionBudget limits how many replicas can be down at once for voluntary disruptions; involuntary ones cannot be blocked but still count against it . kubectl drain evicts through the Eviction API, so it respects PDBs . An eviction returns 429 Too Many Requests when a PDB blocks it .

`yaml apiVersion: policy/v1 kind: PodDisruptionBudget metadata: name: web spec: minAvailable: 2 selector: matchLabels: app: web `

Quota and defaults

ResourceQuota caps a namespace's totals and rejects requests that would exceed it with 403; once CPU or memory quotas exist, Pods must state requests or limits . LimitRange injects defaults and enforces min, max and ratio at admission only . A container with a limit but no request gets the limit copied into its request, so it reserves its full limit .

Investigate it

`bash kubectl describe node worker-1 kubectl taint nodes worker-1 dedicated=gpu:NoSchedule kubectl cordon worker-1 kubectl get pod web-7c9 -o jsonpath='{.status.qosClass}' kubectl drain worker-1 --ignore-daemonsets kubectl get pdb `

kubectl describe node` shows requests and limits under Allocated resources .

24. StatefulSets, DaemonSets and Jobs

Deployments treat Pods as interchangeable. Some workloads cannot: a database replica needs its own disk and a name its peers can find, a log collector must run on every node, a migration must run to completion once. Kubernetes has a controller for each shape.

StatefulSets: stable identity

A StatefulSet gives each Pod a sticky identity that it keeps across rescheduling; its Pods share a spec but are not interchangeable . Pods get ordinals 0 to N-1, and hostnames such as web-0, web-1 and web-2 . Through a governing headless Service, each Pod gets a stable DNS name of the form $(podname).$(service name).$(namespace).svc.cluster.local ; you must create that Service yourself .

Storage follows the identity. Each Pod gets one PVC per volumeClaimTemplate, and a rescheduled Pod mounts the same volumes again . Deleting or scaling down a StatefulSet does not delete its volumes by default .

Order is the default. With OrderedReady, Pods are created from 0 upward and deleted from N-1 downward, and each waits for all lower ordinals to be Running and Ready . So web-1 is not created until web-0 is Running and Ready . podManagementPolicy: Parallel launches or terminates Pods together .

In the lab below, create a StatefulSet and watch each Pod wait for the previous one to become Ready, note the stable names and per-Pod PVCs, then scale down and see the highest ordinal go first while its PVC stays. Switch to the DaemonSet view and add a node to see a Pod appear on it. The simulation compresses time.

DaemonSets: one per node

A DaemonSet runs one copy of a Pod on every matching node; Pods are added when nodes join and garbage-collected when they leave. Typical uses are storage, log collection and node monitoring . The controller creates one Pod per eligible node with a node affinity pinned to that node, and the default scheduler binds it .

Jobs and CronJobs

A Job keeps retrying Pods until the required number complete successfully, with backoffLimit defaulting to 6 . A CronJob creates a Job roughly once per scheduled time; it may occasionally create two or none, so Jobs should be idempotent .

A realistic scenario

A three-replica PostgreSQL cluster runs as a StatefulSet. db-1's node fails. Unlike a Deployment Pod, db-1 is not recreated elsewhere until the old Pod is really gone . When it is, the new db-1 keeps its name and reattaches its existing PVC , if the volume can reach the new node. Whether it can depends on the storage topology and CSI driver.

Updates, partitions and their traps

The default RollingUpdate strategy updates one Pod at a time from the highest ordinal down, waiting for each to be Running and Ready (plus minReadySeconds) . OnDelete updates a Pod only when you delete it . With a partition, only ordinals at or above it get the new template, which supports canary and staged rollouts .

rollingUpdate.maxUnavailable (beta since v1.35) defaults to 1 and cannot be 0. Unlike a Deployment, a percentage here is rounded up .

With OrderedReady, a template that never becomes Ready stops the rollout. Reverting the template is not enough; you must also delete the broken Pods by hand .

``yaml apiVersion: apps/v1 kind: StatefulSet metadata: name: web spec: serviceName: web replicas: 3 selector: matchLabels: app: web updateStrategy: type: RollingUpdate rollingUpdate: partition: 2 persistentVolumeClaimRetentionPolicy: whenDeleted: Retain whenScaled: Retain template: metadata: labels: app: web spec: containers: - name: web image: registry.k8s.io/nginx-slim:0.8 volumeMounts: - name: data mountPath: /usr/share/nginx/html volumeClaimTemplates: - metadata: name: data spec: accessModes: ["ReadWriteOnce"] resources: requests: storage: 1Gi `

persistentVolumeClaimRetentionPolicy (stable since v1.32) sets whenDeleted and whenScaled to Retain (the default) or Delete; a Pod replaced after node failure always keeps its PVC . Deleting a StatefulSet does not guarantee termination order, so scale it to 0 first . A Recreate update strategy is alpha in v1.37 and disabled by default .

DaemonSet tolerations

DaemonSet Pods get NoExecute tolerations for not-ready and unreachable with no tolerationSeconds, so they are never evicted for those reasons, plus NoSchedule tolerations for memory, disk and PID pressure and for unschedulable nodes . That is why they keep running on cordoned nodes and why kubectl drain needs --ignore-daemonsets. The default scheduler may preempt lower-priority Pods to make room for one .

Job retries

The Job controller recreates failed Pods with a back-off of 10 s, 20 s, 40 s and so on, capped at ten minutes; the page said six minutes until a docs fix on 2026-10-12 . backoffLimit becomes MaxInt32 when backoffLimitPerIndex is set . A CronJob's concurrencyPolicy defaults to Allow .

Investigate it

`bash kubectl get statefulset web kubectl get pods -l app=web -w kubectl get pvc kubectl rollout status statefulset/web kubectl patch statefulset web -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":0}}}}' kubectl get daemonset -A kubectl get jobs,cronjobs `

From a Pod in the same namespace, web-0.web` resolves through the headless Service to that one Pod .

25. Security, RBAC and service accounts

Every request to the API server is authenticated first and authorized second, and only then passed to admission control; admission runs only on requests that authorization allowed . Authentication establishes who is asking. Authorization answers a narrower question: may this identity perform this verb on this resource in this namespace? Authorizers see exactly those attributes: the user, groups, extra fields, the verb and, for resource requests, the namespace, resource and API group .

Deny by default, first answer wins

Authorization denies by default: every part of a request must be allowed by some authorization mechanism . Authorizers run in order. The first one that allows or denies decides, none after it are asked, and if every authorizer has no opinion the API server returns HTTP 403 . Cluster installers normally configure the Node and RBAC modes . Node authorization is a special mode that gives kubelets permissions based on the Pods scheduled to them ; RBAC covers everyone else.

RBAC in four objects

RBAC permissions only add access. A Role or ClusterRole lists rules that allow things, and there are no deny rules . A Role belongs to one namespace and grants permissions only there; a ClusterRole is not namespaced . A binding links a role to subjects, which are users, groups or service accounts. A RoleBinding grants access inside one namespace, a ClusterRoleBinding across the whole cluster . A RoleBinding can also point at a ClusterRole, and the permissions then apply only inside the binding's namespace, so one shared definition can be reused in many namespaces .

Rules are written in verbs: get, list, watch, create, update, patch, delete and deletecollection . HTTP methods map onto them, so a DELETE is checked as delete or deletecollection and a PATCH as patch .

``yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: deployment-deployer namespace: default rules: - apiGroups: ["apps"] resources: ["deployments"] verbs: ["get", "list", "watch", "update", "patch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: ci-bot-deployer namespace: default subjects: - kind: ServiceAccount name: ci-bot namespace: default roleRef: kind: Role name: deployment-deployer apiGroup: rbac.authorization.k8s.io `

To check a decision without making the request, kubectl auth can-i asks the authorization layer through the SelfSubjectAccessReview API and prints yes or no; with --as it checks another user or service account .

In the lab below, the user jane and the service account ci-bot each try kubectl delete deployment web. Toggle bindings to the built-in view and edit roles or to a custom Role, and watch the authorization decision, its reason and the matching kubectl auth can-i answer change.

The docs say only that a denied request gets HTTP 403 Forbidden; no current page we checked shows the exact modern wording of the error . The one documented sample is in an older shape: Error from server (Forbidden): User "clark" cannot get nodes at the cluster scope. . The message the lab prints is modelled on the common form, not quoted from the documentation.

A realistic case

A deploy pipeline runs as ci-bot, which an operator bound to view to be safe, and every rollout fails with 403. Because RBAC has no deny rules , there is nothing to remove: the fix is to add a binding that grants update and patch on Deployments in that one namespace, like the Role above, rather than reaching for edit or cluster-admin`.

Default roles

The default ClusterRoles meant for users are cluster-admin, admin, edit and view; the last three are meant to be granted inside a namespace with RoleBindings . cluster-admin is bound by default to the system:masters group and allows any action on any resource .

Role — What to know; view — Cannot read Secrets, because that would expose ServiceAccount credentials and allow privilege escalation; edit — Cannot read or write roles and bindings, but it can read Secrets and run Pods as any ServiceAccount, so it can effectively gain any ServiceAccount's access

RBAC blocks privilege escalation: you cannot create or update a Role with permissions you do not already hold unless you have the escalate verb . A binding's roleRef cannot be changed; you delete the binding and create a new one . And because RBAC cannot deny, putting AlwaysAllow first in the authorizer chain allows everything .

Service accounts and tokens

Kubernetes creates a ServiceAccount named default in every namespace and recreates it if deleted . Pods that name no ServiceAccount get it, and beyond API discovery it has no RBAC permissions . Service account usernames start with system:serviceaccount: and their groups with system:serviceaccounts: .

Since v1.22 a Pod's token is short-lived and rotates automatically; it comes from the TokenRequest API and is mounted as a projected volume . Before v1.22, Pods got a long-lived static token stored as a Secret . A documented describe sample shows the volume kube-api-access-xxxxx with TokenExpirationSeconds 3607, mounted read-only at /var/run/secrets/kubernetes.io/serviceaccount . Pods that never call the API can set automountServiceAccountToken: false .

Pod Security

The Pod Security Standards define three profiles: Privileged, Baseline and Restricted . Baseline forbids hostPath volumes . Restricted requires allowPrivilegeEscalation: false and runAsNonRoot: true, and containers must drop ALL capabilities, adding back only NET_BIND_SERVICE . The built-in Pod Security Admission controller, stable since v1.25, enforces them per namespace , chosen with labels such as pod-security.kubernetes.io/enforce: restricted . Only enforce blocks a Pod; audit and warn do not . enforce judges the Pods, not the Deployment, so a Deployment can be accepted while its Pods are rejected .

In a securityContext, allowPrivilegeEscalation defaults to true when unset , readOnlyRootFilesystem mounts the root filesystem read-only , capabilities adds or drops Linux capabilities , and container-level settings override Pod-level ones .

Secrets

Secrets are stored unencrypted in etcd by default , and their values are only base64-encoded . Without --encryption-provider-config, kube-apiserver writes them as plain text . list access reveals their contents, and anyone who can create a Pod that uses a Secret can read it , so keep watch and list on Secrets to the most privileged system components .

Investigate it

``bash kubectl auth can-i delete deployments -n default kubectl auth can-i delete deployments -n default --as=system:serviceaccount:default:ci-bot kubectl auth can-i --list -n default kubectl get rolebindings,clusterrolebindings -A -o wide kubectl describe clusterrole edit kubectl get namespaces --show-labels `

--list` prints every action allowed in the namespace , and a service account is checked by its full username .

The "4C's" model (Cloud, Clusters, Containers, Code) does not appear on the current security docs, and older versions could not be reached to cite it . The current Cloud Native Security page groups controls by lifecycle phase instead: Develop, Distribute, Deploy and Runtime .

26. Observability and troubleshooting

When something goes wrong in a cluster, the evidence is spread across several places: object status written by controllers and the kubelet, events recorded by components, container logs on the node, and resource metrics. Asking the same questions in the same order gets to the cause faster than guessing.

A debugging order

• kubectl get pods for the overview. It prints the columns NAME, READY, STATUS, RESTARTS and AGE . Values such as CrashLoopBackOff and Terminating in the STATUS column are display values, not Pod phases .
• kubectl describe pod for detail. The docs start here, because it shows container state and recent events : each container's state (Waiting, Running or Terminated), its readiness and its restart count .
• Events for the sequence. The modern Events table has the columns Type, Reason, Age, From and Message; a healthy start shows Scheduled from default-scheduler, then Pulling, Pulled, Created and Started from the kubelet . kubectl events --for pod/NAME filters to one object .
• kubectl logs --previous after a crash, which shows the logs of the previous container instance .
• kubectl top for usage, which only works when metrics-server is installed .

The STATUS value usually tells you where to look next:

Symptom — What it usually means — Look at; Pending — The Pod cannot be scheduled, usually for lack of resources; the scheduler's messages explain why — Events, kubectl describe node; Waiting, ErrImagePull, ImagePullBackOff — Scheduled but cannot run, most often because the image pull failed ; check the name, that it was pushed, and whether a manual pull works — Events; CrashLoopBackOff — State Waiting with reason CrashLoopBackOff, and Last State Terminated with an exit code — kubectl logs --previous; Service gets no traffic — An EndpointSlice with ENDPOINTS <none>, usually a selector that does not match the Pod labels — kubectl get endpointslices

In the lab below, pick an incident and investigate it the way you would on a real cluster: run kubectl get pods, describe, get events, top nodes and logs, then choose the root cause. The outputs are generated from the simulated cluster's state, so their wording is modelled rather than copied from a real kubectl.

A realistic case

After a release, the new Pods of web show CrashLoopBackOff while the old ones keep serving. kubectl describe pod shows Last State Terminated with a non-zero exit code . Plain kubectl logs is the wrong first move here: the crashed instance's output is what you need, and --previous returns it . If the message names a missing setting, the fix belongs in the Deployment's configuration, not in restarting Pods.

Events are short-lived

Events are stored in etcd and are namespaced, so kubectl get events needs --namespace for other namespaces . The API server keeps them for one hour by default (--event-ttl 1h0m0s) . Each has a type (Normal or Warning), a short machine-readable reason and a count . They are best-effort: retention is limited and their triggers and messages can change, so tools should not depend on them . An incident from yesterday has no events left unless you exported them.

Where logs live

Writing to stdout and stderr is the simplest and most common way to log . By default the container runtime writes Pod logs under /var/log/pods on the node , and the kubelet rotates them at containerLogMaxSize 10Mi with containerLogMaxFiles 5 . kubectl logs reads only the latest file, so rotated lines are not returned . The kubelet keeps one terminated container and its logs after a restart, and an evicted Pod takes its logs with it . Kubernetes has no built-in cluster-level logging; the usual approaches are a node-level agent, a sidecar, or pushing straight to a backend .

Metrics and traces

The resource metrics pipeline runs from cAdvisor in the kubelet, through metrics-server, to the Metrics API that the HPA, the VPA and kubectl top read; it covers only CPU and memory . It is not a full monitoring system , and it works only with metrics-server or another adapter deployed and an APIService registered for metrics.k8s.io . CPU is average core usage over a window; memory is the working set at collection time . Control plane components and the kubelet expose Prometheus-format metrics, usually on /metrics , and Alpha metrics carry no stability guarantee . Components export traces with OTLP over gRPC, by default to port 4317 .

Debugging a Service

Check that the name resolves; the full form is <service>.<namespace>.svc.cluster.local . Then check its EndpointSlices, whose output has the columns NAME, ADDRESSTYPE, PORTS and ENDPOINTS . A Pod failing its readiness probe is removed from them too . A Service that does not exist gives a NotFound error .

When exec is not enough

Ephemeral containers (stable since v1.25) run temporarily inside an existing Pod . They help with crashed containers or distroless images that have no shell . They have no resource guarantees, are never restarted, and cannot have ports or probes ; once added they cannot be changed or removed . kubectl debug --copy-to copies a Pod, and kubectl debug node/NAME starts a debugging Pod on a node .

A container can explain its own exit by writing to terminationMessagePath, /dev/termination-log by default ; the kubelet truncates messages to 4096 bytes . With terminationMessagePolicy: FallbackToLogsOnError, an empty file is replaced by the last log lines .

Investigate it

``bash kubectl get nodes kubectl get pods -o wide kubectl describe pod web-7d9c6b5f4-x2x8q kubectl events --for pod/web-7d9c6b5f4-x2x8q kubectl logs web-7d9c6b5f4-x2x8q --previous kubectl top nodes kubectl get endpointslices -l kubernetes.io/service-name=web kubectl debug -it web-7d9c6b5f4-x2x8q --image=busybox --target=web `

The last command adds an ephemeral container that shares the target's process namespace . Start cluster-wide problems with kubectl get nodes` .

The docs' FailedScheduling sample uses an older column layout , and the modern "0/3 nodes are available" wording is not shown on the debugging pages checked . Treat any exact scheduler message, including the lab's, as illustrative.

27. Production architecture and high availability

A cluster whose control plane runs on one machine has a single point of failure. High availability means separating and replicating the control plane, load balancing the API server, and having enough worker nodes . The pieces replicate in different ways: API servers are interchangeable, etcd members vote, and the scheduler and controller manager run one active copy at a time.

Two topologies

Layout — Each control plane node runs kube-apiserver, kube-scheduler and kube-controller-manager plus a local etcd member; workers reach the API server through a load balancer — etcd runs on its own hosts, separate from the control plane; Minimum — At least three control plane nodes — At least three control plane nodes and three etcd nodes; Trade-off — Losing one node loses both an etcd member and a control plane instance; this is the kubeadm default — One failure does less harm, at twice the hosts

The load balancer

kube-apiserver scales horizontally: you add instances and balance traffic across them . For a kubeadm HA cluster the docs put the control plane nodes behind a TCP-forwarding load balancer whose health check is a TCP check on the API server port, 6443 by default .

Leader election

kube-controller-manager and kube-scheduler use Leases for leader election, so only one instance is active and the others wait on standby . It is on by default . The default lease duration is 15 s, effectively the longest a stopped leader can go before a candidate replaces it ; the renew deadline is 10 s and the retry period 2 s . The production guide puts it this way: the scheduler should be fault tolerant, not highly available .

The diagram shows a load balancer in front of three control plane nodes, each with an API server, scheduler, controller manager and local etcd member. Follow which scheduler and controller manager hold the lease, and how the worker nodes reach the API only through the load balancer.

What keeps working without a control plane

If the API server is down you cannot stop, update or start Pods, but existing Pods and Services keep working unless they depend on the API . If its backing storage is lost the API server cannot start, yet kubelets keep running the same Pods and keep service proxying going . If most etcd members fail permanently, Kubernetes cannot change state: scheduled Pods may keep running but no new Pods can be scheduled . A control plane outage freezes the cluster rather than stopping it, but nothing heals during the freeze.

A realistic case

One of three stacked control plane nodes loses power. Quorum for three members is two , so etcd tolerates this one failure and writes continue. The load balancer's TCP check against that node's API server port starts failing. If the dead node held the scheduler's lease, a standby takes over once the lease runs out, which with the defaults is at most about 15 s . Losing a second node would cost etcd its quorum , and with no elected leader Kubernetes cannot change cluster state .

Building it with kubeadm

kubeadm commonly runs the control plane as static Pods . The first node is started with kubeadm init --control-plane-endpoint set to the load balancer address and --upload-certs ; certificates uploaded that way are deleted after two hours . Managed services hide all of this behind their own endpoint .

etcd sizing and backups

Run etcd with an odd number of members; a static five-member cluster is strongly recommended for production, and etcd should not be autoscaled . Probably no more than seven members, because larger clusters write more slowly . Adding members improves availability but never adds capacity . The docs' minimums are etcd 3.4.29+ or 3.5.11+ on dedicated machines or isolated environments .

etcd holds cluster configuration and should be backed up regularly . Take snapshots with etcdctl snapshot save from a live member, which does not affect its performance , and encrypt the files, because they hold all cluster state . Restore with etcdutl snapshot restore; restoring with etcdctl is deprecated since etcd v3.5 . The order matters: stop all API servers, restore every etcd instance, restart the API servers, and preferably restart the scheduler, controller manager and kubelets too . Components lose their leader lock during a restore and restart themselves .

Version skew

Component — Allowed skew relative to kube-apiserver; kube-apiserver (HA) — Newest and oldest within one minor version; kubelet — Not newer; up to three minors older; controller manager, scheduler, cloud-controller-manager — Not newer; up to one minor older; kube-proxy — Not newer; up to three minors older; kubectl — Within one minor, older or newer

Because no other component may be newer than the API server, the API servers are upgraded first .

Release cadence

Minor releases come out about three times a year , and patch releases usually monthly . The project maintains the three most recent minor branches, currently 1.37, 1.36 and 1.35 . Pages disagree on support length: about one year for 1.19 and newer , or roughly 14 months including a 2-month maintenance mode on the Patch Releases page . On the access date the newest release was 1.37.1 .

Scale and zones

Kubernetes v1.37 is designed for at most 110 Pods per node, 5,000 nodes, 150,000 Pods and 300,000 containers . When availability matters, replicate each control plane component across at least three failure zones . Kubernetes does not provide cross-zone resilience for API server endpoints itself; that comes from DNS or load balancers , and nodes do not automatically prefer a control plane endpoint in their own zone . For workloads, the kubelet's zone labels plus topology spread constraints control how Pods spread . Large clusters can move Events into a dedicated etcd .

Investigate it

``bash kubectl get leases -n kube-system kubectl get nodes -L topology.kubernetes.io/zone kubectl version kubectl get --raw='/readyz?verbose' ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \ --cacert=/etc/kubernetes/pki/etcd/ca.crt \ --cert=/etc/kubernetes/pki/etcd/server.crt \ --key=/etc/kubernetes/pki/etcd/server.key \ snapshot save /var/backups/etcd-snapshot.db etcdutl --data-dir /var/lib/etcd-restored snapshot restore /var/backups/etcd-snapshot.db ``

The certificate paths are kubeadm's usual layout; other installers differ.

An alpha feature gate in v1.36, off by default, lets kube-controller-manager release its lock on exit instead of waiting for the lease to expire . Without it, plan for the full lease duration on every leader change.

28. Kubernetes, Docker and Docker Compose

"Kubernetes or Docker?" is a question about two different layers. Docker builds and runs containers on a machine; Kubernetes decides which machine runs them and keeps them running there. Docker Compose sits in between: it describes a multi-container application, but for one Docker host.

Docker Engine is open source software for building and containerizing applications, with a client-server design: the dockerd daemon, its APIs and the docker CLI . The daemon creates and manages images, containers, networks and volumes .

Docker Compose defines and runs a multi-container application from a single YAML file and starts everything with one command . The default file is compose.yaml . A Compose service runs the same image and configuration one or more times, services talk over networks and store data in volumes , and Docker presents production use of Compose as single-host deployment, including to a remote Docker host .

Kubernetes schedules work across a cluster of nodes . It does not build images: you build one, push it to a registry and reference it in a Pod . On each node the kubelet talks to a container runtime over the CRI , and the supported runtimes include containerd and CRI-O .

Side by side

Job — Build images and run containers — Define and run a multi-container app — Run containerized workloads across nodes; Scope — One host — One Docker host, possibly remote — A cluster of nodes; Unit — Container — Service: one image, run one or more times — Pod: containers sharing network and storage; Builds images — Yes — Through Docker — No; Places work on machines — No — No — The scheduler binds Pods to nodes; Replaces failed work — — — Not described on the pages checked — Controllers replace Pods; the kubelet restarts containers; Networking — Networks managed by the daemon — Networks between services — Services and cluster DNS

Same images, different runtime

Images built with docker build run on every CRI runtime , so the artefact does not change when you move. Docker stays useful as a development and image-build tool; what was deprecated was Docker as the runtime underneath Kubernetes .

A realistic case

A small team runs a web app, a worker and a cache with Compose on one VM. That is a reasonable setup while a single host is acceptable : one file, one command . The case for Kubernetes appears when they need several machines, Pods replaced on surviving nodes after a failure , and rolling updates at a controlled rate . Their images carry over unchanged , and Docker's Compose Bridge (Docker Desktop 4.43.0+) can convert the Compose file into Kubernetes manifests as a starting point . Kubernetes does not add the machines for them; that is still a separate job .

The runtime stack

``text kubelet │ CRI (gRPC): runtime service + image service ▼ containerd or CRI-O (or Docker Engine via cri-dockerd) │ OCI runtime-spec ▼ runc ──► Linux namespaces + cgroups `

The CRI is a gRPC plugin interface between the kubelet (the client) and the runtime, stable since v1.23, so runtimes can be swapped without recompiling Kubernetes . Since v1.26 the kubelet requires the v1 CRI API, and a runtime without it stops the node registering . OCI publishes the image and runtime specs, and runc, which implements the runtime spec, is the default low-level runtime for both containerd and CRI-O . Pod networking is set up by CNI plugins that the runtime, not the kubelet, loads and calls .

A Pod gets its own Linux namespaces and cgroups, the same mechanisms that isolate a single container, so a Pod is like a set of containers with shared namespaces and shared volumes . Its containers share one IP and can talk over localhost .

The dockershim story

These points describe earlier versions. Dockershim existed because Docker Engine does not implement CRI; it was always meant to be temporary and became a heavy maintenance burden . Its removal was announced with v1.20 and it was removed in v1.24 . From 1.24 onward a cluster must use a CRI runtime such as containerd, CRI-O, or Docker Engine through cri-dockerd , whose socket defaults to /run/cri-dockerd.sock .

"Kubernetes dropped Docker" is a common misreading. Existing images were not affected by the removal , and Docker remains a build tool; only its role as the node runtime changed .

Images and tags

An image name with no registry host means Docker's public registry, a missing tag means latest, and a digest is an immutable content hash while a tag can move . Leaving out imagePullPolicy gives Always for :latest or untagged images and IfNotPresent otherwise . Containers should be treated as immutable: build a new image and recreate the container rather than patching it . That is the habit both Compose and Kubernetes reward.

Investigate it

`bash kubectl get nodes -o wide sudo crictl ps sudo crictl images docker build -t registry.example.com/shop/web:1.4.2 . docker push registry.example.com/shop/web:1.4.2 kubectl create deployment web --image=registry.example.com/shop/web:1.4.2 --replicas=3 docker compose config `

The CONTAINER-RUNTIME column of kubectl get nodes -o wide shows which runtime each node uses; crictl` talks to the node's CRI runtime directly.

The Docker Compose pages checked do not describe self-healing, rescheduling across machines or a reconciling control plane. They do describe Compose managing the lifecycle of services (start, stop, rebuild) on one Docker host . What Compose does when a container or its host fails is outside what those sources cover.

29. Interactive cluster playground

The earlier labs each isolate one mechanism. The playground puts them together in one simulated cluster, so you can see how they interact: a rolling update during a node failure, a crash loop while the API server is down, or a scale-up that no longer fits on the nodes you have.

How to use it

• Deploy and change. Apply a Deployment, set its replicas and resource requests, scale it, or change its image to start a rolling update.
• Change the cluster. Add or remove nodes, delete Pods, crash a container, fail a node, or send traffic through the Service.
• Control time. Pause, step one event at a time, or replay from the start. Each step is one component doing one thing, shown on the canvas and in the event timeline.
• Inspect. Click any object to see its spec and status, or use the kubectl tab for get and describe views of the same state.
• Share. The link encodes what you did, so someone else can replay the same run.

Scenario presets set up a starting cluster and a disruption, so you can begin from an interesting state instead of an empty one.

What is simulated

Everything. There is no real API server, etcd, kubelet or container: a deterministic model produces the events, and the kubectl output is generated from its state, so treat both as illustrative. Time is simulated and compressed, and the clock on screen is the simulation's, not yours. The playground lists its assumptions; the deep dive explains how the engine works.

Experiments worth trying

• Deploy three replicas and delete one Pod. The ReplicaSet creates a replacement from its template . See controllers.
• Give the Pod a CPU request larger than any node. It stays Pending until it can be placed . See the scheduler.
• Change the image with the default strategy and watch the counts stay within 75% available and 125% total . See rolling updates.
• Crash the container repeatedly and read the growing back-off: 10 s, 20 s, 40 s and so on, capped at 300 s . See probes and restarts.
• Fail a readiness check and watch the Pod leave the Service's endpoints without being restarted . See Services.
• Fail a node. It goes Unknown after 50 s and its Pods are evicted after their 300 s toleration . See node failure.
• Stop the API server while Pods run. Nothing new can be scheduled, but running containers keep running . See production and HA.
• Scale past what the nodes can hold. New Pods stay Pending, because Kubernetes does not create nodes ; adding machines is a node autoscaler's job . See autoscaling.

How the engine works

The simulation is a pure, deterministic function: applying a command to the cluster state returns every event the command causes, each with a snapshot of the whole cluster. The canvas, event log, inspector and kubectl views all read the same frames, so they cannot disagree, and the same inputs always replay the same way.

Each event is one action by one component. On every simulated second the components act in a fixed order: Deployment controller, ReplicaSet controller, scheduler, kubelets, node lifecycle controller, EndpointSlice controller. An object written in one second is seen by the next component one second later. That is a compressed stand-in for watch latency: real controllers read the API through informers that list and then watch , and a watch delivers later changes without missing any , but the delay is not a fixed second.

The ordering enforces the real division of labour. Controllers act by sending requests to the API server, not by touching things directly . The scheduler only binds and never starts containers , and once spec.nodeName is set that node's kubelet owns the Pod .

Documented defaults it uses

Behaviour — Value in the simulation — Source; Node marked Unknown after silence — 50 s — ; Default not-ready and unreachable toleration — 300 s — ; Container restart back-off — 10 s doubling, capped at 300 s — ; Back-off reset — After 10 minutes of clean running — ; Image pull back-off cap — 300 s — ; Probe period and failure threshold — 10 s and 3 — ; maxSurge and maxUnavailable — 25% each; surge rounds up, unavailable rounds down —

The model also follows documented rules about deletion. A ReplicaSet can create a replacement as soon as a Pod is marked for deletion , and Pods on an unreachable node stay Terminating until the kubelet confirms or someone force-deletes them .

What it simplifies or leaves out

• Scheduling. Filtering uses requests, node selectors, taints and cordoning, but scoring is a simplified, illustrative stand-in. The real scheduler combines weighted score plugins , with LeastAllocated as the default resource strategy , and breaks ties at random ; the simulation uses a seeded sequence so runs repeat.
• Probes. Strict 10 s ticks are an approximation, because the kubelet may probe a not-yet-ready container off schedule .
• Durations. Image pulls, container starts and graceful stops take fixed illustrative times; real graceful termination defaults to 30 s .
• Networking. The Service data plane is updated together with the EndpointSlice. Real nodes program their own rules when kube-proxy, or a replacement, sees the change .
• Scope. One namespace, one container per Pod and one Service. No authentication, admission or etcd quorum (the request and etcd labs cover those), no zones, no eviction rate limits , and no PodDisruptionBudgets.

Investigate it

Repeat an experiment on a real test cluster and compare the timings:

``bash kubectl create deployment web --image=nginx:1.27 --replicas=3 kubectl get pods -l app=web -w kubectl delete pod -l app=web --wait=false kubectl set image deployment/web nginx=nginx:1.28 kubectl rollout status deployment/web kubectl get events --sort-by=.metadata.creationTimestamp ``

Real clusters show the same order of events but different timing: watch delays, image pulls and probe schedules vary with the cluster, the runtime and the network.

30. Putting it all together

Every chapter so far has followed one part of the same story. Here it is end to end, from kubectl apply to Pods that serve traffic, scale and heal. No step calls the next one directly: each component watches the API, notices that desired and observed state differ, and does its own small part .

From apply to running

• The request. kubectl apply turns the manifest into an API request . It passes TLS, authentication, authorization, admission control, then validation and storage , and the object is written to etcd .
• The Deployment controller creates a ReplicaSet that it owns .
• The ReplicaSet controller creates Pods from its template until the count matches . They start Pending .
• The scheduler filters and scores nodes and binds each Pod to one; it does not start containers .
• The kubelet on that node creates the Pod sandbox and networking through the CRI , with a CNI plugin called by the runtime , pulls the image as its pull policy says , and starts the containers.
• Readiness. When the readiness probe passes, the Pod's Ready condition becomes true and it can be added to Services' load balancing .
• Serving. The EndpointSlice controller lists the matching Pods , only ready endpoints receive traffic , and kube-proxy, or a replacement, programs each node to redirect the Service's virtual IP . Clients find it by a DNS name such as web.default.svc.cluster.local .

Scaling

Changing replicas scales the same ReplicaSet and starts no rollout . The HPA does that for you from metrics on an interval , using desiredReplicas = ceil[currentReplicas × (currentMetricValue / desiredMetricValue)] . Pods that do not fit stay Pending ; new machines come only from a separate node autoscaler .

Healing

Healing happens at three levels, each by a different component:

Failure — Who reacts — What happens; A container exits — kubelet — Restarts it in place with an exponential back-off; A liveness probe fails — kubelet — Restarts the container on the same node; A Pod is deleted — ReplicaSet controller — Creates a replacement to close the gap; A node goes silent — Node controller, then taint eviction — Ready=Unknown after 50 s , eviction after the 300 s toleration , replacements scheduled elsewhere

You can replay the whole path, and break any step of it, in the playground.

A realistic case

A team ships a new version on a Friday. The image is pulled, the new Pods start, and the readiness probe never passes because a dependency is unreachable. Traffic stays on the old Pods, because only ready endpoints get traffic . After 600 s without progress the Deployment reports ProgressDeadlineExceeded , and kubectl rollout status exits non-zero, which can stop the pipeline . Nothing rolls back by itself ; someone runs kubectl rollout undo .

What Kubernetes does not guarantee

• Zero-downtime rollouts. Recreate causes downtime by design , and a RollingUpdate is only as safe as the readiness probes behind it .
• Automatic rollback. A stalled rollout is only reported .
• Moving a Pod. A Pod is never moved; a controller creates a replacement with a new UID , and anything tied to the old UID, such as an emptyDir, is gone .
• Healing machines. Kubernetes does not configure, maintain or heal hosts .
• One copy during a partition. Pods on a cut-off node may keep running while replacements run elsewhere .
• Exactly-once schedules. A CronJob may occasionally create two Jobs or none .
• Network isolation from a policy object. Without a controller that implements NetworkPolicy, a policy does nothing .
• Secret confidentiality. Secrets are unencrypted in etcd by default .
• Logs and events. There is no cluster-level logging , and events last an hour by default .
• A load balancer. A LoadBalancer Service needs one you provide or a cloud provider .
• Protection from involuntary disruption. A PodDisruptionBudget cannot block hardware loss or a kernel panic .

A production checklist

Configure — Why; CPU and memory requests on every container — The scheduler places Pods by requests, not usage ; utilization-based HPA needs them ; node autoscalers read them; Memory limits you have measured — Memory limits are enforced by OOM kills; Readiness probes — Keep unready Pods out of Service endpoints; Careful liveness and startup probes — A careless liveness probe can cause cascading failures ; slow starters need a startup probe; Several replicas spread across zones — Topology spread constraints and zone labels control spread; PodDisruptionBudgets — Limit voluntary disruptions; kubectl drain respects them; An HA control plane — At least three control plane nodes , each component across three zones when availability matters; Regular, encrypted etcd backups — Recovery from losing every control plane node depends on them; Least-privilege RBAC — Grant admin, edit and view per namespace , and keep Secret list and watch tightly held; Encryption at rest and Pod Security — Turn on encryption at rest and enforce a Pod Security level per namespace; Supported versions — Only the three most recent minors are maintained; No spec.replicas in manifests under an HPA — Otherwise every apply resets the count

Investigate it

``bash kubectl get deployments,replicasets,pods -o wide kubectl get pods -A -o custom-columns=NAME:.metadata.name,CPU:.spec.containers[].resources.requests.cpu,MEM:.spec.containers[].resources.requests.memory kubectl get pdb -A kubectl get hpa -A kubectl get nodes -L topology.kubernetes.io/zone kubectl auth can-i --list -n default kubectl rollout status deployment/web --timeout=10m ``

The checklist covers what core Kubernetes documents. Networking, storage, load balancers and node provisioning depend on the CNI plugin, CSI driver, cloud provider and autoscaler you choose, and their behaviour varies.

Related explainers

• NATS internals: a messaging system whose clustering, failover and JetStream replication sit on top of whatever platform runs the servers.
• PostgreSQL internals: the kind of stateful workload that needs the StatefulSet, storage and failure caveats in chapter 24 and chapter 22.
• How Netflix works: a large system whose own engineering posts describe its container platform.

Technologies

• Kubernetes: Open source platform for running containerized workloads and services; this study describes the 1.37 documentation.
• etcd: The consistent, highly available key-value store that holds all cluster data.
• containerd: A CRI container runtime; the kubelet connects to its socket by default.
• CoreDNS: The default implementation of cluster DNS.
• Docker: Builds and runs containers on one host; images it builds run on Kubernetes, which no longer talks to Docker Engine directly.

Sources

1. The Kubernetes API — Kubernetes documentation, undated (current docs, v1.37)
2. Authenticating — Kubernetes documentation, undated (current docs, v1.37)
3. Authorization — Kubernetes documentation, undated (current docs, v1.37)
4. Admission Control in Kubernetes — Kubernetes documentation, undated (current docs, v1.37)
5. Validating Admission Policy — Kubernetes documentation, undated (current docs, v1.37)
6. Dynamic Admission Control — Kubernetes documentation, undated (current docs, v1.37)
7. Operating etcd clusters for Kubernetes — Kubernetes documentation, undated (current docs, v1.37)
8. FAQ — etcd documentation, undated (etcd v3.7 docs)
9. Encrypting Confidential Data at Rest — Kubernetes documentation, undated (current docs, v1.37)
10. Secrets — Kubernetes documentation, undated (current docs, v1.37)
11. Controllers — Kubernetes documentation, undated (current docs, v1.37)
12. Kubernetes API Concepts — Kubernetes documentation, undated (current docs, v1.37)
13. kube-controller-manager — Kubernetes documentation, undated (current docs, v1.37)
14. Leases — Kubernetes documentation, undated (current docs, v1.37)
15. Garbage Collection — Kubernetes documentation, undated (current docs, v1.37)
16. Use Cascading Deletion in a Cluster — Kubernetes documentation, undated (current docs, v1.37)
17. ReplicaSet — Kubernetes documentation, undated (current docs, v1.37)
18. Operator pattern — Kubernetes documentation, undated (current docs, v1.37)
19. Custom Resources — Kubernetes documentation, undated (current docs, v1.37)
20. ObjectMeta — Kubernetes documentation, undated (current docs, v1.37)
21. Deployment (API reference) — Kubernetes documentation, undated (current docs, v1.37)
22. Objects In Kubernetes — Kubernetes documentation, undated (current docs, v1.37)
23. Server-Side Apply — Kubernetes documentation, undated (current docs, v1.37)
24. Data model — etcd documentation, undated (etcd v3.7 docs)
25. API Overview — Kubernetes documentation, undated (current docs, v1.37)
26. Cluster Architecture — Kubernetes documentation, undated (current docs, v1.37)
27. Communication between Nodes and the Control Plane — Kubernetes documentation, undated (current docs, v1.37)
28. kube-apiserver — Kubernetes documentation, undated (current docs, v1.37)
29. Declarative Management of Kubernetes Objects Using Configuration Files — Kubernetes documentation, undated (current docs, v1.37)
30. Controlling Access to the Kubernetes API — Kubernetes documentation, undated (current docs, v1.37)
31. Overview — Kubernetes documentation, undated (current docs, v1.37)
32. Nodes — Kubernetes documentation, undated (current docs, v1.37)
33. Node Status — Kubernetes documentation, undated (current docs, v1.37)
34. Container Runtime Interface (CRI) — Kubernetes documentation, undated (current docs, v1.37)
35. Container Runtimes — Kubernetes documentation, undated (current docs, v1.37)
36. Node Autoscaling — Kubernetes documentation, undated (current docs, v1.37)
37. Pods — Kubernetes documentation, undated (current docs, v1.37)
38. Pod Lifecycle — Kubernetes documentation, undated (current docs, v1.37)
39. Borg: The Predecessor to Kubernetes — Kubernetes Blog, 2015-04-23
40. Init Containers — Kubernetes documentation, undated (current docs, v1.37)
41. Sidecar Containers — Kubernetes documentation, undated (current docs, v1.37)
42. Resource Management for Pods and Containers — Kubernetes documentation, undated (current docs, v1.37)
43. Containers — Kubernetes documentation, undated (current docs, v1.37)
44. Images — Kubernetes documentation, undated (current docs, v1.37)
45. Don't Panic: Kubernetes and Docker — Kubernetes Blog, 2020-12-02
46. Updated: Dockershim Removal FAQ — Kubernetes Blog, 2022-02-17
47. Docker Engine — Docker documentation, undated
48. Docker Compose — Docker documentation, undated
49. How Compose works — Docker documentation, undated
50. Why use Compose? — Docker documentation, undated
51. Use Compose in production — Docker documentation, undated
52. Overview of Compose Bridge — Docker documentation, undated
53. Releases — Kubernetes documentation, undated (current docs, v1.37)
54. Object Names and IDs — Kubernetes documentation, undated (current docs, v1.37)
55. Labels and Selectors — Kubernetes documentation, undated (current docs, v1.37)
56. Namespaces — Kubernetes documentation, undated (current docs, v1.37)
57. Kubernetes Object Management — Kubernetes documentation, undated (current docs, v1.37)
58. Kubernetes Components — Kubernetes documentation, undated (current docs, v1.37)
59. Services, Load Balancing, and Networking — Kubernetes documentation, undated (current docs, v1.37)
60. Ingress — Kubernetes documentation, undated (current docs, v1.37)
61. Gateway API — Kubernetes documentation, undated (current docs, v1.37)
62. Introduction — Gateway API (Kubernetes SIG Network), undated
63. Getting started with Gateway API — Gateway API (Kubernetes SIG Network), undated
64. Network Policies — Kubernetes documentation, undated (current docs, v1.37)
65. Volumes — Kubernetes documentation, undated (current docs, v1.37)
66. Ephemeral Volumes — Kubernetes documentation, undated (current docs, v1.37)
67. Persistent Volumes — Kubernetes documentation, undated (current docs, v1.37)
68. Storage Classes — Kubernetes documentation, undated (current docs, v1.37)
69. Dynamic Volume Provisioning — Kubernetes documentation, undated (current docs, v1.37)
70. Cluster Networking — Kubernetes documentation, undated (current docs, v1.37)
71. ConfigMaps — Kubernetes documentation, undated (current docs, v1.37)
72. Resource Quotas — Kubernetes documentation, undated (current docs, v1.37)
73. Service — Kubernetes documentation, undated (current docs, v1.37)
74. EndpointSlices — Kubernetes documentation, undated (current docs, v1.37)
75. Virtual IPs and Service Proxies — Kubernetes documentation, undated (current docs, v1.37)
76. DNS for Services and Pods — Kubernetes documentation, undated (current docs, v1.37)
77. Using CoreDNS for Service Discovery — Kubernetes documentation, undated (current docs, v1.37)
78. Customizing DNS Service — Kubernetes documentation, undated (current docs, v1.37)
79. Using RBAC Authorization — Kubernetes documentation, undated (current docs, v1.37)
80. Cloud Native Security and Kubernetes — Kubernetes documentation, undated (current docs, v1.37)
81. Debug Pods — Kubernetes documentation, undated (current docs, v1.37)
82. Debug Running Pods — Kubernetes documentation, undated (current docs, v1.37)
83. Determine the Reason for Pod Failure — Kubernetes documentation, undated (current docs, v1.37)
84. Ephemeral Containers — Kubernetes documentation, undated (current docs, v1.37)
85. Event — Kubernetes documentation, undated (current docs, v1.37)
86. kubectl events — Kubernetes documentation, undated (current docs, v1.37)
87. Logging Architecture — Kubernetes documentation, undated (current docs, v1.37)
88. Resource metrics pipeline — Kubernetes documentation, undated (current docs, v1.37)
89. Metrics For Kubernetes System Components — Kubernetes documentation, undated (current docs, v1.37)
90. Traces For Kubernetes System Components — Kubernetes documentation, undated (current docs, v1.37)
91. Debug Services — Kubernetes documentation, undated (current docs, v1.37)
92. Troubleshooting Clusters — Kubernetes documentation, undated (current docs, v1.37)
93. Production environment — Kubernetes documentation, undated (current docs, v1.37)
94. Options for Highly Available Topology — Kubernetes documentation, undated (current docs, v1.37)
95. Creating Highly Available Clusters with kubeadm — Kubernetes documentation, undated (current docs, v1.37)
96. kube-scheduler — Kubernetes documentation, undated (current docs, v1.37)
97. User Impersonation — Kubernetes documentation, undated (current docs, v1.37)
98. Version Skew Policy — Kubernetes documentation, undated (current docs, v1.37)
99. Kubernetes Release Cycle — Kubernetes documentation, undated (current docs, v1.37)
100. Patch Releases — Kubernetes documentation, undated (current docs, v1.37)
101. Considerations for large clusters — Kubernetes documentation, undated (current docs, v1.37)
102. Running in multiple zones — Kubernetes documentation, undated (current docs, v1.37)
103. kubectl auth can-i — Kubernetes documentation, undated (current docs, v1.37)
104. Service Accounts — Kubernetes documentation, undated (current docs, v1.37)
105. Pod Security Standards — Kubernetes documentation, undated (current docs, v1.37)
106. Pod Security Admission — Kubernetes documentation, undated (current docs, v1.37)
107. Configure a Security Context for a Pod or Container — Kubernetes documentation, undated (current docs, v1.37)
108. Good practices for Kubernetes Secrets — Kubernetes documentation, undated (current docs, v1.37)
109. Deployments — Kubernetes documentation, undated (current docs, v1.37)
110. Assign Memory Resources to Containers and Pods — Kubernetes documentation, undated (current docs, v1.37)
111. CHANGELOG/CHANGELOG-1.32.md — Kubernetes source, undated (master branch)
112. Taints and Tolerations — Kubernetes documentation, undated (current docs, v1.37)
113. Force Delete StatefulSet Pods — Kubernetes documentation, undated (current docs, v1.37)
114. Node Shutdowns — Kubernetes documentation, undated (current docs, v1.37)
115. StatefulSets — Kubernetes documentation, undated (current docs, v1.37)
116. DaemonSet — Kubernetes documentation, undated (current docs, v1.37)
117. Jobs — Kubernetes documentation, undated (current docs, v1.37)
118. CronJob — Kubernetes documentation, undated (current docs, v1.37)
119. kubelet — Kubernetes documentation, undated (current docs, v1.37)
120. Horizontal Pod Autoscaling — Kubernetes documentation, undated (current docs, v1.37)
121. Vertical Pod Autoscaling — Kubernetes documentation, undated (current docs, v1.37)
122. Liveness, Readiness, and Startup Probes — Kubernetes documentation, undated (current docs, v1.37)
123. Kubernetes Scheduler — Kubernetes documentation, undated (current docs, v1.37)
124. Static Pods — Kubernetes documentation, undated (current docs, v1.37)
125. Create static Pods — Kubernetes documentation, undated (current docs, v1.37)
126. Container Lifecycle Hooks — Kubernetes documentation, undated (current docs, v1.37)
127. Scheduling Framework — Kubernetes documentation, undated (current docs, v1.37)
128. Network Plugins — Kubernetes documentation, undated (current docs, v1.37)
129. Assigning Pods to Nodes — Kubernetes documentation, undated (current docs, v1.37)
130. Pod Topology Spread Constraints — Kubernetes documentation, undated (current docs, v1.37)
131. Pod Priority and Preemption — Kubernetes documentation, undated (current docs, v1.37)
132. Pod Quality of Service Classes — Kubernetes documentation, undated (current docs, v1.37)
133. Node-pressure Eviction — Kubernetes documentation, undated (current docs, v1.37)
134. API-initiated Eviction — Kubernetes documentation, undated (current docs, v1.37)
135. Disruptions — Kubernetes documentation, undated (current docs, v1.37)
136. Scheduler Configuration — Kubernetes documentation, undated (current docs, v1.37)
137. Limit Ranges — Kubernetes documentation, undated (current docs, v1.37)
138. kube-scheduler Configuration (v1) — Kubernetes documentation, undated (current docs, v1.37)
139. Scheduler Performance Tuning — Kubernetes documentation, undated (current docs, v1.37)
140. Resource Bin Packing — Kubernetes documentation, undated (current docs, v1.37)
141. Binding — Kubernetes documentation, undated (current docs, v1.37)
142. Pod — Kubernetes documentation, undated (current docs, v1.37)