How Kubernetes Coordinates Through API State
Follow one Deployment through accepted intent, independent reconciliation, placement, node execution and readiness—then break the system and diagnose the first stalled boundary.
One shared exchange, many independent actors
Who actually talks to kube-apiserver?
Hover an actor to preview · click to pinTap an actor to inspect
One Kubernetes cluster: clients above, kube-apiserver at the center, etcd behind it, reconcilers and scheduler beside it, and kubelets with running Pods below.
Try controller manager, scheduler, kubelet or an extension. Each uses the API for the state it owns; etcd remains behind the API server.
kubectl is one client. Controllers, scheduler, kubelets and extensions also use the Kubernetes API to observe or change cluster state. etcd sits behind the API server as durable backing state.
Context: Special API-server-initiated and external-provider calls exist; ordinary application traffic is a different path.
A request must pass the right gates before state changes
Where can an API request stop before an object is stored?
Hover an actor to preview · click to pinTap an actor to inspect
One Kubernetes cluster: clients above, kube-apiserver at the center, etcd behind it, reconcilers and scheduler beside it, and kubelets with running Pods below.
- 01TLS / API routeREQUEST
- 02AuthenticationWHO?
- 03AuthorizationMAY?
- 04Mutating admissionCHANGE?
- 05Validating admissionALLOW?
- 06Resource processingVALID?
- 07PersistenceCOMMIT
Request enters the API route. No Deployment is stored yet.
Teaching pipeline: authentication → authorization → applicable admission → persistence. Ordinary reads bypass admission; a rejected write never reaches etcd.
Authentication identifies the caller. Authorization evaluates this operation. Applicable admission can mutate or reject an object-changing request before persistence.
Context: The rail is conceptual; ordinary GET/LIST/WATCH bypass admission.
Controllers follow API changes instead of hammering etcd
How does a controller learn that a Deployment changed?
Hover an actor to preview · click to pinTap an actor to inspect
One Kubernetes cluster: clients above, kube-apiserver at the center, etcd behind it, reconcilers and scheduler beside it, and kubelets with running Pods below.
- 01 LISTDeployment/demo/web · resourceVersion 10245
- 02 WATCHKeep a change stream from that version
- 03 EVENTMODIFIED · resourceVersion 11020
- 04 RECONCILEQueue demo/web · compare desired with observed
Show client internals
Reflector → local informer cache → handlers → work queue → Reconcile(). The cache can be temporarily behind API state; a logical decision need not issue another GET.
Show server internals
API handlers, watch cache and storage interface sit behind the served API. A WATCH event should not be drawn as a separate physical etcd request every time.
If historical changes are unavailable, `410 Gone` tells a client to clear/re-LIST and restart its watch.
LIST establishes initial state and a resourceVersion. WATCH streams later changes. Client informers and server caches mean a logical decision need not fetch every object directly from etcd.
Context: A too-old watch can require a new LIST after 410 Gone.
Objects persist; reconcilers repair relationships
Why does deleting a managed Pod make another Pod appear?
Hover an actor to preview · click to pinTap an actor to inspect
One Kubernetes cluster: clients above, kube-apiserver at the center, etcd behind it, reconcilers and scheduler beside it, and kubelets with running Pods below.
Desired count and existing active Pods match. Reconciliation can finish without changing API state.
Deployment/web → ReplicaSet/web-7c9 → Pods [web-a, web-b, web-c] desired: 3 · existing: 3 · ready: 3
Identity boundary: restarting a container keeps the same Pod UID. Deleting a managed Pod removes that object, so its controller creates a replacement with a new name and UID.
Deployment, ReplicaSet and Pod are separate API objects linked by selectors and owner references. Each controller compares its slice of observed state with desired state and may decide that no write is needed.
Context: A container restart keeps the same Pod identity. Pod deletion or loss is a different recovery path.
Follow one conversation across the same cluster map
What happens after web is accepted through the API?
Hover an actor to preview · click to pinTap an actor to inspect
One Kubernetes cluster: clients above, kube-apiserver at the center, etcd behind it, reconcilers and scheduler beside it, and kubelets with running Pods below.
Does applying replicas: 3 mean three containers exist already?
Alice sends desired replicas: 3 through the Kubernetes API.
Current API operation and object fields
CREATE Deploymentkind: Deployment
metadata: {name: web, namespace: demo}
spec: {replicas: 3}The actors stay put while the selected operation and accumulated object state change. Deployment playback is timed; every conversation supports direct manual inspection.
All cluster conversations
01 · Create a Deployment
Does applying replicas: 3 mean three containers exist already? Multiple reconcilers, scheduler and kubelets turn one stored intent into running Pods.
- Apply · CREATE Deployment — Alice sends desired replicas: 3 through the Kubernetes API.
- Authenticate · AUTHENTICATE — The API server establishes Alice’s identity.
- Authorize · AUTHORIZE create — The authorizer allows this specific create operation.
- Admission · ADMISSION if matched — Applicable mutating then validating admission runs before persistence.
- Store desired state · PERSIST — The accepted Deployment spec is durable. No web container is implied yet.
- Observe Deployment · WATCH Deployments — The Deployment controller observes the new desired object.
- Create ReplicaSet · CREATE ReplicaSet — Deployment reconciliation creates a ReplicaSet for the Pod template.
- Compare replicas · WATCH ReplicaSets/Pods — ReplicaSet wants three Pods and currently sees none.
- Create Pods · CREATE Pods — ReplicaSet reconciliation creates three Pod API objects.
- Select Nodes · WATCH unscheduled Pods — Scheduler filters and scores candidate Nodes; it does not start containers.
- Bind Pods · BIND Pods — Scheduler records its placement decision through the API.
- Execute on Node · WATCH assigned Pods — Kubelet observes the Pod assigned to worker-a, then invokes node-local runtime work.
- Start containers · CRI create/start — The runtime starts containers under kubelet’s node-local reconciliation.
- Report and converge · PATCH Pod/status — Kubelets report Pod conditions; controllers see observed availability reach three in this fixture.
This example assumes requests are allowed and Pods become Ready; timing is illustrative.
02 · Roll out image v2
Does updating a Deployment edit its existing Pods in place? A changed Pod template produces a new ReplicaSet; rollout strategy controls the overlap while readiness gates progress.
- Update template · PATCH Deployment — Alice changes the Pod template image from web:v1 to web:v2.
- Observe generation · WATCH Deployment — Deployment controller sees a template it has not yet observed.
- Create new ReplicaSet · CREATE ReplicaSet — A new ReplicaSet owns Pods created from the v2 template; the v1 ReplicaSet still exists.
- Surge one Pod · CREATE Pod — The rollout creates one v2 Pod within this fixture’s surge limit.
- Make v2 ready · PATCH Pod/status — Kubelet reports the new Pod Ready after node execution and probes.
- Scale old set down · DELETE old Pod — Only after new capacity is available does the controller reduce the old ReplicaSet in this fixture.
- Converge on v2 · PATCH Deployment/status — The new ReplicaSet reaches three ready Pods and the old ReplicaSet reaches zero desired replicas.
The exact overlap depends on replicas, maxSurge, maxUnavailable, readiness and termination timing.
03 · Delete a managed Pod
Why does a Pod come back with a different name? Deleting a managed Pod removes that object; ReplicaSet reconciliation creates a replacement to restore its desired count.
- Delete Pod · DELETE Pod — Alice requests deletion of one Pod owned by the web ReplicaSet.
- Terminate gracefully · OBSERVE deletionTimestamp — Kubelet stops the Pod using its configured termination behavior; the object is not an instruction to restart its container.
- Observe replica gap · WATCH Pods — ReplicaSet controller sees two active Pods for a desired count of three.
- Create replacement · CREATE Pod — The controller creates a new Pod object with a new UID.
- Place and execute · BIND Pod — Scheduler records placement; the selected kubelet separately starts the Pod.
- Report ready · PATCH Pod/status — The replacement becomes Ready and restores availability.
The replacement is a new Pod and can be scheduled to a different Node.
04 · Resolve an API update conflict
What happens when two clients update the same old object version? Optimistic concurrency rejects a stale update; the client must read current state and retry without erasing another writer’s change.
- Two clients read · GET Deployment — Alice and automation both read the same object version.
- Automation writes · UPDATE replicas=5 — Automation writes first; the accepted object advances to a new resourceVersion.
- Alice writes stale copy · UPDATE image=v2 — Alice submits an update based on the old version.
- Read current state · GET Deployment — Alice refetches the object and sees the other writer’s replica change.
- Retry deliberately · UPDATE image=v2 — Alice retries against current state while preserving replicas=5.
Server-side apply adds field ownership and conflict rules; it is related but distinct from an ordinary stale resourceVersion update.
05 · Schedule a Pod
Does scheduler tell a kubelet to start the Pod? Scheduler records binding; kubelet separately observes the assigned Pod.
- Pending Pod · CREATE Pod — Pod exists with no nodeName.
- Observe · WATCH Pods/Nodes — Scheduler learns about unscheduled Pods and cached Node inputs.
- Filter and score · FILTER / SCORE — Scheduler chooses a suitable Node from scheduling inputs.
- Bind · BIND Pod — Placement is recorded via the Kubernetes API.
- Kubelet observes · WATCH assigned Pod — The kubelet learns its assigned Pod; scheduler did not call it.
Real scheduling also considers capacity, taints, affinity, topology and storage.
06 · Start a Pod on a Node
Who turns an assigned PodSpec into containers? Kubelet drives node-local reality after observing assigned API state.
- Pod assigned · BIND Pod — The API records spec.nodeName.
- Kubelet watch · WATCH Pod — Kubelet observes the Pod assigned to its Node.
- Prepare Node · SET UP network/volumes — Kubelet coordinates required node-local preparations.
- CRI execution · CRI StartContainer — Kubelet asks the local runtime to start containers.
- Report · PATCH Pod/status — Observed container state returns through the API.
CNI, CSI, Secrets and probes can add steps; this is the core route.
07 · Pod Status Reporting
How does actual node state get back to controllers? Kubelet reports Pod status through API; controllers observe it and compare with intent.
- Runtime observes · CONTAINER state — The runtime has started the container.
- Kubelet checks · PROBE / OBSERVE — Kubelet observes phase and readiness conditions.
- Status update · PATCH Pod/status — Pod status is updated through the API.
- Controller reacts · WATCH Pod/status — Controller cache observes the new actual state.
Running phase is distinct from readiness.
08 · Node Joins Cluster
How does a new worker become visible to scheduler? Kubelet registers a Node API object; control-plane components observe it.
- New worker · START kubelet — Kubelet starts with valid client credentials.
- Register Node · CREATE Node — Kubelet registers its Node through the API.
- Report capacity · PATCH Node/status — Status includes addresses, capacity and conditions.
- Observe Node · WATCH Nodes — Scheduler and controllers can use the new Node state.
Bootstrap and authorization steps vary by cluster.
09 · Node Heartbeat
What tells the control plane a Node is still present? Node status and lightweight Lease renewals are related heartbeat signals.
- Node status · PATCH Node/status — Kubelet periodically reports Node conditions.
- Lease renewal · UPDATE Lease — Kubelet updates Lease spec.renewTime.
- Later renewal · UPDATE Lease — A later renewal keeps the liveness signal fresh.
- Lifecycle observes · WATCH Node/Lease — Node lifecycle logic can compare Node condition and Lease freshness.
Renewal timestamps shown are illustrative, not a universal cadence.
10 · Node Failure
Does one missed heartbeat instantly recreate every Pod? Node lifecycle and workload reconcilers react after detecting degraded Node state; replacement is conditional and takes time.
- Power loss · NODE unavailable — Kubelet on worker-b stops renewing its Lease.
- Lease ages · WATCH Lease — Control-plane logic notices sustained absence rather than one missed renewal.
- Condition/taint · UPDATE Node/status — Ready may become Unknown or False; lifecycle handling can taint Node.
- Workload gap · WATCH Pods/ReplicaSet — Affected Pods and desired replica count create a reconciliation gap.
- Replacement · CREATE replacement Pod — A managed workload may get a replacement after lifecycle handling.
- Healthy placement · BIND Pod — Scheduler selects a healthy Node for replacement.
Taint, eviction and replacement behavior depend on configured timings and workload ownership.
11 · Service + EndpointSlice
Who turns selected Pods into Service backends? EndpointSlice controller writes backend state through API; dataplane consumers use it later.
- Declare Service · CREATE Service — Selector app=web and port 80 enter API state.
- Observe members · WATCH Service/Pods — EndpointSlice controller observes matching Pods and conditions.
- Write slice · CREATE EndpointSlice — Controller records eligible backend addresses.
- Proxy observes · WATCH EndpointSlices — A Service forwarding implementation sees updated backends.
Selectors and endpoint readiness matter; slices can be multiple and endpoints may temporarily duplicate.
12 · Service Proxy Programming
How do node forwarding rules learn Service destinations? A Service proxy watches API objects and configures node dataplane rules; implementations can differ.
- Watch Service · WATCH Services — Service proxy observes virtual Service configuration.
- Watch backends · WATCH EndpointSlices — It learns currently eligible backend addresses.
- Program Node · SYNC forwarding rules — iptables, nftables or a replacement implementation configures forwarding.
- Serve traffic · HTTP via Service — Application packet takes a workload path, not an API request path.
kube-proxy is optional when another implementation replaces it.
13 · ConfigMap / Secret
How does referenced application data reach a container? Kubelet obtains referenced API data and materializes it for its assigned Pod.
- Pod references · CREATE Pod — PodSpec references ConfigMap web-config and Secret web-creds.
- Kubelet reads · GET ConfigMap/Secret — Kubelet obtains required API objects.
- Materialize · PROJECT / SET ENV — Kubelet prepares mounted data or environment setup.
- Container starts · CRI StartContainer — Runtime starts the app with prepared inputs.
Environment values are captured at container start; projected/mounted data update behaviour depends on kubelet and volume type.
14 · ServiceAccount API Call
Can a Pod itself be a Kubernetes API client? A workload can send an authenticated API request with a projected ServiceAccount token.
- Projected identity · PROJECT token — Kubelet makes a bound token available to the Pod.
- Call API · GET Pods — Pod client sends bearer token to the Kubernetes API Service.
- Authenticate · AUTHENTICATE token — API server establishes ServiceAccount identity.
- Authorize · AUTHORIZE get — RBAC or configured authorizer checks this operation.
- Read response · RETURN PodList — If allowed, Pod receives API data; ordinary read skips admission.
A token authenticates identity; RBAC still controls each operation.
15 · Horizontal Pod Autoscaler
Does HPA create Pods itself? HPA updates target scale through API; the usual Deployment and ReplicaSet loops create the Pods.
- Metrics available · SERVE metrics.k8s.io — Resource metrics are available through an API.
- HPA reads · GET metrics/scale — HPA controller compares current metric with target.
- Write scale · UPDATE /scale — HPA changes desired replicas from 3 to 8 through API.
- Normal loops · WATCH Deployment/ReplicaSet — Deployment and ReplicaSet reconcilers handle new Pod count.
- Create Pods · CREATE Pods — ReplicaSet creates Pods; scheduler and kubelets then handle placement and execution.
Metrics availability, stabilization and scaling policies affect the actual change.
16 · PersistentVolume / CSI
Where does a PVC become a real volume and mount? API objects coordinate CSI controller provisioning and kubelet/CSI node mounting, with a direct storage-provider call.
- Declare PVC · CREATE PVC — The claim and StorageClass enter API state.
- Provisioner watch · WATCH PVC — CSI external provisioner sees a claim matching its driver.
- Create volume · CSI CreateVolume — Controller-side CSI calls its driver/provider, outside Kubernetes API.
- Write PV · CREATE PV / bind PVC — A PV and binding state represent the external volume.
- Schedule Pod · BIND Pod — A Pod consuming the claim is assigned with topology constraints considered.
- Mount on Node · NodeStage/NodePublish — Kubelet works with CSI node plugin to make the volume available to container.
WaitForFirstConsumer can defer provisioning until scheduling context exists; attach applies only where needed.
17 · Service type LoadBalancer
Who creates the external load balancer? A provider controller watches Service intent, calls a cloud API directly, then reports status through Kubernetes API.
- Declare Service · CREATE Service — Service type LoadBalancer enters API state.
- Cloud watch · WATCH Service — Cloud/provider controller observes desired Service.
- External call · CREATE load balancer — Controller calls external cloud API directly.
- Report status · PATCH Service/status — Controller records ingress address when provider reports it.
- Serve traffic · HTTP via LB/Service — Ordinary application packets use load balancer and Service dataplane.
Availability and fields vary by provider and controller implementation.
18 · Ingress / Gateway Controller
How does an Ingress or HTTPRoute become a live route? A controller observes API route objects and configures a separate proxy or provider dataplane.
- Declare route · CREATE HTTPRoute — Route object refers to the web Service.
- Controller watch · WATCH Gateway/HTTPRoute — Gateway controller observes route and attachment state.
- Configure proxy · CONFIGURE Envoy/LB — Implementation configures its proxy or cloud dataplane.
- Report status · PATCH HTTPRoute/status — Controller reports accepted/programmed conditions.
Gateway and Ingress implementations differ; the API object alone is not the traffic path.
19 · Operator / CRD
Do Operators use a different coordination architecture? Operators reconcile custom API resources by the same observe/compare/write pattern.
- Custom intent · CREATE DatabaseCluster — Custom resource enters versioned Kubernetes API state.
- Operator watch · WATCH DatabaseCluster — Operator learns desired database specification.
- Create dependents · CREATE StatefulSet/PVC — Operator writes dependent Kubernetes objects through API.
- Observe state · WATCH dependents — Operator compares observed dependent state.
- Report status · PATCH custom/status — Status and conditions report reality back through API.
A CRD defines an API type; any external database/provider call is an explicit additional edge.
20 · Admission Webhook
Is an admission webhook a controller watching objects? kube-apiserver calls a matching webhook during request processing; its answer can allow, deny or mutate.
- Client write · CREATE Pod — A Pod create request arrives at API server.
- Gates pass · AUTHN / AUTHZ — Identity and permission checks succeed.
- Call webhook · AdmissionReview — API server initiates a direct call to matched admission webhook.
- Return decision · ALLOW / PATCH / DENY — Webhook may allow, mutate or reject before persistence.
- Continue or stop · PERSIST if allowed — Only an allowed request continues to durable API state.
Timeout outcome depends on failurePolicy; webhooks are optional extension points.
21 · kubectl logs
Does the API server ever initiate a call to kubelet? For Pod logs the API server can contact the kubelet HTTPS endpoint; this is a named exception.
- Ask for logs · GET Pod/log — kubectl requests the log subresource through API.
- Kubelet path · PROXY logs — API server initiates an HTTPS request to kubelet endpoint.
- Read container · READ logs — Kubelet obtains node-local container logs.
- Return stream · RETURN logs — Response returns through API client connection.
Control-plane-to-node networking and kubelet serving-certificate verification deserve operational care.
22 · exec / attach / port-forward
What happens behind a special Pod connection subresource? Client asks API server; API server uses a special kubelet path for exec, attach or port-forward.
- Client request · CONNECT Pod/exec — kubectl exec requests a special Pod subresource.
- Special node path · PROXY exec/attach/portforward — API server initiates a kubelet connection for the selected special operation.
- Node process · EXEC / ATTACH — Kubelet/runtime handle the node-local process or forwarding work.
- Bidirectional stream · STREAM response — Interactive data returns through the API path.
Exact streaming transport/proxy implementation can vary; ordinary application HTTP is separate.
23 · Leader Election
Why do controller-manager or scheduler replicas use Leases? A Lease coordinates an active reconciler; API servers are not modeled as one elected serving leader.
- Two replicas · GET Lease — Controller-manager A and B can reach API.
- A renews · UPDATE Lease — A keeps Lease ownership fresh.
- A fails · PROCESS unavailable — Renewals stop while other components can continue.
- B acquires · UPDATE Lease — After expiry, B can acquire leadership and reconcile.
Leader renewal and failover timing are configured, not fixed here.
24 · API Aggregation / Metrics
Is every API group implemented inside kube-apiserver? APIService can register an extension API; kube-apiserver proxies the claimed path to its aggregated server.
- Discover API · GET /apis — Client discovers a registered API group.
- Request metrics · GET metrics.k8s.io — Client calls the normal Kubernetes API endpoint.
- Proxy extension · PROXY APIService path — Aggregation layer delegates request to extension API server.
- Metric response · READ metrics — Example extension API obtains provider metrics.
- Return result · RETURN API response — Response reaches client through kube-apiserver.
An unavailable extension API need not imply the core API is unavailable.
25 · API Server Failure
Do existing containers die when API serving stops? Existing Pods can keep running while API reads/writes and new coordination are impaired.
- Serving lost · STOP API — No API replica can serve new requests.
- Client blocked · GET / APPLY — kubectl calls fail; no new desired writes are accepted.
- Loops impaired · WATCH / WRITE — Controllers and scheduler cannot coordinate fresh state.
- Nodes continue · KEEP RUNNING — Existing node-local workloads can continue serving.
- Recover endpoint · RESTORE serving — When API service returns, clients relist/reconnect and loops resume.
Node-local runtime and dataplane continuity are conditional on other healthy dependencies.
26 · HA API Servers
Must one API server leader handle all clients? Multiple API replicas may serve concurrently behind a stable endpoint; etcd and endpoint health remain dependencies.
- Endpoint routes · HTTPS Kubernetes API — Clients use one endpoint for API access.
- Concurrent serve · ROUTE requests — Different replicas can serve requests concurrently.
- One fails · API2 unavailable — One replica fails; health checks remove it from serving pool.
- State backing · ACCESS durable state — Serving replicas still rely on backing state and endpoint health.
No fixed quorum survival claim follows from API replica count alone.
Each selected stage names its actor, API operation and object change. Unrelated actors stay in place and muted.
Context: Stage timing and exact workload readiness are illustrative.
A Pod passes several independent boundaries before it serves traffic
Why can a Pod exist but remain Pending, Running or unready?
Hover an actor to preview · click to pinTap an actor to inspect
One Kubernetes cluster: clients above, kube-apiserver at the center, etcd behind it, reconcilers and scheduler beside it, and kubelets with running Pods below.
Scheduler records the binding. Kubelet starts the container and reports readiness separately.
kubectl get pod web-a -o wide web-a 1/1 Running 0 worker-a PodScheduled=True · Ready=True
This fixture isolates four inputs. Real scheduling evaluates more plugins and node execution can also wait on CNI, CSI, Secrets, init containers and runtime health.
Scheduler explains placement from declared requests and constraints. Kubelet then handles node-local preparation, container start and probes. Readiness controls endpoint eligibility without deleting the Pod.
Context: The fixture models a small subset of scheduling and node-execution inputs so every outcome remains explainable.
Inspect each actor’s scope and API relationship
What does this component read, watch, write and why?
Hover an actor to preview · click to pinTap an actor to inspect
One Kubernetes cluster: clients above, kube-apiserver at the center, etcd behind it, reconcilers and scheduler beside it, and kubelets with running Pods below.
The controller manager hosts multiple reconciliation loops: Deployment, ReplicaSet, Node lifecycle, EndpointSlice, HPA and others — each independently observes and acts through API state. A kubelet's scope is its own Node; it never instructs other Nodes or issues commands to controllers. Inspect each actor to see the exact read/watch/write boundary it owns.
Read the component relationship inventory
- Human / automation clients
- READS: Deployments, Pods, Service status. WATCHES: Optional client watches. WRITES: Deployment and other desired objects. WHY: Declare, inspect or automate cluster state..
- kube-apiserver
- READS: Backing state and serving caches. WATCHES: Storage changes for serving API watches. WRITES: Validated accepted objects to backing storage. WHY: Provide one authenticated, authorized, policy-aware state surface..
- etcd
- READS: Committed cluster state. WATCHES: Internal storage changes. WRITES: Durable API object updates. WHY: Remember accepted desired and observed state..
- Controller manager
- READS: Deployments, ReplicaSets, Pods, Nodes and Services. WATCHES: Relevant API objects through client caches. WRITES: ReplicaSets, Pods, EndpointSlices and status. WHY: Move observed state toward desired state..
- Scheduler
- READS: Candidate Pods, Nodes, resources and constraints. WATCHES: Unscheduled Pods and scheduling inputs. WRITES: Pod binding / nodeName through API. WHY: Choose placement; it does not start containers..
- kubelet · worker-a
- READS: Assigned Pods, referenced ConfigMaps/Secrets. WATCHES: Pods assigned to worker-a. WRITES: Pod status, Node status, Node Lease. WHY: Make assigned Pod state real on worker-a..
- kubelet · worker-b
- READS: Assigned Pods. WATCHES: Pods assigned to worker-b. WRITES: Pod status, Node status, Node Lease. WHY: Run the assigned workload within worker-b..
- Container runtime
- READS: Node-local runtime configuration. WATCHES: Node-local container events. WRITES: Node-local container state. WHY: Execute containers, outside the shared Kubernetes API object model..
- web Pods
- READS: Mounted/projected application data. WATCHES: Application traffic. WRITES: Application responses. WHY: Serve the workload, not coordinate the whole cluster..
- Service proxy
- READS: Services and EndpointSlices. WATCHES: Service and EndpointSlice changes. WRITES: Node-local forwarding rules. WHY: Make Service traffic reach eligible backends..
- CSI controller / node plugin
- READS: PVC, PV, StorageClass, attachment objects. WATCHES: Relevant PVC and volume changes. WRITES: PV and attachment-related API objects. WHY: Translate desired storage into provider and node mount operations.. EXTERNAL CALL: CSI CreateVolume / storage provider API.
- Cloud controller
- READS: Services and Nodes. WATCHES: Provider-relevant API objects. WRITES: Service/Node status. WHY: Reconcile Kubernetes intent with external infrastructure.. EXTERNAL CALL: Cloud provider API.
- External provider
- READS: Provider requests. WATCHES: Provider state. WRITES: Load balancer or volume resources. WHY: Supply external infrastructure required by the workload..
- Operator / custom controller
- READS: DatabaseCluster and dependent objects. WATCHES: Custom-resource changes. WRITES: StatefulSet, Service, PVC and status. WHY: Extend Kubernetes with another observer/compare/act loop.. EXTERNAL CALL: Optional provider/database API.
- Admission webhook
- READS: AdmissionReview input. WATCHES: Not a normal controller watch in this flow. WRITES: Allow, deny or mutation patch response. WHY: Extend request-time policy; not a direct client of etcd..
- Metrics source / server
- READS: Resource metrics. WATCHES: Metrics collection. WRITES: Metrics API responses. WHY: Supply observed load without creating Pods directly..
- Aggregated API server
- READS: Extension API requests. WATCHES: Extension resource behavior. WRITES: Extension responses. WHY: Extend the API surface without implementing every group in core kube-apiserver..
- Control-plane endpoint
- READS: Replica health. WATCHES: Endpoint health. WRITES: Routes client requests to serving replicas. WHY: Avoid one API replica becoming a client endpoint dependency..
Select an actor or relationship. Controllers, scheduler and kubelets have different responsibility boundaries even when all use the same API.
Context: Read/watch/write lists are representative, not complete API-client call inventories.
The API does not carry ordinary application requests
Does HTTP traffic to web pass through kube-apiserver?
Hover an actor to preview · click to pinTap an actor to inspect
One Kubernetes cluster: clients above, kube-apiserver at the center, etcd behind it, reconcilers and scheduler beside it, and kubelets with running Pods below.
Control-plane API state tells a node dataplane which backend addresses are eligible. The API server does not relay application HTTP packets.
The EndpointSlice controller writes endpoint state through the API. A node service dataplane uses that state to forward application packets to Pods.
Context: kube-proxy is optional when another implementation provides Service forwarding.
Break one dependency and follow what changes
Which work stops first, and which work keeps running?
Hover an actor to preview · click to pinTap an actor to inspect
One Kubernetes cluster: clients above, kube-apiserver at the center, etcd behind it, reconcilers and scheduler beside it, and kubelets with running Pods below.
- STOPS
- the web-b container and its readiness
- CONTINUES
- the Pod object, its identity, API state and other ready backends
- FIRST DETECTOR
- kubelet observes the container exit through the runtime
- API OBJECT CHANGE
- Pod/web-b status and restartCount; no replacement Pod is created
- REACTING LOOP
- kubelet on worker-b according to restartPolicy
- WORKLOAD TRAFFIC
- web-b leaves ready endpoints until it recovers; healthy backends can continue
- RECOVERY CONDITION
- the same Pod’s container restarts and passes readiness
kubectl get pod web-b kubectl describe pod web-b # restartCount rises; Pod UID stays the same
The container exits; kubelet handles the restart inside the same Pod.
Read all failure outcomes
Crash one container
Stops: the web-b container and its readiness. Continues: the Pod object, its identity, API state and other ready backends. Detector: kubelet observes the container exit through the runtime. API object: Pod/web-b status and restartCount; no replacement Pod is created. Reactor: kubelet on worker-b according to restartPolicy. Traffic: web-b leaves ready endpoints until it recovers; healthy backends can continue. Recovery: the same Pod’s container restarts and passes readiness.
Kill one Node
Stops: kubelet Lease renewals and Pods on worker-b. Continues: API and Pods on healthy Nodes. Detector: Node lifecycle observes stale Lease/Node condition after configured intervals. API object: Node/worker-b condition/taints; affected Pod state; possible replacements. Reactor: Node controller and workload reconcilers. Traffic: Traffic to affected backend can degrade until endpoints update. Recovery: Healthy Nodes, lifecycle handling and scheduler capacity allow replacements.
Kill controller manager
Stops: affected Deployment/ReplicaSet/Node reconciliation. Continues: API reads/writes and existing running Pods. Detector: operator sees stalled controller health and absent object progress. API object: existing desired objects remain; new dependent objects/status may not progress. Reactor: standby controller-manager after Lease failover if available. Traffic: existing Pod traffic can continue. Recovery: healthy replica takes Lease leadership or process is restored.
Kill scheduler
Stops: new Pod placement. Continues: API reads, existing bound Pods and running containers. Detector: Pending Pods remain unscheduled; scheduler health is absent. API object: Pod spec.nodeName remains unset. Reactor: standby scheduler after Lease failover if available. Traffic: existing application traffic can continue. Recovery: scheduler replica or process restores placement decisions.
Stop all API servers
Stops: new API reads/writes, status propagation and fresh coordination. Continues: existing node-local containers and possible existing dataplane rules. Detector: clients see API connection failure; watches disconnect. API object: no accepted API object change until serving returns. Reactor: controllers and kubelets reconnect/relist after restoration. Traffic: existing Pod traffic can continue if other dependencies stay healthy. Recovery: serving endpoint and backing dependencies recover; clients reconnect.
Kill one API replica
Stops: API2 serving. Continues: API1/API3 serving via healthy endpoint, existing Pods. Detector: endpoint health check excludes failed replica. API object: accepted objects remain in shared backing state. Reactor: no controller failover is required for API serving itself. Traffic: normal traffic can continue. Recovery: restart API2; endpoint/etcd remain healthy.
Block admission webhook
Stops: matching object-changing requests may wait or fail. Continues: unmatched requests and ordinary GET/LIST/WATCH can continue. Detector: API server receives timeout or connection error. API object: matching object not persisted if fail-closed; fail-open may continue. Reactor: requester/operator repairs webhook; no controller bypass is implied. Traffic: existing application traffic usually continues. Recovery: webhook recovers or configured failurePolicy permits continuation.
Break RBAC
Stops: a specific forbidden API operation. Continues: authorized reads, unrelated writes and existing Pods. Detector: API authorizer returns 403 Forbidden. API object: requested object unchanged; admission/persistence not reached. Reactor: requester or administrator fixes role/binding after investigation. Traffic: existing application traffic continues. Recovery: grant only required verb/resource/scope, then retry.
Break etcd connectivity
Stops: durable state access and many API operations. Continues: some existing node workloads and local dataplane state. Detector: API servers report storage errors/latency. API object: new desired/status writes may fail; reads depend on request/cache semantics. Reactor: operators restore storage quorum/connectivity; controllers retry after API recovers. Traffic: existing traffic may continue but freshness degrades. Recovery: restore durable backing state and healthy API serving.
Overload API
Stops: some requests queue or receive 429, depending on priority level. Continues: existing Pods and some higher-priority/healthy request classes may continue. Detector: APF queue/reject metrics and API latency reveal saturation. API object: desired/status changes may be delayed; rejected writes remain absent. Reactor: clients back off; operators address noisy source/priority configuration. Traffic: existing dataplane traffic need not use API request seats. Recovery: reduce load or tune capacity/classification with measured evidence.
Remove kube-proxy
Stops: Service forwarding in clusters relying on kube-proxy. Continues: API state and directly addressed Pods; clusters with replacements differ. Detector: node/service traffic fails or routes stale despite healthy EndpointSlices. API object: Service and EndpointSlice objects may remain unchanged. Reactor: network operator restores service proxy or replacement dataplane. Traffic: ClusterIP traffic can fail on affected Nodes. Recovery: restore chosen Service forwarding implementation and resync rules.
A fault affects API availability, reconciliation, node execution or application traffic differently. The first detector and changed API object tell you where the loop stalled.
Context: Recovery and Node eviction timing depend on cluster configuration.
API replicas can serve concurrently
What happens if one API server replica fails?
Hover an actor to preview · click to pinTap an actor to inspect
One Kubernetes cluster: clients above, kube-apiserver at the center, etcd behind it, reconcilers and scheduler beside it, and kubelets with running Pods below.
API1, API2 and API3 can serve requests concurrently. Controller-manager and scheduler leader election use separate Leases.
A healthy control-plane endpoint can route around one failed replica. etcd and endpoint availability remain shared dependencies. Scheduler and controller-manager leadership use separate Lease coordination.
Context: Three API replicas do not imply tolerance of any two control-plane failures.
The API boundary contains policy, serving and extension layers
What else does kube-apiserver do besides accept kubectl requests?
Hover an actor to preview · click to pinTap an actor to inspect
One Kubernetes cluster: clients above, kube-apiserver at the center, etcd behind it, reconcilers and scheduler beside it, and kubelets with running Pods below.
Fundamental
Security
State
Performance
Extensibility
Operations
Clients discover groups such as /api/v1 and /apis/apps/v1 before working with objects.
/api/v1 /apis/apps/v1 /apis/metrics.k8s.io/v1beta1 /apis/example.io/v1
Versioned API and discovery
Clients discover groups such as /api/v1 and /apis/apps/v1 before working with objects.
/api/v1 /apis/apps/v1 /apis/metrics.k8s.io/v1beta1 /apis/example.io/v1
Defaulting, conversion and validation
A versioned request becomes a valid object before accepted state is stored.
# Missing required field is rejected: PATCH Pod/web-a → 422 Unprocessable # Default inserted by API server: spec.restartPolicy: Always
Authentication and authorization
The API identifies a caller and checks the requested verb, resource and scope.
# 401 Unauthorized: bad token # 403 Forbidden: RBAC miss verb: create resource: deployments.apps namespace: demo
Admission and webhooks
Applicable writes can be mutated or denied; a matched webhook is an API-server-initiated call.
# Mutating webhook adds a label # Validating webhook denies invalid policy webhook → 403 Forbidden failurePolicy: Fail | Ignore
Storage interface and etcd
Durable backing state sits behind API serving; ordinary Kubernetes clients do not use etcd as their API.
# Clients go through the API: kubectl → API server → etcd # NOT: kubectl → etcd (direct)
LIST/WATCH and watch cache
Clients follow changes through API watches; serving caches mean every read is not a physical etcd request.
LIST resourceVersion: 10245 WATCH ?resourceVersion=10245 → MODIFIED 11020 # 410 Gone → re-LIST
API Priority and Fairness
Requests can be classified into priority levels and queued or rejected under pressure.
# High priority: system leader election # Default priority: user workloads # Exempt: health-check paths flowSchema → priorityLevelConfig
Aggregation and APIService
Registered extension API paths can be proxied to an aggregated server.
# Registered aggregated server:
apiVersion: apiregistration.k8s.io/v1
kind: APIService
spec:
service: {name: metrics-server}Subresources
/status, /scale, /log, /exec and /portforward have distinct purposes and authorization.
# Subresource calls: PATCH pods/web-a/status GET pods/web-a/log POST pods/web-a/exec POST pods/web-a/portforward
Audit trail
Request stages can generate policy-controlled audit records.
stage: RequestReceived | ResponseStarted | ResponseComplete level: None | Metadata | Request | RequestResponse
Control-plane-to-node proxy
Pod logs, attach and port-forward use special API-server-to-kubelet paths.
# API server initiates call to kubelet: GET /api/v1/namespaces/demo/pods/web-a/log → kubelet serves log stream
Show direct-call exceptions
- API server → kubelet: Pod logs, attach and port-forward.
- API server → matched admission/authentication/authorization webhook.
- API server → aggregated API server for registered extension paths.
- Cloud, CSI, Gateway and Operator controllers → external providers where configured.
Select a capability to see where API processing ends and a storage, webhook, kubelet or aggregated-server call begins.
Context: Request routing, APF and caches are shown conceptually, not as a literal fixed handler stack.
Kubernetes is many loops around one state gateway
Why not directly connect every component to every other one?
Hover an actor to preview · click to pinTap an actor to inspect
One Kubernetes cluster: clients above, kube-apiserver at the center, etcd behind it, reconcilers and scheduler beside it, and kubelets with running Pods below.
Kubernetes is a distributed system of reconciliation loops centered around a shared, versioned API.
Show API objects (grouped resource inventory)
Deployments · ReplicaSets · StatefulSets · DaemonSets · Jobs · CronJobs · Pods
Nodes · Leases · Services · EndpointSlices · Ingresses
ConfigMaps · Secrets · PersistentVolumeClaims · PersistentVolumes · StorageClasses
HorizontalPodAutoscalers · ResourceQuotas · LimitRanges · NetworkPolicies
Roles · RoleBindings · ClusterRoles · ServiceAccounts · CustomResourceDefinitions · APIServices
Check the seven tempting shortcuts
- kubectl is Kubernetes.kubectl is one API client.
- Scheduler starts Pods.Scheduler records placement; kubelet executes assigned PodSpecs.
- Controllers command kubelets.They normally coordinate indirectly through API state.
- Every component uses etcd directly.Ordinary Kubernetes clients use the API server.
- WATCH means continuous etcd polling.API watches plus server and client caches form a reactive path.
- All network traffic uses kube-apiserver.Ordinary workload traffic follows the data plane.
- An API outage instantly kills running containers.Existing node workloads can continue while coordination is impaired.
- Check API endpoint availability, latency and audit evidence.
- Compare object spec with status and the last changed resourceVersion.
- Inspect controller progress, pending scheduling and Node/Lease freshness.
- Check runtime, Service dataplane and storage separately from API state.
- Rehearse recovery of the first broken dependency, then watch loops converge.
A common object model, policy boundary, watch semantics and durable backing state let independent actors converge without a monolithic orchestrator.
Context: This is a state-coordination model, not a claim about every network call or runtime mechanism.