From a stable address to a changing backend
Your application can keep connecting to one Service address while Pods appear, disappear, or move between Nodes. Kubernetes maintains that stable frontend separately from the changing backend list. Controllers publish eligible addresses in EndpointSlices, and the cluster’s Service implementation turns that state into packet forwarding. Service concepts ↗
A Service is one frontend fed by changing backend state
The control plane publishes eligible destinations; every Service data plane reconciles that state locally or through its implementation.
The diagrams use destination translation to explain the common kube-proxy path. Another implementation can provide the same Service behavior through different data-plane machinery, so identify the implementation before inspecting its rules.
From selector to EndpointSlice
When a Service has a selector, the endpoint controller compares that selector with Pod labels and publishes matching addresses in EndpointSlices. Each endpoint also carries conditions such as ready, serving, and terminating, which help consumers decide whether to use it. A Service without a selector needs separately managed EndpointSlices or can represent an ExternalName. EndpointSlices ↗
- 01Labels matchThe Service selector identifies candidate Pods.
- 02Ports resolveA numeric or named targetPort maps the Service port to each backend.
- 03EndpointSlices updateAddresses, ports, conditions, Node, zone, and hints are published.
- 04Consumers reconcileService proxies and DNS observe the new desired backend state.
The deprecated Endpoints API truncates large backend sets and cannot represent newer information such as dual-stack and traffic distribution. Modern clients should list every EndpointSlice carrying the kubernetes.io/service-name label; one slice is not necessarily the complete backend set.
Ports are three different contracts
| Field | Owned by | Meaning |
|---|---|---|
| port | Service | The port clients use on the Service frontend. |
| targetPort | Service-to-backend mapping | The port used at the selected endpoint; it can be numeric or a named Pod port. |
| nodePort | Node-facing Service exposure | A port allocated or selected on eligible Node addresses for NodePort-style traffic. |
If a Pod listens on 8080 while the Service sends traffic to 8081, the connection still fails even when the Pod is healthy. A named targetPort lets Pods expose the same logical port name at different numbers. Kubernetes may place those endpoints in separate slices because a port definition applies to the entire EndpointSlice.
ClusterIP interception
Unless the Service is headless, the control plane allocates a ClusterIP from the Service address range. In a common Linux kube-proxy path, a Node matches packets for that address and port, chooses an endpoint, and rewrites the destination to the endpoint address and targetPort. Connection tracking remembers the translation for later packets in the same connection. Virtual IPs ↗
| Implementation | How it programs forwarding | What to remember |
|---|---|---|
| kube-proxy iptables | Netfilter chains and DNAT rules | Common and mature; rule synchronization grows with Service and endpoint count. |
| kube-proxy nftables | nftables maps and rules | Designed for better update and lookup behavior; NodePort and firewall defaults differ. |
| kube-proxy IPVS | IPVS virtual servers plus iptables integration | Deprecated from Kubernetes v1.35 and not a good choice for new deployments. |
| Windows kernelspace | Windows filtering and virtual-switch APIs | Linux commands and packet hooks do not apply. |
| Proxy replacement | CNI/provider-specific data plane | May use eBPF or cloud networking and expose no kube-proxy rules at all. |
“Random backend” describes default selection for new connections in some proxy modes, not round-robin HTTP requests. Keep-alive, HTTP/2, gRPC, WebSockets, QUIC, application pooling, and conntrack can send many requests through one long-lived backend connection.
Readiness and termination
Endpoint readiness normally follows Pod readiness. A terminating endpoint has terminating: true; for compatibility its ready condition becomes false, while serving can still indicate that it is capable of responding. Proxies normally avoid terminating endpoints for new traffic but can use serving-and-terminating endpoints when every available endpoint is terminating. Termination flow ↗
- 01Readyready=true
serving=true → - 02Termination beginsdeletion timestamp
readiness changes → - 03Drainingterminating=true
ready=false
serving may remain true → - 04RemovedNo longer published
connections finish or close
A graceful shutdown gives readiness changes time to reach EndpointSlices and traffic handlers before the process exits. The application, controller, proxy, and load balancer still reconcile on their own schedules.
Some peer-discovery systems need addresses before their members become ready. Setting publishNotReadyAddresses: true supports that pattern by publishing those endpoints early. It also lets clients reach processes that may not serve normal traffic yet, so it cannot compensate for a readiness probe that tests the wrong thing.
Headless and selectorless Services
A headless Service sets clusterIP: None. Kubernetes does not create a virtual-IP forwarding path; cluster DNS can return endpoint addresses directly. Client libraries then own address choice, retries, and reaction to DNS caching.
A selectorless Service lets operators connect a Kubernetes name and frontend to manually managed EndpointSlices, including non-Pod destinations. Endpoint addresses cannot be another Service ClusterIP. Ownership labels, address types, ports, and endpoint readiness still matter.
NodePort and LoadBalancer
NodePort extends the Service onto selected Node addresses. LoadBalancer asks an implementation controller to provision or configure external infrastructure and often builds on ClusterIP and NodePort semantics, though implementations can disable NodePort allocation or route directly to Pods. Do not infer cloud hops solely from the Service type.
- 01External addressA provider or operator makes the frontend reachable.
- 02Eligible Node or PodThe load-balancer implementation chooses its registered target.
- 03Service policyCluster or Local semantics constrain backend selection.
- 04Endpoint receives trafficThe observed source address depends on the path and translation.
The older externalIPs field does not allocate or advertise addresses; the operator must arrange for traffic to reach cluster Nodes. It is deprecated in Kubernetes v1.36, so prefer a load-balancer controller or Gateway API implementation for new designs.
Locality and traffic distribution
trafficPolicy: ClusterReach any eligible endpointBroader distribution; an extra Node hop and source translation may appear on some external paths.
trafficPolicy: LocalOnly this Node’s endpointsNo local endpoint means no eligible endpoint on that Node. External Local can preserve client IP when the surrounding path supports it.
trafficDistributionPrefer topology; allow fallbackA preference is not the same as Local. Compatible implementations and safeguards decide when fallback occurs.
Local is a hard eligibility restriction; trafficDistribution is a preference with implementation-dependent safeguards and fallback. Internal policy ↗ Source IP ↗ Topology-aware routing ↗
Keeping traffic near the client can reduce latency and cross-zone cost. It can also overload a small local endpoint set while healthy capacity sits elsewhere. Check endpoint counts by zone or Node, autoscaling behavior, fallback during failure, and the distribution you observe in production.
Session affinity and connection state
sessionAffinity: ClientIP asks the implementation to prefer the same backend for connections from one observed client address during the configured timeout. NAT can collapse many clients into one apparent address, and not every protocol’s application session aligns with an IP-based affinity window.
When an endpoint disappears, established conntrack entries or application connections can persist until closed or expired. Debug new connections separately from existing ones. A newly corrected Service can appear broken to one long-lived client while a fresh diagnostic Pod succeeds.
Source address behavior
ClusterIP traffic through kube-proxy’s iptables mode normally preserves the Pod client address. External paths can masquerade the source to keep return traffic symmetric through the receiving Node. externalTrafficPolicy: Local can preserve the external client address when the surrounding load balancer and network support the path, but it changes eligible endpoints and health-check behavior.
Never write application allowlists or NetworkPolicies from the conceptual source address. Capture what the backend actually observes in the deployed implementation.
Scale and convergence
EndpointSlices default to at most 100 endpoints each and prioritize limiting update fan-out over perfectly packing every slice. Every Node-level proxy watching them must reconcile changes. Large clusters should monitor API watch pressure, EndpointSlice update latency, proxy sync duration, rule count, conntrack capacity, and load-balancer controller convergence.
Traffic distribution is not application health. Kubernetes readiness says an endpoint is eligible; it does not measure request latency, queue depth, or downstream saturation unless probes deliberately model them.
Troubleshoot a Service
Verify the application directly at Pod IP and targetPort. If this fails, stop debugging the Service frontend.
Inspect kubectl get svc <name> -o yaml: selector, address families, ports, traffic policies, affinity, and exposure type.
List every slice with kubectl get endpointslice -l kubernetes.io/service-name=<name> -o yaml. Check addresses, port names, ready, serving, and terminating.
Test ClusterIP and port from a Pod on the same Node as a backend, then from another Node. This separates local forwarding from cross-node routing.
Identify kube-proxy mode or replacement. Inspect only the matching rules, maps, metrics, logs, and conntrack state.
For external traffic, inspect controller status, registered targets, health checks, Node firewall rules, source translation, and return routing.
What this means in practice
When a Service fails, compare four things: the frontend you declared, the backends published in EndpointSlices, the forwarding state programmed on the Nodes, and the connection the client actually opened. The first mismatch usually identifies the responsible layer. Record that evidence before changing selectors or restarting the Service implementation.
For new Linux clusters, evaluate nftables or the chosen proxy replacement against kernel support and implementation documentation; do not move to deprecated IPVS merely because an older comparison calls it the high-performance option.