From a name to a connection
Your application connects to payments, but the network cannot route a name. The Pod’s resolver first turns that name into one or more DNS queries. Cluster DNS answers a Kubernetes name from cluster state or forwards another name upstream, and any cache along the path can reuse the answer until its TTL expires. The application opens a connection only after it receives an address. DNS specification ↗
A short name can produce several DNS queries
The Pod's resolver applies its search rules first. Cluster DNS then answers a Kubernetes name or forwards another name upstream.
The examples use cluster.local, the common default. Your cluster may use another domain, so inspect the Pod’s actual resolver configuration before applying these names literally.
Resolver configuration inside a Pod
Kubelet writes DNS settings into each Pod. With the usual ClusterFirst policy, /etc/resolv.conf points to the cluster DNS Service and includes search suffixes for the Pod’s namespace, all Services, and the cluster domain. A typical file looks like this:
nameserver 10.96.0.10
search team-a.svc.cluster.local svc.cluster.local cluster.local
options ndots:5Treat those values as examples. The Pod’s dnsPolicy and dnsConfig, kubelet configuration, Node resolver state, operating system, and runtime all influence the file you actually get.
| Policy | Resolver basis | Use with care |
|---|---|---|
| ClusterFirst | Cluster DNS for cluster names; other names forwarded upstream | The default when dnsPolicy is omitted. |
| Default | Inherits the Node resolver configuration | The name means inheritance; it is not Kubernetes default behavior. |
| ClusterFirstWithHostNet | Cluster-first behavior for hostNetwork Pods | Set explicitly when host-networked workloads need cluster discovery. |
| None | Ignores Kubernetes-generated resolver settings | Requires explicit dnsConfig nameservers and related settings. |
dnsConfig can add nameservers, search domains, and resolver options. Excessive search lists, incompatible Node resolvers, and platform limits can still alter or reject the final configuration.
Search paths and ndots
If an application asks for api, the resolver may not send that exact name first. With ndots:5, a resolver following the usual search-list behavior can append the Pod’s search suffixes because api contains fewer than five dots. The same rule can affect a name that looks external: api.example.com still has only two dots.
- 01Application requests apiThe name has no trailing dot and few dots.
- 02Resolver applies searchThe Pod namespace suffix is a candidate.
- 03Query api.team-a.svc…Cluster DNS evaluates the expanded name.
- 04Return Service answerThe application receives addresses and TTLs.
Each attempted suffix can add a query and wait time before the resolver tries the absolute name. A trailing dot marks a name as absolute for conventional resolvers. Observe the queries from the affected application before changing ndots, because some runtimes use their own resolver behavior or cache.
Normal Service records
For a normal Service, cluster DNS serves A records for IPv4 addresses and AAAA records for IPv6 addresses at a name shaped like <service>.<namespace>.svc.<cluster-domain>. The answer points to the Service ClusterIP for that address family. When the client connects to that virtual IP, the Service data plane chooses a backend Pod. Service DNS records ↗
Named Service ports can create SRV records. An SRV lookup expresses service, protocol, port, and target information, but applications must explicitly perform and understand SRV discovery.
ExternalName is different: DNS returns a CNAME derived from spec.externalName; Kubernetes creates no proxying or backend health checking. TLS hostnames and HTTP Host headers still reflect what the client uses, which can make a valid alias fail at the application layer.
Headless Services and Pod identity
api.team-a.svc…db.team-a.svc…payments.team-a.svc…db-0.db.team-a.svc…A headless Service sets clusterIP: None, so there is no Service virtual IP for DNS to return. Cluster DNS answers with eligible endpoint addresses instead. The client or its library must then choose an endpoint and cope with caching, retries, and backend changes.
StatefulSets commonly pair a headless Service with stable Pod identity. A Pod can receive a name shaped like <pod>.<service>.<namespace>.svc.<cluster-domain> when its hostname/subdomain and governing Service align. Stable DNS identity does not mean a stable Pod IP; a recreated Pod can keep its ordinal name while receiving new addresses. StatefulSet identity ↗
publishNotReadyAddresses can publish endpoints before readiness for peer discovery. It deliberately changes normal safety semantics and can direct clients toward processes that are not ready to serve ordinary traffic.
How CoreDNS learns Kubernetes names
CoreDNS is the common cluster DNS implementation and runs as Pods behind a Service named kube-dns for compatibility. Its kubernetes plugin watches Services, namespaces, Pods where configured, and EndpointSlices, then serves authoritative records for configured cluster zones. The forward plugin commonly sends other names toward upstream resolvers. CoreDNS kubernetes plugin ↗
CoreDNS therefore depends on two working paths. Clients must be able to reach the CoreDNS Pods through the kube-dns Service, and CoreDNS must be able to read the Kubernetes objects from which it builds records. A failure on either side can look like “DNS is down.”
Corefile order matters because CoreDNS is a plugin chain. Caching, rewrites, forwarding, loop detection, health, readiness, metrics, and logging behavior depend on the deployed configuration. Managed platforms may restrict or generate that configuration.
Caching and stale answers
The application, language runtime, operating-system resolver, Node-local cache, CoreDNS, and upstream resolver can each cache an answer. They may also cache negative answers such as NXDOMAIN, often with different TTL behavior. Fixing a Service record does not immediately clear those caches, so some clients can keep seeing the old result.
Compare a failing application’s lookup with a known diagnostic tool from the same Pod namespace and Node. A difference can indicate application-level caching or a custom resolver rather than CoreDNS itself.
NodeLocal DNSCache
NodeLocal DNSCache runs a caching agent on each Node. Pods can query the local agent, which handles cache hits and forwards cluster-name misses toward CoreDNS. This can reduce cross-node DNS traffic, bypass some Service DNAT and UDP conntrack pressure, enable TCP upstream connections, and expose per-Node DNS metrics. NodeLocal DNSCache ↗
- 01Pod sends DNS queryThe configured local address is reachable on the Node.
- 02Node cache checks TTLPositive and negative cache entries may answer immediately.
- 03Miss goes upstreamCluster names reach CoreDNS; other names follow forwarding policy.
- 04Answer returns and cachesNode-local metrics expose local latency and errors.
The local cache also becomes another place a lookup can fail. If Pods fail only on one Node while central CoreDNS remains healthy, inspect that Node’s cache agent, listener, and network path.
DNS and NetworkPolicy
A default-deny egress policy blocks DNS along with other outbound traffic until you allow the path your Pods actually use. A rule based on assumed kube-system labels can miss the deployed DNS Pods, while a rule containing only the ClusterIP can behave differently depending on where the network plugin enforces policy relative to address translation. Check the plugin’s semantics and test from a Pod selected by the policy.
DNS over UDP 53 is common, but TCP 53 is also required for valid DNS behavior such as larger responses and retries. Encrypted DNS used directly by applications can bypass cluster DNS policy entirely.
Diagnose resolution failures
Capture the exact name, record type, response code, latency, and resolver library. “DNS failed” is not enough to distinguish timeout, NXDOMAIN, SERVFAIL, or a wrong answer.
Inspect /etc/resolv.conf in the failing Pod. Verify nameserver, search domains, ndots, hostNetwork, dnsPolicy, and dnsConfig.
Query the short name, namespace-qualified name, and full absolute name. A full name working while a short name fails points toward search configuration or namespace assumptions.
Inspect kubectl -n kube-system get svc kube-dns and its EndpointSlices. The compatibility Service name can still front CoreDNS Pods.
Check CoreDNS readiness, logs, metrics, Corefile, API connectivity, and RBAC for Services, namespaces, Pods, and EndpointSlices.
Query the configured nameserver directly, then test upstream names separately. Cluster names working while external names fail points toward forwarding or upstream DNS.
Compare Nodes. With NodeLocal DNSCache, inspect the local agent and metrics on the failing Node before restarting central CoreDNS.
Account for caches and TTLs. Test from a fresh process or Pod before concluding that a configuration correction did not work.
Where DNS intuition goes wrong
“CoreDNS balances a normal Service across Pods.”
It normally returns the Service VIP. The Service data plane selects an endpoint.
“cluster.local is fixed.”
It is a common default. The cluster domain is configurable.
“A successful lookup proves the app is reachable.”
Discovery can succeed while Service routing, policy, or the application fails.
“One nslookup reproduces every client.”
Language runtimes and applications can use different resolver libraries, caches, search handling, or encrypted DNS.
Monitor DNS request rate, response codes, latency, cache hit behavior, upstream failures, CoreDNS saturation, and Node-local skew. Pair those signals with application connection errors so a name-resolution incident is not confused with a downstream timeout.