DNS: From a Name to an Answer
Follow shop.example.com from your app to the root and back, see every cache it lands in, then move it to a new address and diagnose the ways a lookup fails.
A browser can't connect to a name
You type shop.example.com. Before a single byte of the shop can load, something has to turn that name into an address.
Your browser has a name, but packets need an IP address. It can't open a connection to shop.example.com as it stands.
So it asks a question: what is the IPv4 address for shop.example.com? The question has two parts, a name and a type (A, for IPv4). The answer comes back as a record: 192.0.2.10, reusable for 3600 seconds.
The question and its answer
- name
shop.example.com- type
A (IPv4 address)- answer
192.0.2.10new- TTL
3600 secondsnew
Then the browser connects to the address. DNS is done at this point: it carries none of the shop's traffic and never learns whether the connection worked.
DNS answers one kind of question
Every lookup is a name plus a type, and every answer is a set of records of that type. A, AAAA, MX and the rest are just different questions about the same name. The TTL tells anyone holding the answer how long they may reuse it without asking again.
- Question
- name + type
- Answer
- records, each with a TTL
- DNS carries
- only the lookup, not your traffic
dig +short shop.example.com ADNS doesn't check that the server behind an address is healthy. A perfectly correct answer can point at a machine that is down.
THE POINT TO KEEPDNS turns a name and a type into records; everything after that is a connection DNS never sees.
Your machine answers itself first
Most applications don't speak DNS. They ask the operating system, and the operating system checks a few places before any packet leaves.
Your app calls getaddrinfo(). Browsers and some runtimes check their own cache first (1). On Linux, glibc then follows the hosts line in /etc/nsswitch.conf, commonly the hosts file before DNS (2), then a local caching service such as systemd-resolved (3), and only then the network resolver (4).
What each tool consults
getent ahosts shop.example.com # follows nsswitch, like your appresolvectl query shop.example.com # asks systemd-resolveddig shop.example.com # asks the resolver, nothing elseSomeone added a line to /etc/hosts while testing. The lookup stops at step 2, so the app connects to 10.0.0.5. dig never reads the file, so it shows the public answer and looks healthy.
DNS LOOKS FINE, THE APP IS WRONGdig says 192.0.2.10; curl and your service connect to 10.0.0.5.
Same name, two answers
$ dig +short shop.example.com192.0.2.10$ getent ahosts shop.example.com10.0.0.5 STREAM shop.example.com$ grep shop /etc/hosts10.0.0.5 shop.example.com # added for a test, never removedThe record moved to 203.0.113.20 an hour ago. dig and new processes see the new address, but a long-running JVM still holds the old one in its own cache and never asks again.
RESTART FIXES ITIf a restart cures it, look for a cache inside the process.
dig is not what your app does
dig, host and nslookup send a DNS query straight to a resolver. Your application goes through the system resolver library, which reads /etc/hosts and may use a local cache, and the app may cache on top of that. When dig and the app disagree, the difference is somewhere between them.
- Order is set by
- /etc/nsswitch.conf, hosts line
- Resolver address from
- /etc/resolv.conf (often 127.0.0.53)
- glibc itself
- does not cache answers
getent ahosts shop.example.commacOS and Windows have their own system resolvers and caches, and browsers can send DNS over HTTPS to a provider they choose, skipping the system resolver entirely.
nsswitch.conf(5) ↗getaddrinfo(3) ↗resolv.conf(5) ↗systemd-resolved.service(8) ↗Java networking properties ↗
THE POINT TO KEEPWhen dig and your app disagree, the difference lives between them: an app cache, a hosts file, or a local cache.
The resolver walks the tree for you
Your laptop asks one question. A recursive resolver does the walking: root, then com, then the servers for example.com, keeping everything it learns.
Your laptop's stub resolver sends one question to the recursive resolver it was given by DHCP or VPN settings, and asks it to do the work (the RD flag). The resolver's cache is empty.
Resolver cache
- —
empty
The resolver knows the root servers' addresses from a built-in hints file. With QNAME minimisation, it tells the root only what the root needs: it wants the servers for com, not the full name.
MINIMISEDOlder resolvers send shop.example.com to every server on the way. RFC 9156 limits each server to the part it is responsible for.
The root doesn't know shop.example.com and doesn't need to. It returns a referral: NS records for com and their addresses. No answer, only "ask them".
From a.root-servers.net
;; flags: qr; ANSWER: 0, AUTHORITY: 13, ADDITIONAL: 27;; AUTHORITY SECTION:com. 172800 IN NS a.gtld-servers.net.com. 172800 IN NS b.gtld-servers.net.;; ADDITIONAL SECTION:a.gtld-servers.net. 172800 IN A 192.5.6.30The resolver asks a com server who serves example.com.
Another referral: ns1 and ns2.example.com. Those names are inside example.com itself, so the com server also returns their addresses (glue). Without glue the resolver would need example.com's servers to find example.com's servers.
From a com server
;; AUTHORITY SECTION:example.com. 172800 IN NS ns1.example.com.example.com. 172800 IN NS ns2.example.com.;; ADDITIONAL SECTION:ns1.example.com. 172800 IN A 192.0.2.53ns2.example.com. 172800 IN A 198.51.100.53Now the resolver asks the zone's own server the full question.
ns1.example.com answers with the record and sets the aa flag: the answer comes from a server authoritative for the zone, not from a cache.
From ns1.example.com
;; flags: qr aa; ANSWER: 1;; ANSWER SECTION:shop.example.com. 3600 IN A 192.0.2.10The resolver keeps everything it learned, each for its own TTL: two days for the delegations, an hour for the shop.
Resolver cache
- com NS
172800 sjust saved - example.com NS
172800 sjust saved - shop A
192.0.2.10 · 3600 sjust saved
Your laptop gets the answer. The ra flag says the resolver did recursion; aa is gone because, to you, this is now the resolver's answer.
What dig shows on the laptop
;; ->>HEADER<<- opcode: QUERY, status: NOERROR;; flags: qr rd ra; QUERY: 1, ANSWER: 1;; ANSWER SECTION:shop.example.com. 3600 IN A 192.0.2.10Twenty minutes later a colleague asks the same resolver. It answers from its cache without contacting any server, and the TTL it hands out has counted down to 2400.
Resolver cache
- com NS
171 000 s left - example.com NS
171 000 s left - shop A
192.0.2.10 · 2400 s leftloaded
Referrals, then an answer
Only the last server knows the answer. The root and com servers know who to ask next, and say so with NS records in a referral. Because the resolver caches those referrals for days, the full walk is rare: most lookups start at a server the resolver already knows.
- Root servers
- 13 names, a–m.root-servers.net, each many anycast sites
- Walking done by
- the recursive resolver
- aa flag
- answer from the zone's own server
dig +trace shop.example.comdig +trace does the walk from your machine, not from your resolver. It shows what the delegation says, which is not necessarily what your resolver has cached.
RFC 1034: DNS concepts ↗RFC 9156: QNAME minimisation ↗RFC 9471: glue in referrals ↗IANA: root servers ↗BIND 9: dig ↗
THE POINT TO KEEPA resolver follows referrals from the root down and caches each one, so most lookups skip straight to the last server or to the cache.
The tree is cut into zones
The referrals in the last view happen at zone cuts. A zone is the part of the tree one set of servers answers for; the parent points to it with NS records.
The root zone delegates com, com delegates example.com, and example.com can delegate eu.example.com to another provider. shop and www aren't zones; they are names inside example.com and live in its zone file.
| Name | Answered by |
|---|---|
| com | the com servers |
| example.com, shop, www | ns1 and ns2.example.com |
| eu.example.com | the other provider's servers |
ns1.example.com is a name inside the zone it serves. To break the loop, the com zone holds a copy of its address. That copy is glue: not authoritative data, just enough to reach the server.
| Record | Purpose |
|---|---|
| example.com NS ns1.example.com | the delegation |
| example.com NS ns2.example.com | the delegation |
| ns1.example.com A 192.0.2.53 | glue |
| ns2.example.com A 198.51.100.53 | glue |
| example.com DS 12345 13 2 … | DNSSEC link, if signed |
You moved example.com to new servers and updated NS inside the zone, but the registrar still lists the old provider. Resolvers follow the parent, reach servers that no longer hold the zone, and get REFUSED or nothing. This is a lame delegation.
LAME DELEGATIONThe new servers answer correctly when asked directly, which makes this easy to miss. The registrar's NS list is what resolvers follow.
Authority is handed down
A parent zone doesn't store its children's records. It stores NS records that say who does, plus glue when needed. That one handover is what lets each organisation run its own part of the tree.
- Delegation
- NS records in the parent
- Glue
- addresses for in-zone name servers
- You change the parent's NS
- at your registrar
dig @a.gtld-servers.net example.com NS +norecurseA subdomain is not automatically a separate zone. eu.example.com becomes one only when example.com delegates it with NS records.
RFC 1034: DNS concepts ↗RFC 9471: glue in referrals ↗RFC 2181: clarifications to the DNS spec ↗RFC 9499: DNS terminology ↗
THE POINT TO KEEPResolvers trust the parent's NS records to find a zone, so a provider change is finished only when the registrar says so.
What the zone for example.com holds
A zone is a list of records. Each type answers a different question about a name. Choose a sign to see its fields, an example, and the trap that catches people.
A IPv4 address
“This name's IPv4 address.”
shop.example.com. 3600 IN A 192.0.2.10- owner
shop.example.com.The name the record belongs to. The trailing dot makes it absolute.- TTL
3600Seconds a cache may keep it.- class
INInternet. Effectively always IN.- type
A- data
192.0.2.10One IPv4 address. Several A records at one name give several addresses.
Ask for it
$ dig +short shop.example.com A192.0.2.10Several A records are not a health-checked load balancer. The resolver returns all of them whether the servers behind them are up or not, and clients differ in which one they try first.
RFC 1035: DNS implementation ↗RFC 2181: clarifications to the DNS spec ↗
AAAA IPv6 address
“This name's IPv6 address.”
shop.example.com. 3600 IN AAAA 2001:db8::10- owner
shop.example.com.- TTL
3600- type
AAAAFour times the size of A, hence the name.- data
2001:db8::10One IPv6 address.
Ask for it
$ dig +short shop.example.com AAAA2001:db8::10A client with IPv6 asks for A and AAAA and races connections to both (Happy Eyeballs), often preferring IPv6. A stale or wrong AAAA breaks only IPv6 users, so the site "works for me" on an IPv4-only network. Move AAAA with A during a migration.
CNAME alias to another name
“This name is another name. Look that one up instead.”
www.example.com. 3600 IN CNAME shop.example.com.- owner
www.example.com.The alias.- type
CNAMECanonical name.- data
shop.example.com.The real name. The resolver follows it and returns both records.
Ask for it
$ dig www.example.com A +noall +answerwww.example.com. 3600 IN CNAME shop.example.com.shop.example.com. 3600 IN A 192.0.2.10A CNAME can't share its name with any other record. The zone apex (example.com) always has SOA and NS, so it can't be a CNAME. DNS providers offer an ALIAS, ANAME or "flattening" feature instead: the provider resolves the target and serves plain A and AAAA records. Each hop in a CNAME chain keeps its own TTL.
RFC 1034: DNS concepts ↗RFC 2181: clarifications to the DNS spec ↗
HTTPS how to connect
“How to reach this HTTPS service: protocols, port, address hints, before the first packet.”
shop.example.com. 3600 IN HTTPS 1 . alpn="h3,h2" ipv4hint=192.0.2.10- priority
10 means alias mode (point at another name); 1 or more is service mode, lowest first.- target
.A dot means "this same name".- alpn
h3,h2The client can try HTTP/3 on its first connection.- ipv4hint
192.0.2.10A hint only. Clients still look up A and AAAA.
Ask for it
$ dig +short shop.example.com HTTPS1 . alpn="h3,h2" ipv4hint=192.0.2.10HTTPS records are type 65, a sibling of the general SVCB type. Alias mode (priority 0) is allowed at the zone apex, where a CNAME is not, but only clients that ask for HTTPS records follow it. Forgetting to update ipv4hint during a move leaves clients with a stale address until they check A.
MX mail exchanger
“Where to deliver email for this domain. Lower preference first.”
example.com. 3600 IN MX 10 mail.example.com.- owner
example.com.The domain after the @.- preference
10Lower is tried first; equal values share load.- exchange
mail.example.com.A host name that has A or AAAA records.
Ask for it
$ dig +short example.com MX10 mail.example.com.The exchange must be a host name with address records: not an IP address and not a CNAME. A domain that receives no mail can say so with a null MX, "MX 0 .", so senders fail fast instead of retrying for days.
RFC 5321: SMTP, MX lookup ↗RFC 2181: clarifications to the DNS spec ↗RFC 7505: null MX ↗
TXT text: policies, proofs
“Free text. In practice: published policies (SPF, DKIM keys, DMARC) and domain-ownership proofs.”
example.com. 3600 IN TXT "v=spf1 include:_spf.mail.example.net -all"- owner
example.com.DMARC lives at _dmarc.example.com and DKIM keys at <selector>._domainkey.example.com.- data
"v=spf1 …"One or more strings of up to 255 bytes each, joined by the reader.
Ask for it
$ dig +short example.com TXT"v=spf1 include:_spf.mail.example.net -all""site-verification=4f2a…"Only one SPF record per name. A second "v=spf1" TXT makes SPF evaluation fail with a permanent error. SPF also allows at most 10 DNS lookups for include and similar terms, which long include chains exceed.
SRV service location
“Which host and port serve a named protocol.”
_imaps._tcp.example.com. 3600 IN SRV 10 5 993 mail.example.com.- owner
_imaps._tcp.example.com._service._protocol, then the domain.- priority
10Lower first.- weight
5Share between records with equal priority.- port
993- target
mail.example.com.A host name, not a CNAME.
Ask for it
$ dig +short _imaps._tcp.example.com SRV10 5 993 mail.example.com.A client uses SRV only when its protocol says to: SIP, XMPP, LDAP, Kerberos, mail client set-up. Browsers don't use SRV for HTTP. Kubernetes publishes SRV records for named Service ports.
RFC 2782: SRV records ↗Kubernetes: DNS for Services and Pods ↗
NS name servers
“Which servers answer for this zone.”
example.com. 172800 IN NS ns1.example.com.- owner
example.com.The zone.- data
ns1.example.com.A host name of an authoritative server.
Ask for it
$ dig +short example.com NSns1.example.com.ns2.example.com.NS records exist twice: in the parent (com) as the delegation, and in your zone as its own copy. Resolvers find you through the parent's copy, which you change at your registrar. Editing only the copy in your zone changes nothing for a resolver starting from the root.
RFC 1034: DNS concepts ↗RFC 2181: clarifications to the DNS spec ↗
SOA start of authority
“Facts about the zone itself, including how long to remember a "no".”
example.com. 3600 IN SOA ns1.example.com. hostmaster.example.com. 2026101501 7200 3600 1209600 300- primary
ns1.example.com.- mailbox
hostmaster.example.com.hostmaster@example.com, with the first dot as the @.- serial
2026101501Secondaries copy the zone when this goes up.- refresh / retry / expire
7200 3600 1209600Timers for secondary servers.- minimum
300Since RFC 2308: how long resolvers cache NXDOMAIN and NODATA (the lower of this and the SOA's own TTL).
Ask for it
$ dig +short example.com SOAns1.example.com. hostmaster.example.com. 2026101501 7200 3600 1209600 300The last field is no longer a default TTL. It sets negative caching: if anyone looks up a name before you create it, resolvers can keep answering "no such name" for that long after you add the record.
CAA certificate authority authorization
“Which certificate authorities may issue certificates for this name.”
example.com. 3600 IN CAA 0 issue "letsencrypt.org"- flags
0128 marks a tag the CA must understand.- tag
issueissuewild covers wildcards; iodef gives a reporting address.- value
"letsencrypt.org"The CA's domain.
Ask for it
$ dig +short example.com CAA0 issue "letsencrypt.org"Only CAs read CAA, and only when issuing. Browsers ignore it, existing certificates are unaffected, and it encrypts nothing. A CA checks the exact name first, then climbs (shop.example.com, example.com) and uses the first CAA set it finds.
DS delegation signer
“A fingerprint of the child zone's key, stored in the parent. One link in the chain of trust.”
example.com. 86400 IN DS 12345 13 2 3F5A…C9- key tag
12345Which DNSKEY it fingerprints.- algorithm
13ECDSA P-256 with SHA-256.- digest type
2SHA-256.- digest
3F5A…C9
Ask for it
$ dig +short example.com DS12345 13 2 3F5A…C9The DS lives in the com zone, so you set it at your registrar. Moving DNS providers or rolling keys without updating it makes every validating resolver return SERVFAIL for the whole zone.
DNSKEY zone public key
“The public keys that check this zone's signatures.”
example.com. 3600 IN DNSKEY 257 3 13 mdsswUyr…- flags
257257 is a key-signing key, which the DS points at; 256 is a zone-signing key.- protocol
3Always 3.- algorithm
13- key
mdsswUyr…Base64 public key.
Ask for it
$ dig +multi example.com DNSKEYexample.com. 3600 IN DNSKEY 257 3 13 ( mdsswUyr… ) ; KSK; alg = ECDSAP256SHA256 ; key id = 12345The key tag dig prints must match the tag in the parent's DS. When they differ, the chain breaks at this zone.
RRSIG signature
“A signature over one set of records, valid between two dates.”
shop.example.com. 3600 IN RRSIG A 13 3 3600 20261029000000 20261015000000 34567 example.com. kQ9…- covers
AThe record type it signs.- labels / TTL
3 3600- expires / signed
20261029… 20261015…Outside this window the signature is invalid.- key tag, signer
34567 example.com.Which DNSKEY made it.
Ask for it
$ dig +dnssec shop.example.com A +noall +answershop.example.com. 3600 IN A 192.0.2.10shop.example.com. 3600 IN RRSIG A 13 3 3600 20261029000000 …Signatures expire. If whatever re-signs the zone stops, validation fails on the expiry date even though nobody changed a record.
PTR name for an address
“The name for an IP address. Lives in a reverse zone, not in example.com.”
10.2.0.192.in-addr.arpa. 3600 IN PTR shop.example.com.- owner
10.2.0.192.in-addr.arpa.192.0.2.10 with the octets reversed. IPv6 uses ip6.arpa, one label per hex digit.- data
shop.example.com.
Ask for it
$ dig +short -x 192.0.2.10shop.example.com.The reverse zone belongs to whoever holds the address block: your cloud provider or ISP. Owning example.com doesn't let you set it; you ask the address owner, for example through EC2's reverse DNS setting. Mail servers often check that the PTR name's A record points back to the same address.
Every record has the same shape
Owner name, TTL, class, type, then data whose format depends on the type. A lookup asks for one name and one type and gets back every record that matches both, which DNS calls a record set.
- Shape
- owner TTL IN type data
- A lookup returns
- all records of one name and type
- Reverse records
- live in a different zone
dig example.com MX +noall +answerRecords of one name and type share a TTL. If they differ, RFC 2181 says to treat it as an error and use the lowest.
RFC 1035: DNS implementation ↗RFC 2181: clarifications to the DNS spec ↗
THE POINT TO KEEPA record type is a question; pick the type that asks what the client actually needs, and remember who else reads it.
Every layer keeps a copy
The answer you saw in the lookup doesn't live in one cache. It's copied into the resolver, the operating system and sometimes the app, each counting down on its own.
The zone sets TTL 3600. The resolver got the answer 20 minutes ago, so it hands out the remaining 2400. Each layer counts down from what it received, so no copy can outlive the original by following TTLs alone.
Someone looked up new.example.com before you created it. The resolver cached NXDOMAIN for the lower of the SOA's TTL and its minimum field: 300 seconds. Create the record now and that resolver can keep saying "no" for up to five minutes.
The SOA in a negative answer
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN;; AUTHORITY SECTION:example.com. 300 IN SOA ns1.example.com. hostmaster.example.com. ( 2026101501 7200 3600 1209600 300 )Resolvers apply their own bounds. Unbound caps every TTL at one day and negative answers at an hour by default; operators can also set a floor. A TTL of 604800 in your zone may be held for only 86400.
Your name servers are unreachable and the cached answer has expired. A resolver that implements serve-stale may keep answering with the expired record, using a short TTL, rather than failing. Your site stays reachable; it also means an old answer can outlive its TTL.
OPTIONAL BEHAVIOURRFC 8767 describes serve-stale and suggests a 30-second TTL on stale answers. Whether a resolver does it is a configuration choice.
| Where | What it does |
|---|---|
| OpenJDK | Caches successful lookups for 30 s by default, forever if a security manager is installed; set networkaddress.cache.ttl in java.security. |
| nginx proxy_pass with a fixed host name | Looks the name up when the configuration loads. A variable plus a resolver directive makes it look up at run time. |
| Connection pools, keep-alive, HTTP/2 | Don't look anything up while the connection is open. They keep using the old address until it closes. |
| Browsers | Keep their own short-lived cache in addition to the OS. |
A TTL is a maximum, counted per copy
A cache may drop an answer early, and well-behaved caches never keep it past its TTL. But the clock starts when each copy is made, resolvers bound TTLs with their own limits, and some application caches ignore DNS TTLs altogether.
- TTL set by
- the zone owner, per record set
- Negative TTL
- min(SOA TTL, SOA minimum)
- Unbound default cap
- 86400 s; negative 3600 s
dig shop.example.com +noall +answer # run twiceFlushing your laptop's cache doesn't flush your resolver's, and neither touches the cache inside a running app.
RFC 2308: negative caching ↗RFC 8767: serving stale data ↗Unbound: unbound.conf ↗Java networking properties ↗nginx: proxy_pass ↗RFC 2181: clarifications to the DNS spec ↗
THE POINT TO KEEPEach cache counts down from the TTL it was handed, and some caches ignore TTLs altogether, so a change takes as long as the slowest copy to clear.
Moving the shop to a new address
The shop moves from 192.0.2.10 to 203.0.113.20. Step through the hours around the change and compare four clients that each remember the old address differently.
- LaptopISP resolver, honours TTL
- Office resolvercached seconds before T
- Java servicenetworkaddress.cache.ttl=-1
- Pooled connectionkeep-alive opened at T − 1 h
| Client | T − 48 h | T − 47 h | T: change | T + 2 min | T + 1 h | T + 1 day |
|---|---|---|---|---|---|---|
| TTL published | 3600 → 60 | 60 | 60 | 60 | 60 | 3600 |
| LaptopISP resolver, honours TTL | old address192.0.2.10 | old address192.0.2.10 · TTL 60 | old addressold for ≤ 60 s | new address203.0.113.20 | new address203.0.113.20 | new address203.0.113.20 |
| Office resolvercached seconds before T | old address192.0.2.10 | old address192.0.2.10 · TTL 60 | old addressold, 55 s left | new address203.0.113.20 | new address203.0.113.20 | new address203.0.113.20 |
| Java servicenetworkaddress.cache.ttl=-1 | old address192.0.2.10 | old addresscached forever | old addresscached forever | old address192.0.2.10 | old address192.0.2.10 | brokenfails until restart |
| Pooled connectionkeep-alive opened at T − 1 h | old addressconnected | old addressconnected | old addressno lookup | old addressno lookup | old addressstill connected | brokendrops, reconnects → new |
| Old server | running | running | running | running | running | off |
| Client | T − 48 h | T − 47 h | T: change | T + 2 min | T + 1 h | T + 1 day |
|---|---|---|---|---|---|---|
| TTL published | 3600 | 3600 | 3600 | 3600 | 3600 | 3600 |
| LaptopISP resolver, honours TTL | old address192.0.2.10 | old address192.0.2.10 | old addressold for ≤ 60 min | old addressold for ≤ 58 min | new address203.0.113.20 | new address203.0.113.20 |
| Office resolvercached seconds before T | old address192.0.2.10 | old address192.0.2.10 | old addressold, 59 min left | old addressold, 57 min left | new address203.0.113.20 | new address203.0.113.20 |
| Java servicenetworkaddress.cache.ttl=-1 | old address192.0.2.10 | old addresscached forever | old addresscached forever | old address192.0.2.10 | old address192.0.2.10 | brokenfails until restart |
| Pooled connectionkeep-alive opened at T − 1 h | old addressconnected | old addressconnected | old addressno lookup | old addressno lookup | old addressstill connected | brokendrops, reconnects → new |
| Old server | running | running | running | running | running | off |
You can't recall an answer
Once a cache holds the old record, nothing you change in the zone reaches it before it expires. You control the change through the TTL you published before it: lower it at least one old TTL in advance, then the window after the change is short.
- Lower the TTL
- at least one old TTL before
- Keep the old target
- until its traffic stops
- Not covered by DNS
- pinned caches, open connections
tail the old server's access logClients that cache forever or hold connections open only move when they restart or reconnect. Find them in the old server's logs before you switch it off.
RFC 1034: DNS concepts ↗RFC 2308: negative caching ↗Java networking properties ↗nginx: proxy_pass ↗
THE POINT TO KEEPA DNS change is safe when the old TTL has already been lowered and the old target stays up until its logs go quiet.
How the question travels
Most lookups are one UDP packet each way. Large answers need TCP, and the hop from your laptop to the resolver can be encrypted.
A query and its answer each fit in one UDP datagram on port 53. No connection set-up, so a lookup costs one round trip per server asked.
A signed answer or a long TXT set can exceed the size the client advertised with EDNS (1232 bytes is the common choice). The server sends what fits and sets the TC flag. The client retries over TCP on port 53 and gets the full answer.
dig, when a UDP answer is truncated
;; Truncated, retrying in TCP mode.;; ->>HEADER<<- opcode: QUERY, status: NOERROR;; flags: qr aa; QUERY: 1, ANSWER: 12A firewall allows only UDP 53. Small answers work; large ones never arrive. Lookups for a few names time out or fail, and only after DNSSEC or a long TXT record pushed them over the limit.
TCP IS REQUIREDRFC 7766 makes TCP support mandatory for DNS. Firewalls must allow TCP 53 as well as UDP.
DNS over TLS (port 853) or HTTPS (port 443) hides your questions from the network between your laptop and the resolver. The resolver still sees them, and still talks to authoritative servers in the clear.
Size decides UDP or TCP
Classic DNS limited UDP answers to 512 bytes. EDNS lets each side advertise a bigger buffer, and DNS Flag Day 2020 settled on 1232 bytes to avoid IP fragmentation. Anything larger falls back to TCP, so TCP 53 has to work.
- Plain DNS
- UDP and TCP 53
- DNS over TLS
- TCP 853
- DNS over HTTPS
- 443, alongside web traffic
dig +tcp shop.example.comEncrypting the hop doesn't prove the answer is genuine; a resolver can still pass on a forged record. Proving answers is DNSSEC's job, next.
RFC 1035: DNS implementation ↗RFC 6891: EDNS(0) ↗RFC 7766: DNS over TCP ↗DNS Flag Day 2020: 1232-byte buffer ↗RFC 7858: DNS over TLS ↗RFC 8484: DNS over HTTPS ↗
THE POINT TO KEEPDNS needs both UDP and TCP on port 53; encryption protects one hop from eavesdroppers, not the answer's authenticity.
DNSSEC: a chain of signatures from the root
DNSSEC lets a validating resolver prove that a record came from the zone's owner unchanged. Each zone signs its records; each parent vouches for its child's key.
The resolver trusts the root key it was configured with. The root signs a DS for com, which fingerprints com's key; com signs a DS for example.com; example.com's key made the RRSIG over the shop's A record. Every link checks, so the resolver sets the ad flag.
dig from a validating resolver
;; ->>HEADER<<- opcode: QUERY, status: NOERROR;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1example.com moved to a new DNS provider with new keys, but the registrar still has the old DS. The fingerprint matches no key, the chain breaks, and the resolver refuses to return the answer at all: SERVFAIL. Non-validating resolvers still answer, so it fails for some users only.
dig, and dig with checking disabled
$ dig shop.example.com;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL; EDE: 9 (DNSKEY Missing): (No DNSKEY matches DS RRs of example.com)$ dig +cd shop.example.com +short192.0.2.10com has no DS for example.com, so there is no chain to follow. The answer is returned as insecure: no ad flag and no protection, but no failure either. Most zones are in this state.
| Mechanism | What it protects | What it doesn't |
|---|---|---|
| DNSSEC | The answer came from the zone owner and wasn't changed. | Privacy: queries and answers are readable. |
| DoT / DoH | The hop to the resolver is private and untampered. | Who wrote the answer; the resolver could lie. |
| CAA | Which CAs may issue a certificate for the name. | Lookups. Only CAs read it, only when issuing. |
Validation happens at the resolver
Your laptop usually trusts its resolver's ad flag rather than validating itself. When validation fails, a validating resolver returns SERVFAIL instead of an answer it can't prove, and may add an extended error code saying why.
- DS
- in the parent, set at the registrar
- DNSKEY, RRSIG
- in your zone
- Failure looks like
- SERVFAIL, works with +cd
dig +cd shop.example.com # answers? then DNSSEC is the problemSome signed zones answer a nonexistent name with NOERROR and no records instead of NXDOMAIN (compact denial, RFC 9824). Scripts that look only for NXDOMAIN miss it.
RFC 4033: DNSSEC introduction ↗RFC 4034: DNSSEC records ↗RFC 4035: DNSSEC protocol ↗RFC 8914: extended DNS errors ↗RFC 9824: compact denial of existence ↗RFC 8659: CAA records ↗
THE POINT TO KEEPDNSSEC proves who wrote an answer, encryption hides the question in transit, and CAA governs certificates; none of them does another's job.
Same name, different answer
Who asks matters. The same name can resolve to a public address from the internet and a private one from inside your network.
A laptop asks a public resolver, which walks the public delegation to the public zone and gets 192.0.2.10.
An EC2 instance asks the VPC resolver. A private hosted zone also named example.com is associated with this VPC, so the resolver answers from it: 10.0.1.10, a private address. This is split-horizon DNS; AWS calls it split-view.
The instance looks up status.example.com, which exists only in the public zone. The private zone matches example.com, has no such record, and the resolver returns NXDOMAIN. It does not fall back to the public zone.
NO FALLBACKRoute 53 documents this: a matching private zone without the record returns NXDOMAIN rather than asking public DNS. Copy every name internal clients need into the private zone.
An office DNS server can do the same with BIND views. Each view holds its own copy of example.com and matches clients by source address; a question is answered from the first view that matches. Office PCs get the internal copy.
named.conf: two views of one zone
view "internal" { match-clients { 10.20.0.0/16; }; zone "example.com" { type primary; file "internal/example.com"; };};view "external" { match-clients { any; }; zone "example.com" { type primary; file "external/example.com"; };};When two machines disagree about a name, ask each resolver directly. The answers differ because the questions reached different zones, not because one copy is stale.
Ask both views from the instance
$ dig @10.0.0.2 shop.example.com +short10.0.1.10$ dig @1.1.1.1 shop.example.com +short192.0.2.10The resolver you ask decides the view
Split-horizon DNS serves different records for one name depending on where the question comes from. Cloud private zones, BIND views and corporate DNS servers all do it. Each view is a separate copy of the zone, so a name added to one is missing from the other until you add it there too.
- Route 53 VPC resolver
- VPC CIDR base + 2
- Private zone without the record
- NXDOMAIN, no fallback
- BIND views
- first matching view answers
dig @10.0.0.2 shop.example.com +short; dig @1.1.1.1 shop.example.com +shortA geo or latency routing policy is the same idea driven by the resolver's location instead of its network.
Route 53: private hosted zone considerations ↗BIND 9: views ↗
THE POINT TO KEEPAn answer belongs to a view; when results differ between machines, first ask which resolver each one used.
Which resolver gets the question
Before any view can answer, something decides where a question goes. Pick a place and a name to follow the question, then break one piece of the setup and see which answers change.
The VPN routes corp.example.com, the VPC has a forwarding rule for it, and the corporate server forwards example.com to the VPC.
| Asked from | shop.example.com | status.example.com | db.corp.example.com |
|---|---|---|---|
| Laptop, no VPNhome Wi-Fi | public view192.0.2.10via public zone | public view192.0.2.30via public zone | not foundNXDOMAINvia public zone |
| Laptop on the VPNwg0 · ~corp.example.com | public view192.0.2.10via public zone | public view192.0.2.30via public zone | corporate10.20.0.5via corp zone |
| EC2 instanceVPC resolver 10.0.0.2 | private view10.0.1.10via private zone | not foundNXDOMAINvia private zone | corporate10.20.0.5via corp zone |
| PodCoreDNS, then the VPC | private view10.0.1.10via private zone | not foundNXDOMAINvia private zone | corporate10.20.0.5via corp zone |
| On-premises servercorporate DNS 10.20.0.53 | private view10.0.1.10via private zone | not foundNXDOMAINvia private zone | corporate10.20.0.5via corp zone |
asks for
Check it
- VPN DNS not registered. The tunnel is up, but the VPN client never gave its DNS server and domain to systemd-resolved. It changes: Laptop on the VPN, db.corp.example.com → NXDOMAIN.
- VPN takes all DNS. The VPN sets ~. on wg0, so every name goes to the corporate server. It changes: Laptop on the VPN, shop.example.com → 10.0.1.10; Laptop on the VPN, status.example.com → NXDOMAIN; Laptop on the VPN, db.corp.example.com → 10.20.0.5.
- Rule not associated. The corp.example.com forwarding rule exists but isn't associated with this VPC. It changes: EC2 instance, db.corp.example.com → NXDOMAIN; Pod, db.corp.example.com → NXDOMAIN.
- No forwarder on-premises. The corporate server has no forwarder for example.com and resolves it like any other name. It changes: On-premises server, shop.example.com → 192.0.2.10; On-premises server, status.example.com → 192.0.2.30.
Routing by suffix
Split DNS also means sending different names to different resolvers. A laptop on a VPN sends corp.example.com to the VPN's server and everything else to its usual one. A VPC forwards matching names to your network, and a corporate server forwards names to the VPC. Each picks the longest matching suffix.
- systemd-resolved
- best routing domain, else default route
- Route 53 VPC Resolver
- most specific rule wins
- CoreDNS
- cluster.local itself, the rest forwarded
resolvectl query db.corp.example.com # -- link: wg0A rule for a name also covers its subdomains. In Route 53 a forwarding rule beats a private zone of the same name, and a system rule carves a subdomain back out.
systemd-resolved.service(8) ↗systemd.network(5): Domains= ↗resolvectl(1) ↗Route 53: Resolver rules ↗Route 53: Resolver endpoints ↗Route 53: private hosted zone considerations ↗Kubernetes: customizing DNS ↗Kubernetes: DNS for Services and Pods ↗
THE POINT TO KEEPSplit routing decides which resolver hears a name; when an internal name fails, find which link or rule it left on before blaming the zone.
Diagnose a failed lookup
Applications flatten every DNS problem into "could not resolve". Pick a fault, then run the checks in order to see the evidence that separates them.
curl: (6) Could not resolve host: shop.example.com- CHECK 1
$ dig shop.example.com;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN ;; AUTHORITY SECTION: example.com. 300 IN SOA ns1.example.com. …
NXDOMAIN with the zone's SOA: a server for example.com says the name doesn't exist.
- CHECK 2
$ dig @ns1.example.com shop.example.com +norecurse;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN ;; flags: qr aa
The authoritative server agrees (aa). This isn't a cache; the record really is missing or misspelt.
WHAT TO DOCreate the record. Resolvers that cached the NXDOMAIN keep it for up to the SOA minimum, 300 s here.
curl: (6) Could not resolve host: api.example.com- CHECK 1
$ dig api.example.com A;; ->>HEADER<<- opcode: QUERY, status: NOERROR ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1
NOERROR with no answers: NODATA. The name exists, but has no A record.
- CHECK 2
$ dig api.example.com TXT +short"site-verification=4f2a…"
Only a TXT record was created at this name. Add A (and AAAA) records.
WHAT TO DOAdd the missing type. NODATA is cached like NXDOMAIN, for the SOA minimum.
Node: getaddrinfo EAI_AGAIN shop.example.com · curl: (6) Could not resolve host- CHECK 1
$ dig shop.example.com;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL ; EDE: 9 (DNSKEY Missing): (No DNSKEY matches DS RRs of example.com)
SERVFAIL, and the extended error names a DNSSEC problem.
- CHECK 2
$ dig +cd shop.example.com +short192.0.2.10
With checking disabled the answer comes back, so the data is there and validation is what fails.
- CHECK 3
$ dig example.com DS +short; dig @ns1.example.com example.com DNSKEY +multi12345 13 2 3F5A… … ; KSK; alg = ECDSAP256SHA256 ; key id = 48213
The parent's DS points at key 12345; the zone now publishes 48213.
WHAT TO DOUpdate the DS at the registrar to match the new key, or remove it to go unsigned. Expect the fix to take up to the DS TTL.
Lookups take seconds, then fail with SERVFAIL for some users- CHECK 1
$ dig +trace shop.example.comexample.com. 172800 IN NS ns1.old-dns.example.net. ;; Received 98 bytes from 192.5.6.30#53(a.gtld-servers.net)
The com zone still delegates to the old provider.
- CHECK 2
$ dig @ns1.old-dns.example.net example.com SOA +norecurse;; ->>HEADER<<- opcode: QUERY, status: REFUSED
The old provider no longer serves the zone.
- CHECK 3
$ dig @ns1.example.com shop.example.com +norecurse +short192.0.2.10
Your new servers are fine, which is why this hides so well.
WHAT TO DOChange the NS records at the registrar to the new servers. Resolvers keep the old delegation for up to its TTL, two days for com.
Every lookup hangs for seconds, then: Temporary failure in name resolution- CHECK 1
$ dig shop.example.com;; communications error to 10.0.0.2#53: timed out ;; no servers could be reached
Nothing came back from the configured resolver.
- CHECK 2
$ dig @1.1.1.1 shop.example.com +short192.0.2.10
Another resolver answers, so the zone is fine. The problem is your resolver or the path to it.
- CHECK 3
$ resolvectl status # or cat /etc/resolv.confCurrent DNS Server: 10.0.0.2
Check the resolver is up and that security groups or firewalls allow UDP and TCP 53 to it.
WHAT TO DOFix or remove the dead resolver. glibc waits 5 s for a server before trying the next, so a dead first resolver adds about 5 s to every lookup; with no working server it gives up after two rounds.
ssh: Could not resolve hostname db.corp.example.com: Name or service not known- CHECK 1
$ resolvectl query db.corp.example.comdb.corp.example.com: Name 'db.corp.example.com' not found
A DNS server answered, and the answer was no. The tunnel isn't the problem.
- CHECK 2
$ resolvectl status wg0Link 3 (wg0) Current Scopes: none Protocols: -DefaultRoute -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupportedThe VPN link has no DNS server and no routing domain, so the question went out on the home link.
- CHECK 3
$ dig @10.20.0.53 db.corp.example.com +short10.20.0.5
The VPN's DNS server knows the name. Nothing sent the question there.
WHAT TO DOGive the VPN link its server and routing domain. resolvectl dns wg0 10.20.0.53 and resolvectl domain wg0 '~corp.example.com' fix it until the next reconnect; set them in the VPN client for good.
curl connects to 10.0.0.5 and times out; dig shows 192.0.2.10- CHECK 1
$ dig +short shop.example.com192.0.2.10
DNS is correct.
- CHECK 2
$ getent ahosts shop.example.com10.0.0.5 STREAM shop.example.com
The system resolver library returns something else.
- CHECK 3
$ grep shop /etc/hosts10.0.0.5 shop.example.com
An old hosts entry wins before DNS is asked.
WHAT TO DORemove the entry. If getent agrees with dig and the app still disagrees, look for a cache inside the app.
Read the status, then ask the source
Start with the status and flags, not the application's message. NXDOMAIN and NODATA are answers from the zone. SERVFAIL means a resolver couldn't produce a trustworthy answer. A timeout means nobody answered. Then ask the authoritative server directly with +norecurse to take caches out of the picture.
- NXDOMAIN
- name doesn't exist
- NOERROR, 0 answers
- name exists, type doesn't
- SERVFAIL
- resolver gave up: DNSSEC, lame, unreachable
dig name → dig @auth name +norecurse → dig +trace → dig +cdApplication errors don't keep these distinctions. glibc reports SERVFAIL and timeouts as EAI_AGAIN, a temporary failure that callers often retry.
RFC 2308: negative caching ↗RFC 8914: extended DNS errors ↗BIND 9: dig ↗resolv.conf(5) ↗getaddrinfo(3) ↗RFC 4035: DNSSEC protocol ↗resolvectl(1) ↗
THE POINT TO KEEPName the DNS status before guessing: NXDOMAIN and NODATA come from the zone, SERVFAIL from the resolver, a timeout from the path.
A loop for every DNS change
Most DNS incidents come from changes. The same six steps cover a record move, a provider migration and a key roll.
Lower the TTL on the records you will change, then wait at least the old TTL so every cache holds the short one.
Change one thing. Confirm the authoritative servers serve it, with the serial raised, before anyone else sees it.
Ask every authoritative server
for ns in $(dig +short example.com NS); do dig @$ns shop.example.com A +norecurse +shortdoneCheck from outside through several public resolvers and from inside your networks. Keep the old target running until its logs go quiet.
Raise the TTL back. Long TTLs reduce query load and keep names resolving through short outages of your name servers.
| Signal | Why |
|---|---|
| SERVFAIL and timeout rate at your resolvers | Rises with DNSSEC breaks, lame delegations and unreachable servers. |
| Registrar NS matches zone NS | Catches half-finished provider moves. |
| RRSIG expiry dates and DS key tags | Signatures expire on schedule; DS must track key rolls. |
| Lookup latency from your apps | A slow first resolver adds seconds to every request. |
Plan for the caches you can't see
Every step in the loop is about time. You can't flush other people's caches, so you shorten their memory before a change and wait for evidence, not a clock, before removing the old target.
- Before
- lower TTL, wait old TTL
- During
- one change, verify at the source
- After
- drain, then raise TTL
dig +trace example.com NSAvoid TTL 0 as a permanent setting. It sends every lookup to your servers and some resolvers raise it anyway.
RFC 1034: DNS concepts ↗RFC 2308: negative caching ↗RFC 4035: DNSSEC protocol ↗BIND 9: dig ↗
THE POINT TO KEEPShorten caches before a change, verify at the authoritative servers, and keep the old target alive until the evidence says nobody uses it.