Networking & Sockets
Follow network packets from physical wire reception through Netfilter firewall chains, the Forwarding Information Base (FIB), and socket queues into application memory buffers.
What this chapter builds
This chapter builds from sockets and routes to DNS, firewall hooks, listener scope, and network namespaces so “the port is open” stops being the whole diagnosis.
By the end, you can
- Trace an outbound packet through lookup, route, policy, neighbour, and device
- Explain what bind address changes
- Separate DNS, routing, firewall, handshake, and application failures
- Map container networking to namespaces, veth pairs, and bridges
Objects to keep straight
socketsk_buffFIBneighbour tablenetwork namespacevethApplied task: Prove why a service that listens on 127.0.0.1:443 is healthy locally but unreachable remotely.
Before looking at the mechanics, what actually is a Socket or a Port?
What are the foundational primitives of Linux Networking?
IP Address
A numerical label assigned to a device (like 10.0.1.17). It ensures packets can find their way across the internet to the correct machine.
Port
Once a packet reaches the correct machine (IP), the Port (like 443 or 80) tells the kernel WHICH specific application should receive the data.
Socket
A Socket is a software mailbox created by an application. It binds to a Port so the kernel knows to deliver incoming packets for that Port into this specific mailbox.
Netfilter (Firewall)
A kernel subsystem that intercepts every packet arriving or leaving. It checks rules (iptables/nftables) and decides whether to Accept (let it through) or Drop (discard it silently).
Networking in Linux is simply about moving bytes between memory buffers. An application asks the kernel for a Socket (a mailbox). When packets arrive off the physical wire, the kernel checks the IP, checks the Firewall, checks the Port, and finally drops the payload into the Socket's buffer for the application to read.
Model scope. A "Connection Refused" error means the packet reached the machine, but there was no Socket listening on that Port. A "Timeout" means a Firewall dropped it entirely.
State Recv-Q Send-Q Local Address:Port Peer Address:Port LISTEN 0 511 0.0.0.0:443 0.0.0.0:*
inet 10.0.1.17/24 brd 10.0.1.255 scope global eth0
A listening port is an entry in the kernel TCP protocol control table.
How does an application bind a port, and how does the kernel queue connections?
All steps are visible. Advance to inspect one transition.
A port is not physical hardware; it is a 16-bit integer index into the kernel TCP protocol state table. When a process calls listen(fd, backlog), the kernel allocates two FIFO queues: the SYN backlog (for connections completing the 3-way handshake) and the Accept queue (for fully established connections waiting for accept()). If the application event loop stalls and the Accept queue fills up, incoming client SYN packets are silently dropped or rejected with TCP resets.
Model scope. Linux normally protects ports below the namespace's ip_unprivileged_port_start value; CAP_NET_BIND_SERVICE can satisfy that check. Listener queue columns and their exact meaning depend on socket state and the ss view, so corroborate them with socket and kernel counters.
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 511 0.0.0.0:443 0.0.0.0:* users:(("nginx",pid=421,fd=11),("nginx",pid=418,fd=11))4096 # Maximum listen backlog queue length
The 6-stage journey of an egress packet: from socket to wire.
What kernel decisions execute when a process calls curl https://example.com?
Initiating an outbound network connection requires six sequential kernel operations. First, DNS translates the hostname to an IP. Next, the kernel allocates an ephemeral client port. The routing engine evaluates the Forwarding Information Base (FIB) to select the egress interface (eth0) and next hop. The Netfilter OUTPUT chain verifies firewall policy. The neighbor table (ARP) resolves the gateway IP to an Ethernet MAC address. Finally, the packet is queued in the device driver TX ring for DMA transmission.
Model scope. If the gateway MAC is unresolvable (ARP failure), the packet is dropped with "Destination Host Unreachable", even if DNS and routing tables are completely valid.
* Trying 93.184.215.14:443... * Connected to 93.184.215.14 port 443 * TLS Handshake completed
93.184.215.14 via 10.0.1.1 dev eth0 src 10.0.1.17 uid 33
cacheNext-hop routing (FIB) vs physical hardware addressing (ARP/NDP).
How does Linux know where to forward a packet on the local subnet?
The Linux kernel does not know the physical path to a remote internet destination; it only knows the next hop gateway. The Forwarding Information Base (FIB) uses longest prefix matching to select the best route. Once the next-hop IP is determined, the kernel queries its Neighbor Cache (ARP for IPv4, NDP for IPv6) to locate the gateway physical 48-bit MAC address. The packet IP payload is untouched, but the L2 Ethernet header is stamped with the gateway MAC.
Model scope. Neighbor states progress: INCOMPLETE → REACHABLE → STALE → DELAY → PROBE → FAILED. A STALE entry still sends packets while triggering background reachability probes.
default via 10.0.1.1 dev eth0 proto dhcp src 10.0.1.17 metric 100 10.0.1.0/24 dev eth0 proto kernel scope link src 10.0.1.17
10.0.1.1 dev eth0 lladdr 00:16:3e:5f:aa:11 REACHABLE
Domain name resolution: NSS, systemd-resolved, and stub resolvers.
How does Linux resolve names, and why can dig succeed while curl fails?
Stage 01: nsswitch.conf governs system resolution order. files checks /etc/hosts first; dns invokes resolver libraries.
hosts: files dns
nsswitch.conf governs system resolution order. files checks /etc/hosts first; dns invokes resolver libraries.
nameserver 127.0.0.53 options edns0 trust-ad search c.internal
127.0.0.53 is the local loopback stub resolver managed by systemd-resolved. It provides local DNS caching and search domain expansion.
Link 2 (eth0)
Current DNS Server: 10.0.0.2
DNS Servers: 10.0.0.2
DNS Domain: c.internalresolvectl shows the real upstream recursive DNS servers assigned per-interface via DHCP or cloud metadata.
93.184.215.14 example.com
Use getent to test the host's configured NSS lookup path. Use dig when the narrower question is what a DNS server returns; the two commands answer different questions.
Name resolution in modern Linux involves multiple abstraction layers. Applications call glibc getaddrinfo(), which consults /etc/nsswitch.conf. If /etc/hosts has no matching entry, it delegates to the resolver specified in /etc/resolv.conf. In modern distributions, this points to 127.0.0.53—the local systemd-resolved caching daemon—which forwards queries to the interface DNS servers assigned by DHCP.
Model scope. Alpine Linux and musl-libc do not support /etc/nsswitch.conf or search domain ndots logic the same way glibc does, leading to famous Kubernetes DNS timeouts in Alpine images.
devbox-01: 10.0.1.17 -- Information acquired via protocol DNS in 1.2ms.
nameserver 127.0.0.53 search c.internal
Listener binding: how the IP address restricts connection scope.
Why does an app respond to curl on localhost but reject connections from the network?
Only clients in this network namespace can use the loopback listener.
When an application calls bind(fd, sockaddr, len), the IP address in sockaddr determines the packet reception filter. Binding to 0.0.0.0 (INADDR_ANY) instructs the kernel to accept packets destined for any IP assigned to any local network interface. Binding to 127.0.0.1 restricts the socket to packets originating on the loopback interface (lo); packets arriving from eth0 are rejected at the routing layer before reaching the socket.
Model scope. IPv6 uses :: for wildcard binding and ::1 for loopback. By default on Linux, binding to [::]:443 also accepts IPv4 connections via IPv4-mapped IPv6 addresses unless IPV6_V6ONLY is set.
State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
LISTEN 0 128 127.0.0.1:8080 0.0.0.0:* users:(("app",pid=2048,fd=3))
LISTEN 0 511 0.0.0.0:443 0.0.0.0:* users:(("nginx",pid=421,fd=11))HTTP/1.1 200 OK Server: internal-worker
Netfilter architecture: tables, hooks, and packet processing order.
Where do firewall rules evaluate during packet transit through the kernel?
| Hook Point | When it Executes | Common Tables | Operational Purpose |
|---|---|---|---|
| PREROUTING | Immediately after NIC driver receives frame | raw, mangle, nat | DNAT (port forwarding), connection tracking (conntrack) |
| INPUT | After routing confirms packet is for local host | mangle, filter, security | Host firewall rules: allow/drop incoming traffic to local sockets |
| FORWARD | After routing determines packet is for another host | mangle, filter, security | Container/router forwarding: Kubernetes Pod-to-Pod traffic |
| OUTPUT | When a local process transmits an outbound packet | raw, mangle, nat, filter | Egress firewall rules: restrict outbound connections from host |
| POSTROUTING | Just before packet is handed to NIC driver | mangle, nat | SNAT / Masquerade: replace source IP with public interface IP |
Netfilter is the kernel framework; iptables, nftables, and firewalld are user space frontends. Rules are evaluated sequentially within priority-ordered tables.
Netfilter provides five hooks inside the kernel network stack where packet processing can be intercepted. Inbound packets hit PREROUTING where DNAT translates destination addresses. The kernel routing engine makes a routing decision: if addressed to the host, the packet hits INPUT before entering the socket; if addressed to another host (e.g. a container), it traverses FORWARD. Outbound packets traverse OUTPUT and POSTROUTING for SNAT masquerading.
Model scope. Connection tracking (conntrack) maintains state for stateful firewalls. If the conntrack table fills up (/proc/sys/net/netfilter/nf_conntrack_max), the kernel drops all new connections!
table inet filter {
chain input {
type filter hook input priority filter; policy accept;
ct state established,related accept
tcp dport 443 accept
iifname "lo" accept
drop
}
}num pkts bytes target prot opt in out source destination 1 4210K 3.2G ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443
Container network isolation: private interfaces, routes, and veth pairs.
How does Linux give Docker containers and Kubernetes Pods isolated networking?
A network namespace provides an isolated copy of the entire network stack: its own network interfaces, routing tables, firewall rules, and socket lists. To connect a container to the host, the kernel creates a Virtual Ethernet pair (veth). One end remains in the host root namespace plugged into a bridge (like cbr0 or docker0), while the peer end is moved into the container namespace and renamed eth0.
Model scope. To inspect container networking from the host without ip netns, use nsenter: sudo nsenter -t <container-pid> -n ip addr.
cgroup_ns_web (id: 1) container_db (id: 2)
1: lo: <LOOPBACK,UP> mtu 65536 2: eth0@if14: <BROADCAST,MULTICAST,UP> mtu 1500 inet 10.244.1.5/24
Isolating dropped traffic: firewall reject, routing blackhole, or port closed.
A client connection fails with timeout or connection refused. Where is it breaking?
Default fixture: Connection failures present three distinct symptoms that identify the failing layer: 1. "Connection Refused" means the packet reached the host and was allowed by the firewall, but no process was listening on the port (kernel sent TCP RST); 2. "Connection Timed Out" means the packet was dropped by a firewall (or routing blackhole) without responding; 3. "No route to host" means ARP failed or the routing table has no gateway.
Connection failures present three distinct symptoms that identify the failing layer: 1. "Connection Refused" means the packet reached the host and was allowed by the firewall, but no process was listening on the port (kernel sent TCP RST); 2. "Connection Timed Out" means the packet was dropped by a firewall (or routing blackhole) without responding; 3. "No route to host" means ARP failed or the routing table has no gateway.
Model scope. tcpdump captures packets BEFORE Netfilter ingress rules drop them! Seeing packets in tcpdump proves they reached the NIC, but does NOT prove they reached the application.
19:40:00.120 IP 192.168.1.5.54321 > 10.0.1.17.443: Flags [S], seq 123456 # SYN arrives 19:40:01.124 IP 192.168.1.5.54321 > 10.0.1.17.443: Flags [S], seq 123456 # Retried! Host never responded (DROP)
chain input {
type filter hook input priority 0; policy accept;
tcp dport 443 drop # CAUSE OF OUTAGE
}