Observability & Troubleshooting
Put on your detective hat and diagnose unreachable services, saturated resources, and kernel faults by gathering empirical system evidence.
What this chapter builds
This chapter combines the previous seven. Start from the caller symptom, preserve evidence, localize the first broken boundary, and change one variable at a time.
By the end, you can
- Turn a vague outage report into competing hypotheses
- Choose probes that discriminate between layers
- Recognize CPU, memory, storage, FD, and network saturation
- Verify recovery from both system and caller perspectives
Objects to keep straight
pressure countersqueuesresource limitsjournal recordssocket and task stateApplied task: Diagnose eight “site down” fixtures without restarting the service as the opening move.
Deconstruct service outages into verifiable failure signatures.
When a user reports "the site is down," how do you isolate network, host, and application root causes?
A client error code is the first clue in an incident investigation. A "Connection refused" error means the target IP received the packet, but no socket was listening on that port, causing the kernel to respond immediately with a TCP RST packet. A "Connection timed out" error means packets vanished into the void—typically dropped by an ingress firewall or security group. Form a hypothesis based on the error signature and verify each layer with specific commands before modifying system state.
Model scope. Reverse proxies (nginx, AWS ALB) return HTTP 502 (Bad Gateway) when the upstream application crashes, and HTTP 504 (Gateway Timeout) when the application hangs or exhausts its worker pool.
* Trying 10.0.1.17:443... * TCP_NODELAY set * connect to 10.0.1.17 port 443 failed: Connection refused * Failed to connect to devbox-01 port 443: Connection refused * Closing connection 0 curl: (7) Failed to connect to devbox-01 port 443: Connection refused
PING 10.0.1.17 (10.0.1.17) 56(84) bytes of data. 64 bytes from 10.0.1.17: icmp_seq=1 ttl=64 time=0.210 ms 64 bytes from 10.0.1.17: icmp_seq=2 ttl=64 time=0.198 ms --- 10.0.1.17 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss
The USE Method: Systematically inspect Utilization, Saturation, and Errors.
How do senior SREs triage an unfamiliar server incident without guessing or running random commands?
Instead of guessing or blindly restarting daemons, apply Brendan Gregg's USE Method: for every hardware and software resource (CPU, Memory, Storage, Network), inspect Utilization (percentage of time the resource was busy), Saturation (the degree to which extra work is queued waiting for service), and Errors (explicit fault counters). Gathering empirical evidence across these four dimensions isolates the constrained bottleneck within minutes.
Model scope. High CPU utilization alone is not an incident if latency and error rates remain within acceptable SLOs; prioritize diagnosing Saturation (processes queued waiting for CPU or disk).
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 0 204800 50000 1500000 0 0 12 45 500 800 15 5 78 2 0 1 0 0 204650 50000 1500000 0 0 0 0 450 750 12 4 84 0 0 1 0 0 204600 50000 1500000 0 0 0 20 480 790 14 5 81 0 0
Filesystem Size Used Avail Use% Mounted on /dev/sda1 50G 18G 30G 38% /
Systemd journald and /var/log provide high-fidelity chronological flight data.
How do you navigate systemd's structured binary journal to find the exact moment an outage began?
Stage 01: Filter strictly by unit name (-u) and time window (--since) to isolate the target service failure without drowning in unrelated system logs.
Sep 16 14:02:11 devbox systemd[1]: Starting A high performance web server... Sep 16 14:02:11 devbox nginx[890]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use) Sep 16 14:02:11 devbox systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE Sep 16 14:02:11 devbox systemd[1]: nginx.service: Failed with result 'exit-code'.
Filter strictly by unit name (-u) and time window (--since) to isolate the target service failure without drowning in unrelated system logs.
Sep 16 14:02:11 devbox nginx[890]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use) Sep 16 14:02:11 devbox systemd[1]: Failed to start A high performance web server.
Filter by syslog priority (-p err..alert) for the current boot session (-b) to reveal all critical host-level warnings and errors.
Sep 16 13:55:01 devbox kernel: Out of memory: Killed process 842 (python3) total-vm:2097152kB, anon-rss:1048576kB, file-rss:0kB Sep 16 13:55:01 devbox kernel: oom_reaper: reaped process 842 (python3), now anon-rss:0kB
Use -k to inspect the kernel message ring buffer (dmesg equivalent), exposing hardware faults, link state changes, and OOM killer events.
2026/09/16 14:02:11 [emerg] 890#890: bind() to 0.0.0.0:443 failed (98: Address already in use)
Not all application logs route to journald; traditional daemons frequently maintain dedicated text log files in /var/log/.
Modern Linux systems centralize logging in systemd-journald, which captures standard output, standard error, syslog events, and kernel messages into structured binary journal files. Querying journald with targeted flags (-u for service unit, -p for severity priority, -k for kernel events, and --since for time boundaries) allows operators to isolate failure sequences in seconds.
Model scope. If /var/log/journal does not exist, journald stores logs in /run/log/journal/ (volatile RAM), causing all historical logs to be lost on system reboot.
Sep 16 14:02:11 devbox nginx[890]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use) Sep 16 14:02:11 devbox systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
[Wed Sep 16 13:55:01 2026] Out of memory: Killed process 842 (python3) total-vm:2097152kB, anon-rss:1048576kB, file-rss:0kB
Resource exhaustion hits 4 distinct operational cliffs; each has unique symptoms.
Which resource limit causes cryptic application crashes when CPU and general memory appear healthy?
Operating system failures frequently stem from hitting one of four critical resource ceilings: Data Blocks (df -h full), Inode Slots (df -i full, usually caused by millions of tiny session or cache files), File Descriptors (reaching per-process RLIMIT_NOFILE or system file-max), or Memory (where the kernel invokes the OOM killer to terminate processes and salvage the host). Each condition triggers distinct error codes in application logs.
Model scope. Per-process file descriptor limits (configured via /etc/security/limits.conf or systemd LimitNOFILE=) are typically set much lower (1024 or 65536) than the system-wide maximum.
Filesystem Size Used Avail Use% Mounted on /dev/sda1 50G 49G 0 100% / Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda1 3.2M 3.2M 0 100% /
1280 0 524288 # Allocated FDs: 1280 · Free allocated: 0 · Maximum allowable FDs: 524,288
Deconstructing CPU load average, memory pressure, and swap thrashing.
Why can a system have high load average with low CPU utilization, and what does "free" memory really mean on Linux?
Stage 01: Load average reflects threads in Runnable (R) state PLUS Uninterruptible Sleep (D, waiting for disk/NFS I/O). A load average higher than your CPU core count with low %CPU indicates an I/O bottleneck, not CPU saturation.
14:20:01 up 10 days, 4:12, 2 users, load average: 8.52, 6.10, 4.85
Load average reflects threads in Runnable (R) state PLUS Uninterruptible Sleep (D, waiting for disk/NFS I/O). A load average higher than your CPU core count with low %CPU indicates an I/O bottleneck, not CPU saturation.
total used free shared buff/cache available Mem: 7.8Gi 4.1Gi 300Mi 50Mi 3.4Gi 3.3Gi Swap: 2.0Gi 100Mi 1.9Gi
Ignore "free" memory. Linux aggressively repurposes unused RAM as Page Cache and buffers to accelerate disk reads. "Available" memory is the true metric—the RAM reclaimable without swapping.
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 4 2 102400 307200 51200 3481600 1024 512 2048 1024 2500 4000 45 20 15 20 0
Inspect "si" (swap-in) and "so" (swap-out). Non-zero continuous swap I/O indicates memory exhaustion and active thrashing, leading to massive latency spikes.
%Cpu(s): 45.0 us, 20.0 sy, 0.0 ni, 15.0 id, 20.0 wa, 0.0 hi, 0.0 si, 0.0 st MiB Mem : 8000.0 total, 300.0 free, 4200.0 used, 3500.0 buff/cache PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 842 myapp 20 0 4.2g 3.1g 20000 D 45.0 40.0 120:05.10 python3
Inspect CPU states: "us" (user space), "sy" (kernel syscalls), "wa" (iowait waiting on disk), and "st" (hypervisor steal time). Notice PID 842 in "D" state (uninterruptible disk wait).
Understanding Linux performance requires looking past surface metrics. A machine with only 300 MB of "free" RAM may actually have 3.3 GB of "available" memory ready to be reclaimed from the kernel Page Cache. Similarly, a high load average combined with high "%wa" (iowait) indicates that processes are blocked waiting on slow storage disks rather than competing for CPU compute cores.
Model scope. In virtualized cloud instances (AWS EC2, GCP Compute Engine), high "%st" (steal time) in top means the underlying physical CPU is oversubscribed and other virtual machines are starving your CPU.
total used free shared buff/cache available Mem: 7.8Gi 4.1Gi 300Mi 50Mi 3.4Gi 3.3Gi Swap: 2.0Gi 100Mi 1.9Gi
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu----- r b swpd free buff cache si so bi bo in cs us sy id wa st 1 0 102400 307200 51200 3481600 0 0 0 0 500 800 10 4 86 0 0
Interactive incident simulation matrix: Diagnose 8 real-world host failure modes.
Can you correctly identify whether a service failure is caused by DNS, bind scope, firewall drop, OOM, or disk exhaustion?
Service active; socket bound to 127.0.0.1:443. Remote connections cannot reach that bind address.
Identical surface symptoms—such as "website is down" or "API unreachable"—can stem from completely distinct root causes across the stack. A stopped service refuses connections instantly; a firewall DROP causes client connection timeouts; a DNS failure aborts before network transmission; and kernel OOM kills the worker process without warning. You must cross-examine evidence from multiple subsystems to isolate the truth.
Model scope. Use the interactive Incident Simulation selector above to load and diagnose all 8 real-world failure fixtures.
active LISTEN 0 511 127.0.0.1:443 0.0.0.0:*
Sep 16 10:00:00 devbox systemd[1]: Started A high performance web server.
The 60-Second Linux Triage Runbook: First commands on an unfamiliar broken host.
When an emergency alert fires and you SSH into a broken server, what exact command sequence should you run?
Action 01: Check Uptime & Load Average
14:20:01 up 10 days, 4:12, 2 users, load average: 0.15, 0.20, 0.18
Field Note: How long has the machine been up? Did an unexpected reboot occur? Is load average spiking above CPU core count?
[Wed Sep 16 14:10:00 2026] EXT4-fs error (device sda1): remounted read-only
Field Note: Look for hardware errors, filesystem remounts to read-only (EROFS), OOM killer executions, and network link drops.
total used free shared buff/cache available Mem: 7.8Gi 4.1Gi 300Mi 50Mi 3.4Gi 3.3Gi Swap: 2.0Gi 100Mi 1.9Gi
Field Note: Verify available memory headroom and ensure swap is not actively thrashing.
Filesystem Size Used Avail Use% Mounted on /dev/sda1 50G 48G 2G 96% / Filesystem Inodes IUsed IFree IUse% Mounted on /dev/sda1 3.2M 3.1M 100K 97% /
Field Note: Check whether data blocks or inode slots are approaching 100% exhaustion on root (/) or /var.
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
tcp LISTEN 0 128 0.0.0.0:22 0.0.0.0:* users:(("sshd",pid=710,fd=3))
tcp LISTEN 0 511 0.0.0.0:443 0.0.0.0:* users:(("nginx",pid=750,fd=6))Field Note: Are expected service ports open and bound to 0.0.0.0 (all interfaces) rather than 127.0.0.1 (loopback)?
UNIT LOAD ACTIVE SUB DESCRIPTION 0 loaded units listed.
Field Note: Did any core background service or scheduled timer unit crash or enter a failed restart state?
%Cpu(s): 12.0 us, 4.0 sy, 0.0 ni, 82.0 id, 2.0 wa, 0.0 hi, 0.0 si, 0.0 st PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 750 nginx 20 0 145200 24500 12000 S 5.0 0.3 5:12.30 nginx
Field Note: Identify processes consuming excessive CPU, memory, or pinned in uninterruptible disk wait (D state).
When responding to a production incident, follow a standardized diagnostic ladder before making changes or restarting daemons. The 60-second triage runbook systematically checks CPU load, kernel hardware logs, memory pressure, storage block/inode headroom, listening sockets, failed systemd units, and top processes. Gathering this baseline evidence prevents destroying diagnostic clues.
Model scope. Never blindly reboot or restart services during an incident before gathering triage evidence; restarting clears transient logs, memory states, and network sockets.
14:20:01 up 10 days, 4:12, 2 users, load average: 0.15, 0.20, 0.18
total used free shared buff/cache available
Mem: 7.8Gi 4.1Gi 300Mi 50Mi 3.4Gi 3.3Gi
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 50G 18G 30G 38% /0 loaded units listed.