Processes & Services
Watch an executable file on disk wake up and become an active kernel task with virtual memory segments, open file descriptors, signal masks, and a supervised systemd lifecycle.
What this chapter builds
A program file is passive. This chapter follows it through process creation, execution, scheduling, descriptors, signals, service supervision, and failure.
By the end, you can
- Separate an executable, process, thread, and service unit
- Read a process tree and recognize zombie/orphan behavior
- Predict what common signals can and cannot guarantee
- Debug a failed systemd service using state and logs
Objects to keep straight
task_structmm_structfiles_structsignal statecgroupApplied task: Determine why nginx is installed but has no healthy worker or listening socket.
Before looking at the mechanics, what actually is a Process?
What are the foundational primitives of Linux Processes?
Process
A program is just a dead file on disk. When you execute it, the kernel creates a "Process" — a living instance with its own memory, CPU time, and a unique integer ID called a PID.
Thread
A thread is a lightweight execution unit inside a process. Multiple threads share the same memory space, allowing them to collaborate quickly (unlike separate processes).
Fork & Exec
Linux creates processes via a two-step dance. "Fork" clones the parent exactly. "Exec" replaces the clone's brain with a brand new program.
Signal
A primitive way for the OS or other processes to interrupt a running process. For example, SIGKILL tells the process to drop dead immediately, while SIGTERM asks it to shut down gracefully.
Linux views everything that executes as a "task" represented by a massive C structure in the kernel called task_struct. Whether it is a full-blown process or a lightweight thread, the kernel schedules them all fundamentally the same way. The only difference is what resources (like memory) they are told to share.
Model scope. The POSIX standard strictly defines "processes" and "threads", but the Linux kernel internally only knows about "tasks". A thread is just a task that shares its virtual memory pointer with another task.
UID PID PPID LWP C NLWP SZ RSS PSR STIME TTY TIME CMD root 1 0 1 0 1 25484 9636 2 10:00 ? 00:00:02 /sbin/init root 132 1 132 0 1 12344 4321 1 10:00 ? 00:00:00 /lib/systemd/systemd-journald
Tgid: 4512 Pid: 4512
A program is passive ELF machine code; a process is an active kernel task.
How does an executable file on disk become a live running process?
All steps are visible. Advance to inspect one transition.
Transforming an executable into a running process requires two fundamental kernel operations. First, fork() or clone() creates a duplicate of the calling process, allocating a new task_struct and duplicating memory page table pointers marked Copy-on-Write (CoW). Next, execve() unmaps the old address space, inspects the ELF binary header, creates fresh virtual memory segments (Text, Data, Heap, Stack), loads shared libraries, and transfers CPU execution to the program entry point.
Model scope. Copy-on-Write (CoW) ensures fork() is fast: physical RAM pages are shared read-only until either parent or child writes to memory, triggering a minor page fault to copy the page.
418 1 root Ss nginx 421 418 www-data S nginx
/usr/sbin/nginx
The process hierarchy: parentage, orphan adoption, and zombie states.
Who started this process, and what happens when a parent process crashes?
Click a process in the tree to view its live kernel struct, memory map, and signal controls.
All Linux processes exist within a strict genealogical tree rooted at PID 1 (systemd). When a child process terminates, it transitions to state EXIT_ZOMBIE (Z) where its memory is freed, but its exit status remains in the kernel task list until its parent calls waitpid(). If a parent terminates before its children, the kernel reparents the surviving orphaned children to PID 1, which periodically reaps them.
Model scope. If a parent process is stuck in an uninterruptible sleep (D state) and fails to call wait(), zombie children accumulate in the process table and cannot be killed by kill -9.
nginx(418)─┬─nginx(421)
└─nginx(422)PPid: 418
File Descriptors: how processes interface with files, pipes, and sockets.
How does a process address open files, network connections, and pipes?
To a Linux process, all external resources are accessed through File Descriptors (FDs)—small non-negative integers indexing into the process files_struct table. Each FD points to a kernel struct file containing the current read/write offset, access mode flags, and pointers to the underlying inode or socket buffer. When an application exhausts its configured file descriptor limit (RLIMIT_NOFILE), further open() or accept() calls fail with EMFILE (Too many open files).
Model scope. The system-wide open file limit is controlled by /proc/sys/fs/file-max, while per-process limits are configured via ulimit -n or LimitNOFILE= in systemd units.
lrwx------ 1 www-data www-data 64 Sep 18 19:40 0 -> /dev/null lrwx------ 1 www-data www-data 64 Sep 18 19:40 1 -> /dev/null l-wx------ 1 www-data www-data 64 Sep 18 19:40 7 -> /var/log/nginx/access.log lrwx------ 1 www-data www-data 64 Sep 18 19:40 11 -> socket:[123456]
Max open files 65535 65535 files
Asynchronous kernel notifications and process interrupt handling.
Which signal should you send, and what happens when the kernel delivers it?
[kernel scheduler] PID 421 active. Ready to receive signal...
| Signal | Number | Catchable? | Default Action | Operational Purpose |
|---|---|---|---|---|
| SIGHUP | 1 | Yes | Terminate | Reload configuration without dropping connections (nginx, sshd) |
| SIGINT | 2 | Yes | Terminate | Terminal interrupt (Ctrl+C): polite request to cancel active work |
| SIGQUIT | 3 | Yes | Core Dump | Terminal quit (Ctrl+\): stops execution and writes core dump file |
| SIGKILL | 9 | NO (Uncatchable) | Terminate | Kernel unconditionally evicts task from runqueue; no cleanup runs |
| SIGUSR1 | 10 | Yes | Terminate | Application-defined: commonly used to reopen rotated log files |
| SIGTERM | 15 | Yes | Terminate | Polite shutdown request: allows process to finish work and close sockets |
| SIGSTOP | 19 | NO (Uncatchable) | Stop / Freeze | Kernel immediately freezes process execution; resumes on SIGCONT |
| SIGCONT | 18 | Yes | Continue | Resumes execution of a previously stopped process |
SIGTERM is the default signal sent by kill and systemctl stop. Always attempt SIGTERM before SIGKILL; SIGKILL prevents in-flight buffers from flushing and corrupts database state.
Signals are asynchronous software interrupts delivered by the kernel to a process task_struct. When a signal arrives, the kernel checks the process signal mask (sigset_t). If unblocked, the kernel interrupts user space execution and jumps to an application-installed signal handler function. If no handler exists, the kernel executes the default action (terminate, core dump, ignore, or stop). Two signals CANNOT be caught, blocked, or ignored: SIGKILL (9) and SIGSTOP (19).
Model scope. Processes in state D (Uninterruptible Sleep) are blocked waiting for hardware I/O and will NOT respond to any signal—including SIGKILL—until the I/O completes.
1) SIGHUP 2) SIGINT 3) SIGQUIT 4) SIGILL 5) SIGTRAP 6) SIGABRT 9) SIGKILL 10) SIGUSR1 11) SIGSEGV 12) SIGUSR2 13) SIGPIPE 14) SIGALRM 15) SIGTERM 18) SIGCONT 19) SIGSTOP 20) SIGTSTP
SigBlk: 0000000000000000 SigIgn: 0000000000000006 SigCgt: 0000000180014a01
Installed vs Enabled vs Active: separating boot configuration from runtime.
How do you determine whether a systemd service is active, enabled, or failed?
System administration requires separating persistent boot intent from dynamic runtime state. An installed unit exists on disk but does nothing. An enabled unit has a symlink in /etc/systemd/system/*.wants/ ensuring PID 1 launches it when the boot target is reached. An active unit is currently running inside its own dedicated cgroup slice. A service can be enabled but failed (crashed on boot), or active but disabled (running now, but will not restart after reboot).
Model scope. systemd normally places a service in a dedicated cgroup. Stop behavior depends on KillMode, configured signals, timeouts, process cooperation, and whether tasks escaped the unit cgroup; inspect the resulting unit and cgroup state rather than assuming every descendant vanished.
enabled active
● nginx.service - A high performance web server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; preset: enabled)
Active: active (running) since Wed 2026-09-18 19:40:00 UTC
Main PID: 418 (nginx)
Tasks: 2 (limit: 9514)
Memory: 14.2M (max: 512.0M)
CPU: 120ms
CGroup: /system.slice/nginx.service
├─418 "nginx: master process /usr/sbin/nginx -g daemon on; master_process on;"
└─421 "nginx: worker process"Unit file anatomy: declarative process supervision and security sandboxing.
What directives configure a systemd service, and how do you enforce least-privilege runtime?
Stage 01: The [Unit] section defines metadata and ordering. After= specifies sequence; Wants= specifies a non-blocking dependency.
[Unit] Description=High Performance Web Server After=network-online.target remote-fs.target Wants=network-online.target
The [Unit] section defines metadata and ordering. After= specifies sequence; Wants= specifies a non-blocking dependency.
[Service] Type=forking PIDFile=/run/nginx.pid ExecStart=/usr/sbin/nginx ExecReload=/bin/kill -s HUP $MAINPID Restart=on-failure RestartSec=5s
Type=forking expects the parent to fork and exit. Restart=on-failure automatically restarts crashes but ignores clean exits.
ProtectSystem=strict ProtectHome=yes PrivateTmp=yes NoNewPrivileges=yes CapabilityBoundingSet=CAP_NET_BIND_SERVICE
systemd built-in sandboxing: mounts / read-only except /srv, hides /home, gives isolated /tmp, and strips all root capabilities except binding port 443.
# systemd unit cache reloaded in memory
ALWAYS run systemctl daemon-reload after editing unit files. systemd caches unit graphs in memory; edits are invisible until reloaded.
A systemd unit file is a declarative service contract. The [Unit] section configures dependency ordering. The [Service] section configures execution lifecycle, process types (simple, forking, notify), restart policies, and kernel security hardening. Directives like ProtectSystem=strict, PrivateTmp=yes, and CapabilityBoundingSet allow systemd to build tight kernel namespaces and seccomp filters around the service without modifying application source code.
Model scope. systemd-analyze security audits any unit file against modern Linux kernel security directives and outputs an exposure score from 0 (safest) to 10 (unsafe).
ActiveState=active SubState=running MainPID=418 MemoryCurrent=14888960
→ Overall exposure level for nginx.service: 2.1 OK (Protected by systemd sandboxing)
Context propagation: how environment strings are passed via the process stack.
How do processes receive environment variables, and why do background services lack shell profiles?
Stage 01: PATH is an environment variable in your interactive shell process. When you run a command without an absolute path, the shell searches these directories in order.
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
PATH is an environment variable in your interactive shell process. When you run a command without an absolute path, the shell searches these directories in order.
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin STAGE=production
The kernel stores environment variables as a null-delimited array on the process stack. /proc/<pid>/environ provides unalterable evidence of what the process was given at launch.
STAGE="production" WORKER_PROCESSES=auto
systemd daemons do NOT source ~/.bashrc or /etc/profile! They read variables exclusively from Environment= or EnvironmentFile= directives.
When execve() creates a process, it places an array of key=value string pointers onto the new stack frame above argv. In C, these are accessed through the global pointer environ. Children inherit a copy of their parent environment variables during fork(). A universal operational gotcha is running a command manually in a shell where it succeeds, but watching it fail under systemd or cron because shell profile variables (PATH, AWS_PROFILE) were never loaded into the daemon environment.
Model scope. Never pass passwords or API tokens as command-line arguments (ps aux leaks them); pass them via environment variables or secret files, but restrict /proc/<pid>/environ access.
STAGE=production
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful
Systematic service triage: from unit failure to root-cause isolation.
A service fails to start or crashes in a restart loop. Where do you look first?
When a service fails to start, never restart blindly in a loop. Follow an evidence-based triage ladder: 1. Inspect unit status for the exact exit code (e.g. exit 1 = app error, exit 137 = OOM killer, exit 203 = binary not found); 2. Read journalctl -u <service> -n 50 for the exact stderr error message; 3. Distinguish syntax errors from port collisions, permission denials, and memory exhaustion.
Model scope. Status code 203/EXEC indicates systemd could not execute the binary (missing file, wrong architecture, or execute permission missing). Status 137 indicates the OOM killer or kill -9.
● nginx.service - A high performance web server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Wed 2026-09-18 19:40:12 UTC
Process: 9812 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)nginx[9812]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use) nginx[9812]: nginx: [emerg] still could not bind()