Packages & Software
Follow a package from signed repository to unpacked files and runtime dynamic linking, and see why installing is not running.
What this chapter builds
A package manager is a trust, dependency, filesystem, and lifecycle system. This chapter follows software from repository metadata to a running, linked service.
By the end, you can
- Explain what repository signatures establish
- Locate the files and scripts a package installed
- Trace dynamic library resolution before main()
- Separate installed, enabled, active, and listening states
Objects to keep straight
ELF interpretermemory mappingsfilesystem pathsservice cgroupsocketApplied task: Find the first broken boundary when a package is installed but its service will not accept connections.
Before looking at the mechanics, what actually is a Package or a Dependency?
What are the foundational primitives of Linux Package Management?
Package (.deb / .rpm)
A package is just a compressed archive of compiled binaries, config files, and a manifest. It also contains "Maintainer Scripts" that run as root before and after extraction.
Dependency
Software relies on other software. If Nginx requires OpenSSL to handle HTTPS, OpenSSL is a dependency. The package manager downloads and installs these automatically.
Repository
A static HTTP server hosting thousands of packages and a cryptographically signed index file (Release/Packages). Your system checks this index to find what is available.
Shared Library (.so)
Instead of every program containing its own copy of common code (like printing text), they use "Shared Objects" (.so files) stored in /usr/lib. Programs link to them at runtime.
Linux software distribution is fundamentally decentralized. You do not download random installers from websites. Instead, your distribution (Ubuntu, RHEL) compiles the software, signs it with a cryptographic key, and hosts it on a Repository. Tools like APT or DNF download these packages, verify the signature, extract the files to standard locations (/usr/bin, /etc), and wire up Systemd services.
Model scope. Adding random PPA repositories or running "curl | sudo bash" bypasses this entire trust chain, granting unknown third parties root access to your machine.
new Debian package, version 2.0. Package: nginx Version: 1.22.1-9 Architecture: amd64 Depends: libc6 (>= 2.34), libpcre3, libssl3, zlib1g Description: small, powerful, scalable web/proxy server
Depends: libc6 Depends: libcrypt1 Depends: libpcre2-8-0 Depends: libssl3
A package manager is a cryptographically verified distribution pipeline.
Where do deb and rpm packages originate, and how does the system prove authenticity before execution?
Configured in /etc/apt/sources.list.d/. Validated via detached GPG signatures on InRelease metadata in /usr/share/keyrings/. Packages.gz supplies per-package SHA256 checksums.
nginx:
Installed: 1.22.1-9
Candidate: 1.22.1-9
Version table:
*** 1.22.1-9 500
500 http://deb.debian.org/debian bookworm/main amd64 PackagesConfigured in /etc/yum.repos.d/*.repo. Validated via repomd.xml signatures; individual RPM package headers embed detached GPG signatures verified before unpacking.
Name : nginx Version : 1.24.0 Release : 1.el9 Architecture : x86_64 Repository : @System From repo : epel
Modern APT stores armored GPG public keys in /usr/share/keyrings/ rather than legacy trusted.gpg. The signed-by directive strictly limits key authority to a single repo.
pub rsa2048 2011-08-19 [SC] [expires: 2027-04-12]
573B FD6B 3D8F BC64 1079 A6AB ABF5 BD82 7BD9 BF62
uid nginx signing key <signing-key@nginx.com>Packages execute maintainer scripts (preinst, postinst, prerm, postrm) as root during installation with zero container sandboxing. Malicious PPAs can compromise the host.
#!/bin/sh
set -e
if [ "$1" = "configure" ]; then
if ! getent passwd nginx >/dev/null; then
adduser --system --quiet --group --home /var/cache/nginx nginx
fi
fiPackage managers do not compile software; they download pre-built binary archives from static HTTP mirrors. Trust is established mathematically: the repository Release file is signed with a private GPG key, which matches a public key in your local keyring. The Release file lists SHA256 hashes for all package index files, which in turn specify hashes for each .deb archive. If a single byte is modified in transit, the package manager halts installation.
Model scope. Maintainer scripts (preinst/postinst) run as unrestricted root. Adding random PPAs or running "curl | bash" allows third parties to execute arbitrary code on your system.
nginx:
Installed: 1.22.1-9
Candidate: 1.22.1-9
Version table:
*** 1.22.1-9 500
500 http://deb.debian.org/debian bookworm/main amd64 Packagesdeb [signed-by=/usr/share/keyrings/nginx-archive-keyring.gpg] http://nginx.org/packages/mainline/debian/ bookworm nginx
A package is an archive that unpacks into the filesystem hierarchy.
Where do binaries, configs, and unit files land when unpacking, and how does the database track ownership?
Stage 01: The package database records every file extracted into the Filesystem Hierarchy Standard (FHS). Binaries go to /usr/sbin, configs to /etc, and unit files to /lib/systemd/system.
/etc/nginx /etc/nginx/nginx.conf /lib/systemd/system/nginx.service /usr/sbin/nginx /usr/share/man/man8/nginx.8.gz
The package database records every file extracted into the Filesystem Hierarchy Standard (FHS). Binaries go to /usr/sbin, configs to /etc, and unit files to /lib/systemd/system.
nginx: /etc/nginx/nginx.conf
Query the database backwards: "Which package owns this path?" Essential for identifying unknown binaries or validating config file origins during incident response.
??5?????? c /etc/nginx/nginx.conf
Verifies installed files against database MD5 checksums. "5" denotes that nginx.conf MD5 digest has changed from the upstream maintainer version due to local editing.
nginx 1240 install ok installed
Inspect low-level package status in /var/lib/dpkg/status. Status indicates whether the package is cleanly installed, half-configured, or scheduled for removal.
Linux packages do not install into isolated application folders like macOS bundles. Instead, the archive extracts files across standard directories: executables into /usr/bin or /usr/sbin, configuration files into /etc, libraries into /usr/lib, and man pages into /usr/share/man. The local database (/var/lib/dpkg or /var/lib/rpm) stores a complete index of all extracted files and MD5 checksums.
Model scope. Files generated dynamically at runtime (access logs in /var/log, databases in /var/lib, cache in /var/cache) are not tracked by dpkg and remain on disk after "apt remove".
nginx: /usr/sbin/nginx
??5?????? c /etc/nginx/nginx.conf
The dynamic linker binds shared ELF libraries at runtime before main().
Why does an installed binary fail with "error while loading shared libraries" even though the file exists?
Default fixture: Most Linux binaries are dynamically linked ELF executables. When executed, the kernel starts the dynamic linker (ld-linux.so), which parses the binary's DT_NEEDED ELF tags to find required shared libraries (.so files). The linker searches in a strict priority order: DT_RPATH/RUNPATH, LD_LIBRARY_PATH, the compiled binary cache in /etc/ld.so.cache, and standard system paths (/lib, /usr/lib). If any library is missing or incompatible, the process aborts with exit code 127 before main() ever executes.
Most Linux binaries are dynamically linked ELF executables. When executed, the kernel starts the dynamic linker (ld-linux.so), which parses the binary's DT_NEEDED ELF tags to find required shared libraries (.so files). The linker searches in a strict priority order: DT_RPATH/RUNPATH, LD_LIBRARY_PATH, the compiled binary cache in /etc/ld.so.cache, and standard system paths (/lib, /usr/lib). If any library is missing or incompatible, the process aborts with exit code 127 before main() ever executes.
Model scope. ldd may execute untrusted binaries under certain linker configurations; for untrusted binaries, inspect ELF headers safely using "readelf -d" or "objdump -p".
linux-vdso.so.1 (0x00007ffe12345000) libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f1234000000) libpcre2-8.so.0 => /lib/x86_64-linux-gnu/libpcre2-8.so.0 (0x00007f1234100000) libssl.so.3 => /lib/x86_64-linux-gnu/libssl.so.3 (0x00007f1234200000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1234300000)
0x0000000000000001 (NEEDED) Shared library: [libcrypt.so.1] 0x0000000000000001 (NEEDED) Shared library: [libpcre2-8.so.0] 0x0000000000000001 (NEEDED) Shared library: [libssl.so.3] 0x0000000000000001 (NEEDED) Shared library: [libc.so.6]
Package installation extracts files; it does not guarantee a running process or socket.
Why does an application fail to respond immediately after running "apt install" or "dnf install"?
Installing software simply writes files into the filesystem hierarchy. Transforming those static files into an active, listening server requires distinct orchestration steps: the package manager must invoke systemd triggers, systemd must parse the unit file, spawn the executable in an isolated cgroup slice, and the application must successfully bind to its configured network sockets. On Debian/Ubuntu, packages attempt to start immediately upon install; on RHEL/Fedora, services default to disabled and stopped.
Model scope. If the default configuration contains an error or the target port is occupied, the package post-install script may fail, leaving the package in a half-configured state ("iF" in dpkg).
● nginx.service - A high performance web server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; preset: enabled)
Active: active (running) since Wed 2026-09-16 10:00:00 UTC; 2h ago
Main PID: 418 (nginx)
Tasks: 2 (limit: 4915)
Memory: 8.2M
CPU: 120ms
CGroup: /system.slice/nginx.service
├─418 "nginx: master process /usr/sbin/nginx -g daemon on; master_process on;"
└─421 "nginx: worker process"enabled active
Package pinning protects production systems from unintended breaking upgrades.
How do you freeze critical database and runtime packages while applying security patches across the host?
Stage 01: Query repository version candidate table. We have 15.4 installed, while 15.6 is available in upstream repositories.
postgresql-15/stable 15.6-0+deb12u1 amd64 [upgradable from: 15.4-1] postgresql-15/now 15.4-1 amd64 [installed,upgradable]
Query repository version candidate table. We have 15.4 installed, while 15.6 is available in upstream repositories.
postgresql-15 set on hold.
Sets the package state flag to "hold" in the package database, instructing apt upgrade to skip this package during automated updates.
The following packages have been kept back: postgresql-15 0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
Simulate system upgrade: apt deliberately skips the held package, preserving database compatibility until the planned maintenance window.
Package: postgresql-15 Pin: version 15.4* Pin-Priority: 1001
For fleet automation, enforce version locks via APT Pin-Priority. A priority > 1000 prevents upgrades even across repository suite transitions.
Unattended upgrades are critical for host security, but automated major or minor version bumps can break database schemas or API contracts. Holding a package (or configuring APT Pin-Priority) instructs the resolver to freeze that specific package at its current version while continuing to patch the rest of the operating system.
Model scope. Held packages do not receive security updates. Track pinned CVEs in your vulnerability management pipeline and schedule explicit upgrade windows.
postgresql-15 kubelet kubeadm kubectl
postgresql-15 hold kubelet hold
A systematic diagnosis tree to isolate runtime application failures.
When an application fails to start or respond after an update, how do you isolate where the failure occurred?
Troubleshooting software requires isolating system layers from the inside out: Package database -> Binary dynamic linking -> Configuration syntax -> Systemd unit manager -> Network socket binding. Blindly restarting services without checking these boundaries destroys ephemeral logs and masks the root cause.
Model scope. AppArmor and SELinux policies can silently block an otherwise valid binary from reading configuration files or binding ports. Check dmesg or audit.log for AVC denials.
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok nginx: configuration file /etc/nginx/nginx.conf test is successful
● nginx.service - A high performance web server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; preset: enabled)
Active: active (running)