Purpose: Map the Linux systems engineering knowledge base into a navigable field manual for deep Linux, kernel, container, performance, security, and eBPF mastery.

Linux Systems Engineering

This compendium treats Linux as a layered production system: hardware and firmware start the machine, the kernel arbitrates privileged resources, user space composes policy and services, and operators keep the whole stack observable, secure, and recoverable. It is not a beginner command cheat sheet. It is a reference for engineers who need to predict how Linux behaves when systems are slow, isolated, overloaded, attacked, upgraded, or extended with eBPF.

Linux is a kernel, not a complete operating environment by itself. A distribution combines the kernel with a boot chain, init system, libc, package manager, user space tools, default security policy, service manager, logging stack, and release process. Production behavior often depends on both layers: a kernel feature may exist, while the distribution chooses a different default, disables a config option, backports a patch, or wraps it with systemd, NetworkManager, firewalld, SELinux, AppArmor, or a container runtime.

Rendering diagram...

Core Path

Stage	Notes	Mastery outcome
Mental model	01 Linux Mental Model User Space Kernel and Hardware, 06 System Calls ABI libc and User Kernel Boundaries	Explain where an operation runs, which ABI it crosses, and why kernel boundaries shape reliability.
Process model	02 Processes Threads Scheduling Signals and Jobs	Debug process lifecycles, signal behavior, job control, CPU scheduling, and daemon failures.
Memory model	03 Memory Virtual Memory Paging Allocators and OOM	Interpret RSS, page cache, reclaim, swap, NUMA, overcommit, and cgroup memory failures.
Storage model	04 Filesystems VFS Block IO Page Cache and Storage	Trace a file operation through VFS, page cache, block IO, journaling, and storage saturation.
Network model	05 Linux Networking TCP IP Routing Firewalling and DNS	Diagnose routes, sockets, conntrack, NAT, nftables, DNS, TLS symptoms, packet loss, and throughput limits.
Service model	07 systemd Boot Init Units Timers Journald and Services	Operate systemd units, dependencies, logs, timers, hardening, resource controls, and boot failures.
Security model	08 Permissions Users Groups Capabilities and LSMs, 12 Linux Security Hardening Secrets and Incident Response	Reason about identity, capabilities, LSMs, seccomp, secrets, patching, and incident containment.
Isolation model	09 cgroups Namespaces Containers and Runtime Isolation	Explain containers as Linux process isolation using namespaces, cgroups, mounts, runtime specs, and host kernel policy.
Observability model	10 Observability Logs Metrics Tracing and Debugging, 11 Performance Engineering perf Flamegraphs and Capacity	Collect useful evidence without distorting the workload, then connect symptoms to bottlenecks.
Kernel model	13 Kernel Architecture Modules Drivers and Device Model	Understand kernel subsystems, modules, drivers, locking, RCU, panics, taint, and build boundaries.
eBPF model	14 eBPF Fundamentals Verifier Maps Programs and Helpers, 15 eBPF Networking XDP TC Cilium and Service Dataplanes, 16 eBPF Observability Uprobes Kprobes Tracepoints and CO-RE	Use eBPF as constrained kernel extension machinery for tracing, networking, and security without treating it as magic.
Operations model	17 Production Operations Troubleshooting and Runbooks, 18 Linux Ecosystem Tools and Learning Projects	Practice repeatable incident response, tool fluency, and learning projects that map to production realities.

Operating Principles

Principle	Practical consequence
The kernel owns enforcement.	A CLI can request behavior, but the kernel decides based on credentials, namespaces, LSM policy, cgroup limits, device state, and current resources.
User space owns policy.	systemd units, package hooks, DNS resolvers, firewalls, PAM stacks, container runtimes, and orchestrators express policy through kernel APIs.
Most incidents are boundary failures.	Slow DNS, stuck mounts, OOM kills, packet drops, service restarts, and permission denials usually involve a mismatch between user space intent and kernel state.
Observability changes systems.	strace, perf, eBPF, tcpdump, debugfs, and verbose logs add overhead, expose sensitive data, or require privileges. Use bounded scope in production.
Containers share the host kernel.	Namespaces hide global resources and cgroups meter resources, but the kernel, many drivers, and many attack surfaces remain shared.
Version details matter.	Kernel config, distribution backports, systemd version, cgroup mode, BTF availability, nftables backend, and container runtime defaults can change behavior.

Local Machines, Production Hosts, and Clusters

Environment	Bias	What to avoid
Local learning machine	Experiment freely with namespaces, custom kernels, bpftrace one-liners, mount labs, and toy cgroups.	Confusing permissive local privileges with production access.
Production Linux host	Prefer read-only inspection, bounded tracing, reversible unit changes, staged firewall changes, and evidence capture before mutation.	Running unbounded tracers, deleting logs, killing processes blindly, flushing conntrack, or changing sysctls without rollback.
Production cluster	Treat the node as part of a scheduler, network dataplane, storage plane, and security policy set.	Debugging a node as if Kubernetes, containerd, CNI, CSI, and cloud metadata are irrelevant.

Fast Diagnostic Triage

Symptom	First boundary to check	Notes
Service will not start	07 systemd Boot Init Units Timers Journald and Services	Unit dependency, environment, permission, mount, cgroup, or executable path failure.
High load with low CPU	02 Processes Threads Scheduling Signals and Jobs, 04 Filesystems VFS Block IO Page Cache and Storage	Often IO wait, uninterruptible sleep, lock contention, or blocked storage path.
OOM killed container	03 Memory Virtual Memory Paging Allocators and OOM, 09 cgroups Namespaces Containers and Runtime Isolation	Compare host memory pressure with memcg limits and memory.events.
Cannot bind port	05 Linux Networking TCP IP Routing Firewalling and DNS, 08 Permissions Users Groups Capabilities and LSMs	Existing socket, namespace mismatch, privilege, SELinux/AppArmor, or ephemeral port exhaustion.
DNS slow or wrong	05 Linux Networking TCP IP Routing Firewalling and DNS, 17 Production Operations Troubleshooting and Runbooks	Resolver order, systemd-resolved split DNS, search domains, packet loss, or firewall.
BPF program rejected	14 eBPF Fundamentals Verifier Maps Programs and Helpers, 16 eBPF Observability Uprobes Kprobes Tracepoints and CO-RE	Verifier proof failure, helper mismatch, unbounded access, missing BTF, or unsupported attach point.

Operational Vocabulary Anchors

These exact phrases are intentionally present for Obsidian search and review. The detailed treatment lives in the linked notes.

Anchor	Operational reading
Kernel space vs user space	01 Linux Mental Model User Space Kernel and Hardware explains privilege, address space separation, and why failures often appear at this boundary.
libc and ABI boundaries	06 System Calls ABI libc and User Kernel Boundaries explains libc wrappers, syscall ABI contracts, errno, and stable user space behavior.
CPU privilege rings	01 Linux Mental Model User Space Kernel and Hardware ties rings to traps, interrupts, exceptions, syscall entry, and device access.
Interrupts and exceptions	01 Linux Mental Model User Space Kernel and Hardware separates asynchronous hardware interrupts from synchronous CPU exceptions.
Distribution vs kernel distinction	01 Linux Mental Model User Space Kernel and Hardware separates upstream kernel mechanisms from distribution defaults and support policy.
Package managers overview	18 Linux Ecosystem Tools and Learning Projects compares package managers as distribution policy and supply chain machinery.
stdin stdout stderr	01 Linux Mental Model User Space Kernel and Hardware treats standard streams as inherited file descriptors, not special shell magic.
Tasks in the Linux kernel	02 Processes Threads Scheduling Signals and Jobs maps processes and threads onto kernel tasks.
SIGKILL vs SIGTERM	02 Processes Threads Scheduling Signals and Jobs distinguishes cooperative termination from uncatchable forced termination.
CFS scheduler	02 Processes Threads Scheduling Signals and Jobs places CFS in context with EEVDF documentation and real-time classes.
jobs and daemons	02 Processes Threads Scheduling Signals and Jobs connects shell job control to long-running service supervision.
Pages and huge pages	03 Memory Virtual Memory Paging Allocators and OOM explains base pages, transparent huge pages, hugetlbfs, and TLB tradeoffs.
brk and heap	03 Memory Virtual Memory Paging Allocators and OOM separates classic heap growth from mmap-backed allocations.
debugging OOM and memory pressure	03 Memory Virtual Memory Paging Allocators and OOM uses dmesg, cgroup events, PSI, smaps, and workload evidence.
symbolic links	04 Filesystems VFS Block IO Page Cache and Storage covers symlink path resolution and security footguns.
Btrfs overview	04 Filesystems VFS Block IO Page Cache and Storage treats Btrfs as copy-on-write storage with snapshots and operational complexity.
LVM overview	04 Filesystems VFS Block IO Page Cache and Storage places LVM between filesystems and block devices.
RAID overview	04 Filesystems VFS Block IO Page Cache and Storage separates redundancy, availability, rebuild risk, and backup.
dm-crypt overview	04 Filesystems VFS Block IO Page Cache and Storage maps encryption at the device-mapper layer.
io_uring overview	04 Filesystems VFS Block IO Page Cache and Storage frames io_uring as an async submission and completion interface, not a universal speed switch.
IOPS vs throughput	04 Filesystems VFS Block IO Page Cache and Storage distinguishes operation rate from sustained bytes per second.
latency and queue depth	04 Filesystems VFS Block IO Page Cache and Storage explains why deeper queues can improve throughput while hurting tail latency.
filesystem troubleshooting	04 Filesystems VFS Block IO Page Cache and Storage starts from mounts, inodes, free space, dirty writeback, and block saturation.
firewalld overview	05 Linux Networking TCP IP Routing Firewalling and DNS treats firewalld as policy management over nftables or iptables backends.
MTU and fragmentation	05 Linux Networking TCP IP Routing Firewalling and DNS maps packet size failures to PMTU, tunnels, ICMP, and drops.
TCP states	05 Linux Networking TCP IP Routing Firewalling and DNS uses ss state output to reason about listeners, handshakes, close, and TIME_WAIT.
ordering vs requirement	07 systemd Boot Init Units Timers Journald and Services separates After= ordering from Requires= dependency.
setuid and setgid	08 Permissions Users Groups Capabilities and LSMs explains legacy privilege transitions and modern capability alternatives.
Landlock overview	08 Permissions Users Groups Capabilities and LSMs describes unprivileged filesystem access restrictions as an LSM feature.
module signing overview	08 Permissions Users Groups Capabilities and LSMs and 13 Kernel Architecture Modules Drivers and Device Model connect signed modules to secure boot and lockdown posture.
cgroup v1 vs cgroup v2	09 cgroups Namespaces Containers and Runtime Isolation explains unified hierarchy behavior and migration tradeoffs.
CPU controller	09 cgroups Namespaces Containers and Runtime Isolation covers CPU weights, quotas, throttling, and scheduler impact.
memory controller	09 cgroups Namespaces Containers and Runtime Isolation covers memcg accounting, limits, reclaim, and OOM.
IO controller	09 cgroups Namespaces Containers and Runtime Isolation covers block IO control where supported by kernel and device stack.
cpuset controller	09 cgroups Namespaces Containers and Runtime Isolation covers CPU and NUMA node placement constraints.
uts namespace	09 cgroups Namespaces Containers and Runtime Isolation explains hostname and domain-name isolation.
container runtime model	09 cgroups Namespaces Containers and Runtime Isolation separates image manager, CRI service, OCI runtime, and kernel primitives.
CRI-O overview	09 cgroups Namespaces Containers and Runtime Isolation compares CRI-O with containerd as a Kubernetes CRI implementation.
overlayfs for containers	09 cgroups Namespaces Containers and Runtime Isolation links image layers to merged writable container root filesystems.
container security boundaries	09 cgroups Namespaces Containers and Runtime Isolation treats boundaries as layered controls, not VM equivalence.
diagnosing high memory	10 Observability Logs Metrics Tracing and Debugging and 17 Production Operations Troubleshooting and Runbooks start with RSS, cache, cgroups, swap, PSI, and OOM logs.
diagnosing slow disk	10 Observability Logs Metrics Tracing and Debugging uses latency, utilization, queue depth, dirty pages, and filesystem evidence.
diagnosing network latency	10 Observability Logs Metrics Tracing and Debugging combines routes, DNS, retransmits, packet loss, tcpdump, and service metrics.
Module loading and unloading	13 Kernel Architecture Modules Drivers and Device Model covers modprobe, insmod, dependencies, reference counts, and production risk.
syscalls table overview	13 Kernel Architecture Modules Drivers and Device Model places syscall dispatch in the architecture boundary without promising a stable internal table layout.
tasklets status and modern alternatives	13 Kernel Architecture Modules Drivers and Device Model notes legacy tasklets and prefers workqueues, threaded IRQs, NAPI, and other maintained mechanisms when appropriate.
building a kernel overview	13 Kernel Architecture Modules Drivers and Device Model treats kernel builds as lab or vendor-governed production work.
What eBPF is and is not	14 eBPF Fundamentals Verifier Maps Programs and Helpers frames eBPF as verified kernel extension, not arbitrary kernel scripting.
BPF instruction set overview	14 eBPF Fundamentals Verifier Maps Programs and Helpers describes registers, instructions, helpers, maps, and JIT context.
kretprobes	14 eBPF Fundamentals Verifier Maps Programs and Helpers and 16 eBPF Observability Uprobes Kprobes Tracepoints and CO-RE cover return probes and fragility.
raw tracepoints	14 eBPF Fundamentals Verifier Maps Programs and Helpers covers lower overhead tracepoint variants and type safety tradeoffs.
Aya overview	14 eBPF Fundamentals Verifier Maps Programs and Helpers places Aya as Rust tooling over the same kernel BPF APIs.
when eBPF is the wrong tool	14 eBPF Fundamentals Verifier Maps Programs and Helpers lists cases where logs, metrics, perf, tcpdump, or code changes are safer.
TC ingress and egress	15 eBPF Networking XDP TC Cilium and Service Dataplanes compares TC hooks with XDP and socket hooks.
socket-level hooks	15 eBPF Networking XDP TC Cilium and Service Dataplanes covers socket filters and socket-related cgroup hooks.
Hubble style flows	15 eBPF Networking XDP TC Cilium and Service Dataplanes connects flow observability to service and policy context.
USDT probes overview	16 eBPF Observability Uprobes Kprobes Tracepoints and CO-RE explains application-defined tracing points.
production overhead management	16 eBPF Observability Uprobes Kprobes Tracepoints and CO-RE covers sampling, filtering, aggregation, and rollout limits.
cardinality control	16 eBPF Observability Uprobes Kprobes Tracepoints and CO-RE limits map keys, labels, and emitted events.
kernel version compatibility	16 eBPF Observability Uprobes Kprobes Tracepoints and CO-RE covers helper availability, BTF, CO-RE, and attach point differences.
Boot failure troubleshooting	17 Production Operations Troubleshooting and Runbooks starts from firmware, bootloader, kernel command line, initramfs, root mount, and PID 1.
high CPU incidents	17 Production Operations Troubleshooting and Runbooks moves from saturation confirmation to process, thread, syscall, and profile evidence.
memory pressure incidents	17 Production Operations Troubleshooting and Runbooks checks PSI, reclaim, cgroups, swap, page cache, and OOM sequence.
OOM incidents	17 Production Operations Troubleshooting and Runbooks separates global OOM from memcg OOM and captures killer logs.
load average investigation	17 Production Operations Troubleshooting and Runbooks separates runnable work from uninterruptible sleep.
zombie processes	17 Production Operations Troubleshooting and Runbooks treats zombies as wait failures owned by parents or service supervisors.
systemd unit failures	17 Production Operations Troubleshooting and Runbooks checks unit state, logs, dependency graph, credentials, cgroups, and sandboxing.
data collection checklist	17 Production Operations Troubleshooting and Runbooks prioritizes volatile evidence before mutation.
escalation checklist	17 Production Operations Troubleshooting and Runbooks defines when to involve application, kernel, network, security, or vendor owners.
post-incident review checklist	17 Production Operations Troubleshooting and Runbooks connects symptoms, timeline, contributing factors, and durable follow-up.

Reference Sources Used For Drift-Prone Facts

Official and primary sources should be preferred for current behavior:

Domain	Primary references
Kernel user visible APIs	https://docs.kernel.org/, https://man7.org/linux/man-pages/
cgroup v2	https://docs.kernel.org/admin-guide/cgroup-v2.html
eBPF verifier and APIs	https://docs.kernel.org/bpf/, https://docs.kernel.org/userspace-api/ebpf/
systemd units and execution	https://www.freedesktop.org/software/systemd/man/
nftables	https://www.netfilter.org/projects/nftables/manpage.html
iproute2 and sockets	https://man7.org/linux/man-pages/man8/ip.8.html, distribution man pages for ss(8)

Related Notes

• 00 Linux Systems Mastery Roadmap
• 17 Production Operations Troubleshooting and Runbooks
• 18 Linux Ecosystem Tools and Learning Projects