knaxuslearn · build · ship

Roadmaps / Linux

Foundations · 16 study weeks · 4 phases

Linux Roadmap Beta

16 study weeks on one Debian 13 virtual machine. You learn to run, limit, secure and diagnose a Linux service until you can find why a host is slow or dying from evidence. At 8 hours a week the core path is 128 hours and takes 16 weeks. At 12 hours a week it takes about 11 weeks, and at 16 hours a week about 8 weeks. You build one system across all four phases, orders-api, a small order service on a 2-CPU, 4 GB host, and every phase adds a checkpoint to it. At the end you can say, with numbers, what a process is allowed to use, why it was killed or slowed, and how you would restore the host after a bad upgrade.

This roadmap is in beta and keeps improving over time. If you spot an error, please email your suggestions and corrections to [email protected].

Read first

  1. Phase 1 · Study weeks 1-4

    The machine, the process and the service

    This phase makes orders-api run as a proper service that restarts, logs and is sandboxed, so Phase 2 has something to limit. Read man 5 systemd.unit, man 5 systemd.service, man 5 systemd.exec and the kernel's Documentation/admin-guide as you go.

    • 2 repos
    • Milestone checkpoint
    1. Study week 1Boot: firmware, bootloader, kernel and initramfs, systemd as process 18 h
    2. Study week 2Files and permissions: inodes, links, mounts, /etc and /var, users and groups, mode bits, umask, setuid, ACLs, capabilities8 h
    3. Study week 3systemd units: service types, ordering and dependencies, Restart=, drop-ins, journald, timers8 h
    4. Study week 4Sandbox options (ProtectSystem, NoNewPrivileges, PrivateTmp, CapabilityBoundingSet, StateDirectory)8 h
  2. Phase 2 · Study weeks 5-8

    Memory, CPU, storage and cgroups

    This phase lets you state what orders-api may use and prove that a fault stays inside that limit. Read the kernel documents for cgroup v2 (docs.kernel.org/admin-guide/cgroup-v2.html) and pressure stall information (PSI), and Brendan Gregg's USE method page.

    • 2 repos
    • Milestone checkpoint
    1. Study week 5Virtual memory: pages, page faults, mmap, RSS, PSS and VSZ, the page cache, swap, the OOM killer8 h
    2. Study week 6CPU: the scheduler (EEVDF since kernel 6.6), run queues, load average (counts waiting tasks and tasks in uninterruptible sleep), nice, affinity, PSI8 h
    3. Study week 7Storage: block layer, ext4 and XFS, page cache writeback, fsync and durability, fio, iostat8 h
    4. Study week 8cgroup v2 and limits: hierarchy, memory.max, memory.high, cpu.max, io.max, pids.max, systemd slices8 h
  3. Phase 3 · Study weeks 9-12

    Network and security

    This phase lets you say who can reach orders-api, what it can reach and what it may do, and diagnose a network fault from packets. Read man 8 ip, man 8 ss, the nftables wiki and man 7 capabilities.

    • 3 repos
    • Milestone checkpoint
    1. Study week 9One host's network: interfaces, routes, neighbour tables, ip and ss, the TCP handshake and states, TIMEWAIT, ephemeral ports8 h
    2. Study week 10DNS and TLS: the resolver path (nsswitch, systemd-resolved, glibc timeout and attempts), dig, openssl sclient, certificate expiry8 h
    3. Study week 11Network namespaces, veth, bridges, nftables chains, hooks, default drop and connection tracking, tc netem8 h
    4. Study week 12Security model: capabilities, seccomp filters, AppArmor (on by default in Debian), SSH hardening, audit logs8 h
  4. Phase 4 · Study weeks 13-16

    Diagnose, operate and hand over

    This phase makes you fast and systematic when orders-api breaks, and able to change the host without losing orders. Read Brendan Gregg's Systems Performance (2nd edition, 2020; not checked for a newer edition) chapters on methodology, CPUs and file systems, and the perf and bpftrace manual pages.

    • 4 repos
    • Milestone checkpoint
    1. Study week 13Profiling: perf stat, perf record and flame graphs, strace -c -T (VM note: hardware counters may be missing, so fall back to the cpu-clock event)8 h
    2. Study week 14eBPF tools: bcc and bpftrace one-liners (on Debian 13 the bcc tools are -bpfcc and bpftrace is 0.23.x), BTF and kernel requirements, overhead8 h
    3. Study week 15Operating: apt upgrades and pins, snapshot.debian.org for older versions, kernel updates and reboots, backups and restores, sysctl discipline, journal retention8 h
    4. Study week 16The incident game, the runbook and the postmortem8 h

Also in this roadmap