knaxuslearn · build · ship

Roadmaps / Linux / Phase 2

Phase 2 of 4 · Study weeks 5-8

Memory, CPU, storage and cgroups

This phase lets you state what orders-api may use and prove that a fault stays inside that limit. Read the kernel documents for cgroup v2 (docs.kernel.org/admin-guide/cgroup-v2.html) and pressure stall information (PSI), and Brendan Gregg's USE method page.

Topics every week

  1. Week 58 h
    • Virtual memory: pages, page faults, mmap, RSS, PSS and VSZ, the page cache, swap, the OOM killer

    Hand inA table of RSS, PSS and cache for orders-api idle and loaded, from /proc/PID/smaps_rollup and free, with your prediction first; repeat the leak once with a temporary 1 GiB swap file, then remove it

  2. Week 68 h
    • CPU: the scheduler (EEVDF since kernel 6.6), run queues, load average (counts waiting tasks and tasks in uninterruptible sleep), nice, affinity, PSI

    Hand inA CPU-bound and an I/O-bound load generator, with load average, utilisation and /proc/pressure/cpu side by side

  3. Week 78 h
    • Storage: block layer, ext4 and XFS, page cache writeback, fsync and durability, fio, iostat

    Hand inA durability test: hard power-off the VM (not a guest shutdown) mid-write 20 times and count lost orders with synchronous=FULL against OFF; this tests loss of the guest's page cache, not of the host's disk cache

  4. Week 88 h
    • cgroup v2 and limits: hierarchy, memory.max, memory.high, cpu.max, io.max, pids.max, systemd slices

    Hand inCheckpoint 2, built in four steps: the limits; the leak kill; the throttling capture; the USE table (then the captures)

Practice projects

  1. Reproduce an OOM kill outside a cgroup and read the kernel log. Done when you can name the victim, its oom_score and the memory state at the kill from dmesg.
  2. Measure the cost of fsync. Done when fio gives writes per second with and without fsync on your disk and you explain the ratio.
  3. (Optional for a fresher) Build a cgroup by hand under /sys/fs/cgroup with no systemd and put a process in it. Done when memory.max kills a test allocator.

GitHub repos to study

RepoWhat to readWeek
iovisor/bcctools/memleak.py, tools/biolatency.py and tools/runqlat.py for what each measures5 to 7
systemd/systemdThe resource-control settings in man/systemd.resource-control.xml and src/oom/ for systemd-oomd8

Milestone checkpoint

Apply the limits, run the leak and the spin, capture the cgroup files and fill the USE table. Done when the Checkpoint 2 row passes. Evidence: the prediction-against-observation table (see the worked reference), the raw memory.events, memory.stat and cpu.stat captures, the latency histogram before and after, ADR-2.

Check yourself

Answer each question in your head first, then open it to check.

  1. Why can RSS add up to more than physical memory across processes?
    Shared pages (libraries, shared memory) are counted in each process's RSS; PSS divides shared pages among users.
  2. High load average with low CPU use: name two causes.
    Tasks in uninterruptible sleep (disk or network-filesystem stalls count towards Linux load), and a burst that has ended, because the 1-minute average lags.
  3. What does fsync guarantee that write does not?
    That the data has reached stable storage (or what the device reports as such); write only copies to the page cache.
  4. What is the difference between memory.high and memory.max?
    memory.high throttles and reclaims above the line; memory.max is a hard limit where the kernel kills inside the cgroup if reclaim fails.
  5. How does a cgroup memory kill differ from a host OOM kill?
    A cgroup kill picks a victim inside that cgroup only, so other services are not touched; a host OOM kill picks across the whole machine by score.