Phase 2 of 4 · Study weeks 5-8
Memory, CPU, storage and cgroups
This phase lets you state what orders-api may use and prove that a fault stays inside that limit. Read the kernel documents for cgroup v2 (docs.kernel.org/admin-guide/cgroup-v2.html) and pressure stall information (PSI), and Brendan Gregg's USE method page.
Topics every week
- Week 58 h
- Virtual memory: pages, page faults,
mmap, RSS, PSS and VSZ, the page cache, swap, the OOM killer
Hand inA table of RSS, PSS and cache for
orders-apiidle and loaded, from/proc/PID/smaps_rollupandfree, with your prediction first; repeat the leak once with a temporary 1 GiB swap file, then remove it - Virtual memory: pages, page faults,
- Week 68 h
- CPU: the scheduler (EEVDF since kernel 6.6), run queues, load average (counts waiting tasks and tasks in uninterruptible sleep),
nice, affinity, PSI
Hand inA CPU-bound and an I/O-bound load generator, with load average, utilisation and
/proc/pressure/cpuside by side - CPU: the scheduler (EEVDF since kernel 6.6), run queues, load average (counts waiting tasks and tasks in uninterruptible sleep),
- Week 78 h
- Storage: block layer, ext4 and XFS, page cache writeback,
fsyncand durability,fio,iostat
Hand inA durability test: hard power-off the VM (not a guest shutdown) mid-write 20 times and count lost orders with
synchronous=FULLagainstOFF; this tests loss of the guest's page cache, not of the host's disk cache - Storage: block layer, ext4 and XFS, page cache writeback,
- Week 88 h
- cgroup v2 and limits: hierarchy,
memory.max,memory.high,cpu.max,io.max,pids.max, systemd slices
Hand inCheckpoint 2, built in four steps: the limits; the leak kill; the throttling capture; the USE table (then the captures)
- cgroup v2 and limits: hierarchy,
Practice projects
- Reproduce an OOM kill outside a cgroup and read the kernel log. Done when you can name the victim, its
oom_scoreand the memory state at the kill fromdmesg. - Measure the cost of
fsync. Done whenfiogives writes per second with and withoutfsyncon your disk and you explain the ratio. - (Optional for a fresher) Build a cgroup by hand under
/sys/fs/cgroupwith no systemd and put a process in it. Done whenmemory.maxkills a test allocator.
GitHub repos to study
| Repo | What to read | Week |
|---|---|---|
| iovisor/bcc | tools/memleak.py, tools/biolatency.py and tools/runqlat.py for what each measures | 5 to 7 |
| systemd/systemd | The resource-control settings in man/systemd.resource-control.xml and src/oom/ for systemd-oomd | 8 |
Milestone checkpoint
Apply the limits, run the leak and the spin, capture the cgroup files and fill the USE table. Done when the Checkpoint 2 row passes. Evidence: the prediction-against-observation table (see the worked reference), the raw memory.events, memory.stat and cpu.stat captures, the latency histogram before and after, ADR-2.
Check yourself
Answer each question in your head first, then open it to check.
Why can RSS add up to more than physical memory across processes?
Shared pages (libraries, shared memory) are counted in each process's RSS; PSS divides shared pages among users.High load average with low CPU use: name two causes.
Tasks in uninterruptible sleep (disk or network-filesystem stalls count towards Linux load), and a burst that has ended, because the 1-minute average lags.What does
fsyncguarantee thatwritedoes not?That the data has reached stable storage (or what the device reports as such);writeonly copies to the page cache.What is the difference between
memory.highandmemory.max?memory.highthrottles and reclaims above the line;memory.maxis a hard limit where the kernel kills inside the cgroup if reclaim fails.How does a cgroup memory kill differ from a host OOM kill?
A cgroup kill picks a victim inside that cgroup only, so other services are not touched; a host OOM kill picks across the whole machine by score.