Phase 4 of 4 · Study weeks 13-16
Diagnose, operate and hand over
This phase makes you fast and systematic when orders-api breaks, and able to change the host without losing orders. Read Brendan Gregg's Systems Performance (2nd edition, 2020; not checked for a newer edition) chapters on methodology, CPUs and file systems, and the perf and bpftrace manual pages.
Topics every week
- Week 138 h
- Profiling:
perf stat,perf recordand flame graphs,strace -c -T(VM note: hardware counters may be missing, so fall back to thecpu-clockevent)
Hand inA flame graph of the
FAULT=spinbuild with the hot function named (for Python 3.12 or later run the service by hand withpython3 -X perf, outside the unit, becausePrivateTmphides the/tmp/perf-PID.mapfile thatperfneeds to name frames), and astracesummary of a slow request - Profiling:
- Week 148 h
- eBPF tools: bcc and bpftrace one-liners (on Debian 13 the bcc tools are
*-bpfccand bpftrace is 0.23.x), BTF and kernel requirements, overhead
Hand inThree measurements (run-queue latency, block I/O latency, TCP connect latency) with the overhead you measured
- eBPF tools: bcc and bpftrace one-liners (on Debian 13 the bcc tools are
- Week 158 h
- Operating:
aptupgrades and pins,snapshot.debian.orgfor older versions, kernel updates and reboots, backups and restores,sysctldiscipline, journal retention
Hand inAn upgrade with rollback, and a restore from backup with the recovery time measured
- Operating:
- Week 168 h
- The incident game, the runbook and the postmortem
Hand inCheckpoint 4, built in four steps: the fault script; six diagnoses; the runbook; the restore and rollback captures
Practice projects
- Diagnose each of two faults using only
perf,straceand/proc(no eBPF). Done when each write-up states the hot function or syscall and the evidence line. - Write a USE check script that prints utilisation, saturation and errors for CPU, memory, disk and network on one page. Done when it flags each of your seeded faults and writes a one-minute time series of
NRestarts, theoom_killcount and CPU and memory pressure (PSI) to a file. - (Optional for a fresher) Write one
bpftraceprogram of your own for a question aboutorders-api. Done when its output answers the question and its overhead is stated.
GitHub repos to study
| Repo | What to read | Week |
|---|---|---|
| brendangregg/FlameGraph | flamegraph.pl and stackcollapse-perf.pl (stable scripts, little recent activity) | 13 |
| iovisor/bcc | tools/*_example.txt files for output you can compare with yours | 14 |
| brendangregg/perf-tools | Read for ideas only; unmaintained since 2020 | 13 |
| bpftrace/bpftrace | The one-liner tutorial under docs/ | 14 |
Milestone checkpoint
Run the incident game, write the runbook, do the restore and one rollback. Done when the Checkpoint 4 row passes. Evidence: six write-ups (cause, evidence, fix, prevention), the runbook, the restore log with the recovery time, the rollback log, ADR-4 (backup method and what is not covered, for example "a snapshot taken during a write is crash-consistent, not application-consistent") and ADRs 5 and 6 for Senior and above.
Check yourself
Answer each question in your head first, then open it to check.
What is the first question in the USE method for each resource?
Utilisation, then saturation, then errors, for each resource.A host shows 90% CPU. How do you tell a hot loop from a stalled lock?
Sample on-CPU stacks (perf record -g): a hot loop shows one function dominating.perfsamples only time on the CPU, so a lock stall shows little CPU at all; confirm it with off-CPU evidence (strace -f -Tshowing longfutexwaits,/proc/PID/task/*/wchan, or a bccoffcputimerun).Why can
perfshow no hardware events inside a VM?The hypervisor may not expose the performance counters, so only software events such ascpu-clockwork.What makes a backup you have never restored unsafe to trust?
It may be incomplete, corrupt or missing a file; only a restore test proves the recovery and gives a recovery time.After a kernel update, what must you check before the reboot?
That the bootloader entry and initramfs for the new kernel exist and the old kernel is still selectable, so you have a way back.