knaxuslearn · build · ship

Roadmaps / Linux / The milestone project

Read first

The milestone project

Scenario. orders-api is an HTTP service that takes orders from a shop front end at 200 requests a second on average and 600 at the evening peak. It writes each order to a SQLite file (write-ahead log mode, synchronous=FULL) and answers GET /orders/{id} and GET /healthz. The business wants 99.9% of requests under 150 ms over a month (at 200 requests a second that is about 17,280 slow or failed requests a day out of 17.28 million) and no lost orders after a crash. The failures that matter are a memory leak, a runaway CPU loop, a full disk, slow storage, port exhaustion and a bad DNS answer. Use the starter service below or write your own, with two fault switches set by environment variable: FAULT=leak (retain 8 MiB each second) and FAULT=spin (busy processes). The other faults come from the system, not the code (fault table below).

PhaseCheckpointDone when
1The service and its sandboxA script builds the VM and the service from a clean image; kill -9 on the main process is followed by a restart within 5 seconds in 20 of 20 trials; systemd-analyze security orders-api falls from your recorded baseline to an overall exposure of 2.5 or lower; with the /var/lib/orders-api disk filled by fallocate, /healthz still returns 200 and writes return 503, and the service does not crash
2Limits and resource useA 5-minute run at the peak rate (600 requests a second, 90% reads and 10% writes) shows at least 99.9% of requests under 150 ms (if synchronous=FULL limits writes on your disk, use the highest rate you reach and say so); with MemoryMax=512M, the leak is killed inside the service's cgroup (the kernel log names orders-api, the journal shows the unit failed with result oom-kill and NRestarts rises), host MemAvailable falls by at most the limit (512 MiB, 12.5% of 4 GiB) and is back within 2% of its starting value a minute after the kill, and the service is healthy again within 5 seconds; with the CPU spin on, cpu.stat shows throttling and p99 and p99.9 under load are recorded before and after; a USE table (utilisation, saturation, errors) is filled in for CPU, memory, disk and network
3Network and securityOnly ports 22 and 8080 accept connections from outside (checked with ss and a connection test from the host); the default nftables policy drops; three network faults are reproduced and each is diagnosed from captured evidence; the unit runs with an empty capability set and a syscall filter, and a seeded forbidden call is blocked and logged
4Incident game and operationsThe faults in the fault table are switched on one at a time in an order you did not choose (a friend or a seeded random script picks it); each (four for a fresher) is diagnosed from evidence within 20 minutes and written up as cause, evidence, fix and prevention; a restore from backup meets a recovery time you stated; one package upgrade is rolled back; the runbook lets a stranger repeat the diagnosis of any one fault

Build the VM by script. Either boot the Debian 13 generic cloud image under QEMU (or Multipass on another distribution) with a cloud-init user-data file, or install from the netinst ISO with a preseed file. Put the machine settings in that file and everything inside the VM in provision.sh, which creates the orders-api user, installs the unit and starts the service. Done when provision.sh runs twice with no error and the same end state. Image and tool availability not checked.

Starter service (optional: write your own in under 200 lines if you prefer). Python 3, standard library only, 44 lines; the endpoints and both fault switches were run on 22 Nov 2026, the load rate and the DNS fault were not.

import os, socket, sqlite3, threading, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

db = sqlite3.connect(os.environ.get("DB", "/var/lib/orders-api/orders.db"),
                     check_same_thread=False, isolation_level=None)
db.execute("PRAGMA journal_mode=WAL")
db.execute("PRAGMA synchronous=FULL")
db.execute("CREATE TABLE IF NOT EXISTS orders(id INTEGER PRIMARY KEY, body TEXT)")
lock, hog = threading.Lock(), []

def leak():                      # FAULT=leak: keep 8 MiB more each second
    while True:
        hog.append(b"x" * (8 << 20)); time.sleep(1)

def spin():                      # FAULT=spin: busy child processes (threads would share the GIL)
    for _ in range(int(os.environ.get("SPIN_PROCS", "2"))):
        if os.fork() == 0:
            while True: pass

if os.environ.get("FAULT") == "leak": threading.Thread(target=leak, daemon=True).start()
if os.environ.get("FAULT") == "spin": spin()

class H(BaseHTTPRequestHandler):
    def reply(self, code, body=b""):
        self.send_response(code); self.send_header("Content-Length", str(len(body)))
        self.end_headers(); self.wfile.write(body)
    def do_GET(self):
        if self.path == "/healthz": return self.reply(200, b"ok")
        if self.path.startswith("/orders/"):
            if os.environ.get("RESOLVE"):             # a lookup per request, for the DNS fault
                try: socket.getaddrinfo(os.environ["RESOLVE"], 80)
                except socket.gaierror: return self.reply(502)
            row = db.execute("SELECT body FROM orders WHERE id=?", (self.path[8:],)).fetchone()
            return self.reply(200, row[0].encode()) if row else self.reply(404)
        self.reply(404)
    def do_POST(self):
        body = self.rfile.read(int(self.headers.get("Content-Length", 0))).decode()
        try:
            with lock: cur = db.execute("INSERT INTO orders(body) VALUES (?)", (body,))
        except sqlite3.OperationalError: return self.reply(503)   # disk full or locked
        self.reply(201, str(cur.lastrowid).encode())
    def log_message(self, *args): pass

ThreadingHTTPServer(("0.0.0.0", 8080), H).serve_forever()

Fault table. Checkpoint 4 uses six faults; fresher scope is the first four.

#FaultHow to injectWhat you should see
1Memory leakFAULT=leakCgroup OOM kill at MemoryMax, then a restart
2Runaway CPUFAULT=spin (two busy processes, more than a 150% quota)Throttling in cpu.stat, p99 up
3Disk fullfallocate a file larger than the free space on the /var/lib/orders-api disk (it fails and fills what is free; fill it in steps)Writes 503, health 200
4Port exhaustionA client in a second network namespace (the Week 11 lab) opens short connections to port 8080 and closes first (right after it has read the reply, without waiting for the server to close; whoever closes first holds TIME_WAIT, and the starter service closes first if the client waits). Use a veth path: on loopback tcp_tw_reuse=2 (the default) reuses ports. Narrow ip_local_port_range to 1,000 ports to see it in secondsConnect errors from the client once its ports are used up. In a run on 22 Nov 2026 on loopback with tcp_tw_reuse=0, a client that closed first failed after about 1,500 connections on a 1,000-port range (an order-of-magnitude check, not the exact limit), and a client that waited for the server to close never failed
5Slow storagesystemctl set-property orders-api.service IOWriteBandwidthMax="/dev/vdb 1M" (vdb is the data disk; use your disk name)Write latency up, io.stat and PSI I/O pressure
6Bad DNSRun with RESOLVE=payments.internal and set the only resolver to an unused address such as 192.0.2.1Each GET /orders/{id} waits about 10 seconds (glibc's 5 second timeout, 2 attempts), then returns 502. The image decides who owns /etc/resolv.conf (some cloud images use netplan or systemd-resolved): check ls -l /etc/resolv.conf and set the resolver where that tool reads it, so DHCP does not overwrite it

Evidence to hand in. A repository with the provisioning script and the unit file; a README that opens with the scenario and its numbers; ADRs (each names a rejected option) for sandbox settings, memory limit policy, firewall policy, backup method, and (Senior and above) upgrade and rollback strategy and fleet limits; the capture tables and raw files; a one-page design doc and one diagram of the host (processes, cgroups, network path, disks).

Level variants.

LevelAdds to the core scope
FresherCore: four faults diagnosed (leak, spin, disk full, port exhaustion), restore drill on a small database
SeniorAll six faults, the restore drill with a stated recovery time and recovery point, ADRs 5 and 6
StaffA host-standards memo (8 hours), measured on two of the three classes on your VM (the web service and a CPU-bound batch job), not guessed: one page, default limits for three service classes (for example web, batch and database), a sandbox baseline, an upgrade cadence and alert thresholds. Done when each setting has a number, a cost and the check that would show it failing. Example line: "Web class: MemoryMax at 1.5 times the p99 working set, TasksMax 512; cost: a spike above it kills a request; check: oom_kill count and NRestarts per week"
PrincipalA review checklist (6 hours): 15 yes-or-no checks a team lead can run in 30 minutes on any service (limits set, restart policy, sandbox score, disk and log caps, backup tested). Done when two engineers other than you run it on different services and agree on 12 of 15 answers. A decision memo (8 hours): one platform choice (for example a default limit policy), three options, the numbers from your VM and the extrapolation to a fleet, and the rollback. Done when two readers can state the decision and its cost after one read. Both feed Checkpoint W

Do this at work (Checkpoint W, 24 hours, plus a 90-day recheck). Pick one real service and run Checkpoints 2 and 4 on it in a test or staging environment: record its limits, find what kills it first, and run one seeded fault with the on-call engineer watching. Then apply the review checklist to at least three services in your area. Evidence a promotion committee can accept: the real decision (for example "set MemoryMax and TasksMax on the three services that share a host"), who made it, the named stakeholder who approved it, its scope (services and teams covered), adoption (how many teams now use the checklist) and the measured outcome (restarts, throttling, p99 before and after), re-checked at 90 days with the date recorded. Fleet policy from one VM is an extrapolation; label it so. A lab hand-in alone is not promotion evidence.

Checkpoint W, Done when.

  • The decision is written down with its approver's name and a link or screenshot of the approval (a ticket, review comment or email).
  • A before and after table on the same service and load shows restarts, throttling, p99 and incident count, plus time to restore for any incident, and a line listing what else changed in that period.
  • A negative result is acceptable if it is reported with its cause; an unreported one is not.
  • The 90-day recheck date is recorded now, and the recheck result is added when it is due (outside the 24 hours).

Solo version. Nothing needs a team. A friend or a seeded random script switches the faults, a peer or a forum thread scores the hand-in, and replayed requests from a load generator (wrk, hey or a script of your own) stand in for traffic. For senior and above without a staging host, you can run Checkpoint W on a VM copy of the service with its real unit file and limits, but that is practice, not promotion evidence: nobody else owns the decision or the outcome. It cannot count towards a Principal's core path.

Laptop-scale note. 200 requests a second is easy; 600 is reachable with a load generator on the host. Fleet effects (a hundred hosts, a kernel upgrade wave) become a written extrapolation from your numbers, not a run. An optional cloud run for Week 15 is priced above.

Worked reference: what a "3 Proven" hand-in for Checkpoint 2 looks like

Illustrative numbers, not a recorded run; yours will differ, but the shape of the table and the arithmetic should match. Settings: MemoryMax=512M, CPUQuota=150% on two vCPUs, TasksMax=200, no swap. Throttling uses FAULT=spin with two busy processes; the default quota period is 100 ms, so nr_throttled can rise by at most 600 a minute.

MeasurementPredictionObserved
Idle RSS55 MiB60 MiB
Time for the leak (8 MiB each second) to reach MemoryMax(512 - 55) / 8 = 57.1 s57 s
Host MemAvailable fall during the leak; and one minute after the killAt most 512 MiB (12.5%); within 2% of start497 MiB (12.1%); 0.6%
Time from kill to healthy /healthzUnder 5 s (RestartSec=1)1.9 s
nr_throttled per minute with FAULT=spinClose to 600 (two spinners want 200%, the quota is 150%)587
p99 and p99.9 latency, normal and with the spin40 ms and 90 ms; 250 ms and 400 ms38 ms and 84 ms; 310 ms and 520 ms

ADR-2 excerpt: "Decision: set MemoryMax (512 MiB) and no MemoryHigh. Rejected: adding MemoryHigh below it. With no swap, leaked memory cannot be reclaimed, so MemoryHigh only throttles the leaking process, which keeps the service slow for longer before the same kill. With OOMPolicy=stop (the default) the kill stops the unit, and Restart=on-failure restarts it; OOMPolicy=kill would also kill every process in the cgroup together. Rejected: relying on systemd-oomd alone. It acts only on units that opt in (ManagedOOMMemoryPressure=kill) and only after pressure stays above a limit for a window (check oomd.conf for your version's default), and the leak grows 8 MiB each second; the cgroup limit acts at the limit. Rejected: no limit. The leak then pushes the whole 4 GB host into reclaim and every service slows. Cost: a real spike above 512 MiB kills a healthy request. Check: raise the peak-request test to 500 MiB and confirm no kill."