knaxuslearn · build · ship

Roadmaps / Rust / The milestone project

Read first

The milestone project

Scenario. A fleet of 20,000 devices sends small measurements to a collector. quillbox appends each record (up to 1 KB) to a log, keeps an index in memory and answers reads by key over TCP. Write load is 50,000 records a second. The node has 2 CPUs and 1 GB, and holds about 2 GB of data on disk. Targets: p99 write acknowledgement at or under 5 ms with group commit every 10 ms; after kill -9 the process recovers a 1 GB log in 5 s or less and no acknowledged write is lost; RSS at most 300 MB; no unsafe outside one module of 150 lines or fewer; that module passes Miri; no data race in the shard handoff under loom.

PhaseCheckpointDone when
1Safe coreThe record codec and the in-memory index pass 40 tests and a property test that round-trips 100,000 random records. A fuzz target runs 10 minutes on the decoder with no panic. Clippy passes with warnings as errors. A seeded bug (an index that survives past the log it points into) is rejected by the compiler or caught by a test, and you write down which. The one unsafe module passes Miri.
2Async serverA Tokio server frames requests, applies a limit of 256 connections and a 1 MB body cap, and times out idle clients at 30 s. A seeded cancellation bug (a write future dropped between two steps) leaves the log consistent because the write is cancellation safe, and a test proves it. On SIGTERM the server drains in-flight writes within 10 s.
3Durable and fastWrites are batched with group commit. In 20 runs of kill -9 during 100,000 writes, every acknowledged write is present after recovery and recovery takes 5 s or less for 1 GB. A loom model of the shard handoff passes. A Criterion benchmark with spread reports p50 and p99 at 50,000 writes a second. A seeded slowdown (a lock held across await) is found from a trace and fixed.
4ShippedCI runs tests, Clippy, Miri and cargo-deny. A seeded vulnerable or unlicensed dependency fails the build. The release builds from a pinned toolchain, ships an SBOM and has a Python or C interface with a test. An adoption memo says where Rust paid off, where it did not and when you would say no.
WWork version (Staff and Principal)On a real project: add one gate (Miri, cargo-deny or a fuzz target) to CI, fix what it finds and record the result. Pass: a target agreed in writing beforehand is met, or the number of findings falls at least 5 times over the same code. The ADR is signed by a named stakeholder, scored by someone other than you, and a recheck is booked three months later. Timebox: six weeks alongside Phases 3 and 4.

Evidence to hand in. A repository whose README opens with the problem and the numbers above; one ADR (architecture decision record) per decision, each naming a rejected option; raw measurement output and a test report; a one-page design doc; a diagram of the write path. Skeletons: Checkpoint 1 is the storage comparison below with your numbers and the Miri output. Checkpoint 3 is the 20-kill table (kill, acknowledged, present after recovery, recovery time) and the benchmark report. Checkpoint 4 is the CI output with the failing and passing dependency check.

LevelWhat you build and hand in
FresherCore scope: one shard, a thread pool, no unsafe of your own, the four checkpoints, ADRs 1 to 4.
SeniorAdds the unsafe module checked by Miri, the loom model, a benchmark in CI and ADRs 5 to 7.
StaffAdds written decisions with numbers: an unsafe policy, an adoption memo, a toolchain upgrade policy and a cost model.
PrincipalAdds a decision on whether a team adopts Rust, with a measured outcome from the work version and a checklist other teams reuse.

Do this at work. A solo lab is not promotion evidence. Take a real Rust or C or C++ component and run Checkpoint W on it. Start with a baseline over a month: findings from the gate you add, crashes and review time. Make one change. Measure again. A promotion committee accepts the real decision and who made it, the named stakeholder who relied on it, before and after numbers, and a result checked again three months later.

Solo version. With no real project, use a named open-source Rust crate, run Miri and cargo-deny on it, and write a one-page review of what you found. Ask a peer or mentor to check your measurements. Generate synthetic records with a seeded random generator.

Laptop scale. Everything runs on a laptop. 50,000 writes a second to a laptop SSD is a scaled number: report what your machine reaches, state its disk model and extrapolate in the memo. Nothing needs cloud spend.

Honest sizing. A working append-only store takes an experienced engineer a few days. The checkpoints add a fuzzer, Miri, loom and crash tests, which is where the hours go. Phase 2 is the heaviest for a fresher (async and cancellation together), so use the extra fresher week there.

Worked reference: what a "3 Proven" hand-in for Checkpoint 1 looks like

The numbers are illustrative. The shape (the unsafe version gains a little and costs a lot of review) is what you should expect. Hand in your own measurements.

50,000 writes a second, 1 KB recordsA Vec<u8> per recordOne arena, safe offsetsOne arena, raw pointers (unsafe)
Allocations per write10.0010.001
p99 write (without disk)38 us21 us19 us
Lines of unsafe00120
Miripassespassespasses after two fixes

ADR-2 excerpt: Use one arena with safe offsets, and no unsafe on the write path. The raw-pointer version is about 10% faster at p99, which is 2 us of a 5 ms target. The disk write dominates, so the gain does not move the target. The 120 lines of unsafe needed two fixes under Miri, and each future change would need the same proof. Rejected: a Vec<u8> per record, because 50,000 allocations a second raised the p99 under memory pressure. Rejected: the unsafe arena, until a measurement shows the codec is on the critical path. Revisit if the disk is replaced by memory (for example a cache tier) and the 2 us matters.