knaxuslearn · build · ship

Roadmaps / Fundamentals / The milestone project

Read first

The milestone project

Scenario. A small team runs a message queue for internal events. Producers send messages (up to 1 KB) over TCP, and consumers read them in order. echoq is a single-process server with a framed binary protocol, an in-memory queue per topic and an append-only file for durability. Load is 20,000 messages a second on average with 1,000 to 10,000 open connections, some of them slow. The node has 2 CPUs and 1 GB. Targets: p99 publish acknowledgement at or under 10 ms at 20,000 messages a second; a slow consumer never blocks other consumers; no message lost on a clean shutdown and none acknowledged and then lost on kill -9; RSS at most 200 MB at 10,000 connections; the race detector or sanitizers report nothing; a packet capture explains every connection that closed abnormally.

PhaseCheckpointDone when
1Measured processA single-connection server stores and returns messages. A table shows allocations per message, bytes per message and the cost of a copy against a reference, with your predictions beside them. A memory profile shows where the heap goes. A seeded leak (a map that is never cleaned after a topic closes) is found from the profile and fixed, with before and after numbers.
2Talks to the OSThe server appends to a file with an explicit flush policy. A system call trace shows the calls per message. A table compares one write per message, buffered writes and batched writes with predicted and measured throughput. In 20 runs of kill -9 during writes, every acknowledged message is present after restart. A seeded fault (a write that returns fewer bytes than asked) is handled and tested.
3ConcurrentThe server handles 1,000 connections with threads or a pool and a shared queue. The race detector or sanitizers pass with a stress test of 10 million operations. A seeded race (an unlocked counter) is caught by the tool and fixed. A seeded deadlock (two locks taken in opposite order) is found from a thread dump and fixed with a lock-order rule you write down.
4NetworkedThe server handles 10,000 connections within the memory limit. A slow consumer, a half-open connection and a client that sends a partial frame each have a test and the server stays responsive. A packet capture of a connection reset is annotated line by line. A load test reports p50, p99 and spread at 20,000 messages a second over 5 minutes.
WWork version (Staff and Principal)On a real incident or service: write a diagnosis playbook (what to check, in order, with the command for each) and use it on three real or replayed problems. Pass: the time to root cause on the three problems falls by at least half against your earlier habit, measured, or a target agreed in writing beforehand is met. The playbook is reviewed by a named stakeholder, scored by someone other than you, and a recheck is booked three months later. Timebox: six weeks alongside Phases 3 and 4.

Evidence to hand in. A repository whose README opens with the problem and the numbers above; one ADR (architecture decision record) per decision, each naming a rejected option; raw tool output (profiles, traces, captures) and a test report; a one-page design doc; a diagram of the data path from socket to disk. Skeletons: Checkpoint 1 is the table below with your numbers. Checkpoint 2 is the 20-kill table (kill, acknowledged, present after restart). Checkpoint 4 is the annotated capture and the load report with spread.

LevelWhat you build and hand in
FresherCore scope: one language, threads or a pool, 1,000 connections, the four checkpoints, ADRs 1 to 4.
SeniorAdds 10,000 connections, a port of the framing and queue code to a second language with a comparison table, a benchmark in CI and ADRs 5 to 7.
StaffAdds written decisions with numbers: a concurrency policy (lock order, sharing rules), a diagnosis playbook and a cost model.
PrincipalAdds a decision on a shared diagnosis method for several teams, with a measured outcome from the work version and a checklist other teams reuse.

Do this at work. A solo lab is not promotion evidence. Take a real slow or stuck service, or a past incident, and run Checkpoint W on it. Start with a baseline over a month: the time your team takes to find the root cause and how many incidents end with "unknown". Make one change. Measure again. A promotion committee accepts the real decision and who made it, the named stakeholder who relied on it, before and after numbers, and a result checked again three months later.

Solo version. With no real service, take a named open-source server (for example a small Redis-like or HTTP server), reproduce a reported performance or hang issue and write the diagnosis. Ask a peer or mentor to check your measurements. Generate clients with a seeded random generator.

Laptop scale. 10,000 connections run on a laptop if you raise the file descriptor limit. 20,000 messages a second to a laptop SSD is a scaled number: report what your machine reaches, state its disk model and extrapolate in the memo. Nothing needs cloud spend.

Honest sizing. A working queue server takes an experienced engineer a few days. The checkpoints add profiling, crash tests, a race detector, a deadlock hunt and packet analysis, which is where the hours go. Phase 3 is the heaviest for a fresher, so use the extra fresher week there.

Worked reference: what a "3 Proven" hand-in for Checkpoint 2 looks like

The numbers are illustrative. The shape (one write per message is slow, batching raises throughput and raises the loss window) is what you should expect. Hand in your own measurements.

100-byte messages, one producer, SSDOne write per messageBuffered, flush every 10 msBatched, one fsync per 64 messages
Throughput (messages/s)38,000410,00095,000
System calls per message1.00.010.03
Messages lost on kill -9 (acknowledged)0up to 4,0000
Messages lost on power cutup to the OS bufferup to 4,000 plus the bufferup to 63

ADR-3 excerpt: Acknowledge a message after the batched fsync. The buffered policy is fastest but acknowledges messages that are still in memory, so a kill -9 loses them. One write per message survives kill -9 but not a power cut, because the data sits in the operating system buffer. The batched fsync reaches 95,000 messages a second, four times the target, and loses nothing that was acknowledged. Rejected: buffered flush, because it breaks the no-acknowledged-loss rule. Rejected: fsync per message, because it capped throughput at 4,000 messages a second on the test disk. Revisit if the disk is replaced or if the loss window of 63 unacknowledged messages is not acceptable.