knaxuslearn · build · ship

Roadmaps / Go / The milestone project

Read first

The milestone project

Scenario. A SaaS product tells its customers about events (an invoice paid, a user added) by calling a URL each customer registers. hookd accepts events over HTTP, stores them, and delivers each one to its endpoint with a signed payload, retries and a limit per endpoint. Load is 2,000 events a second across 400 endpoints. Some endpoints are slow and some are down. Targets: enqueue p99 under 50 ms; delivery starts within 2 s of enqueue at p99; at most 5 deliveries in flight per endpoint; one hanging endpoint does not delay the others; RSS at most 256 MB; the goroutine count is flat over a 1-hour run; no event is lost when the process is killed with SIGKILL.

PhaseCheckpointDone when
1Bounded deliveryA worker pool delivers 100,000 events to 400 fake endpoints, 10% of which hang. A counter in the test shows in-flight deliveries never pass 5 per endpoint. Cancelling the context stops all work within 1 s and the goroutineleak profile reports none. The race detector passes 100 runs. A seeded leak (a goroutine blocked on a channel nobody reads) is caught by the leak profile.
2Durable serviceEvents are stored in Postgres and claimed with leases. With the process killed with SIGKILL 20 times during 10,000 events, none is lost, and a duplicate delivery carries the same idempotency key. net/http has read, write and idle timeouts. On SIGTERM the service drains in-flight work within 10 s. Payloads are signed with an HMAC and a test rejects a tampered body.
3Operated serviceTraces and metrics show an event from enqueue to delivery. A dashboard has an alert for a delivery-delay objective (99% within 2 s). A load test at 2,000 events a second meets every target, and a ramp finds the saturation point. A drill with one endpoint hanging 30 s leaves the other 399 unaffected. Time-dependent tests use testing/synctest and take under 1 s each.
4ShippedCI runs go vet, golangci-lint, the race detector and govulncheck. A seeded vulnerable dependency fails the build. A release build is reproducible, signed and ships an SBOM. The runbook covers a hanging endpoint, a leak and a database outage. An upgrade memo says which Go version you run and when you move.
WWork version (Staff and Principal)On a real service: measure a baseline (goroutine count, p99 and RSS over a week), make one change that removes a leak, stall or lost-work case, and measure again. Pass: a target agreed in writing beforehand is met, or the worst stall falls at least 5 times on the same load. The ADR is signed by a named stakeholder, scored by someone other than you, and a recheck is booked three months later. Timebox: six weeks alongside Phases 3 and 4.

Evidence to hand in. A repository whose README opens with the problem and the numbers above; one ADR (architecture decision record) per decision, each naming a rejected option; raw measurement output and a test report; a one-page design doc; a diagram of the event path. Skeletons: Checkpoint 1 is the worker-pool comparison table below with your numbers and the failing seeded-leak run. Checkpoint 2 is the 20-kill table (kill, events sent, events stored, deliveries, duplicates with the same key). Checkpoint 3 is the dashboard screenshot, the load report and the hanging-endpoint drill.

LevelWhat you build and hand in
FresherCore scope: one process, constant limits, the four checkpoints, ADRs 1 to 4.
SeniorAdds the load test in CI, a fault suite, a gRPC intake and ADRs 5 to 7.
StaffAdds written decisions with numbers: delivery objective, alert thresholds, runbook, cost of the instances and a Go upgrade policy.
PrincipalAdds a service template for many teams, a review checklist other teams reuse, and a measured outcome from the work version.

Do this at work. A solo lab is not promotion evidence. Take a real Go service that has leaked goroutines, stalled or dropped work and run Checkpoint W on it. Start with a baseline over a week. Make one change. Measure again. A promotion committee accepts the real decision and who made it, the named stakeholder who relied on it, before and after numbers, and a result checked again three months later.

Solo version. With no real service, use a named open-source Go service, read how it bounds goroutines and shuts down, and write a one-page review. Ask a peer or mentor to check your measurements. Replay a recorded event file instead of live traffic.

Laptop scale. Everything runs on a laptop against fake endpoints and a Postgres container. 2,000 events a second is a scaled number: report what your laptop reaches and extrapolate in the memo. Nothing needs cloud spend.

Honest sizing. A first worker pool takes an experienced engineer a day. The checkpoint adds per-endpoint limits, cancellation tests, a leak profile and a seeded fault. Phase 2 has the heaviest load for a fresher (leases, signing and shutdown together), so use the extra fresher week there.

Worked reference: what a "3 Proven" hand-in for Checkpoint 1 looks like

The numbers are illustrative. The shape (unbounded goroutines grow with the slow endpoints; a global pool lets one endpoint take every slot) is what you should expect. Hand in your own measurements.

100,000 events, 400 endpoints, 10% hunggo per eventGlobal pool of 64Pool of 64 and 5 per endpoint
Peak goroutines31,0007070
Peak in flight to one endpoint78064 (a hung endpoint fills the pool)5
Healthy events delivered in 60 s72,00024,00090,000
Peak RSS740 MB90 MB90 MB

ADR-2 excerpt: Use a pool of 64 workers and a limit of 5 per endpoint, with a bounded queue of 100 per endpoint. Little's law (requests in flight equal arrival rate times time in system) gives, for a healthy endpoint at 5 events a second and 0.2 s a call, 1 delivery in flight. A hung endpoint with a 10 s timeout at the same arrival rate would hold 50, so a limit of 5 protects the pool. Normal load needs about 20 workers across 400 endpoints, so 64 leaves 3 times headroom. Rejected: one goroutine per event, because memory grows with the slowest endpoints. Rejected: a global limit alone, because one endpoint can take every slot. Revisit if the timeout or the event rate changes.