knaxuslearn · build · ship

Roadmaps / Java / The milestone project

Read first

The milestone project

Scenario. A shop's checkout calls orders-service to place an order. The service writes the order to Postgres and publishes an OrderPlaced event to Kafka for billing and shipping. Average load is 300 requests a second, peak 900. The container has 2 CPUs and 1 GB. Targets: p99 under 300 ms at 300 requests a second; heap at most 512 MB; GC pause p99 at or under 30 ms; ready to serve within 20 s of start; no lost and no duplicated event after a broker restart; zero failed requests during a rolling deploy.

PhaseCheckpointDone when
1Measured serviceA JMH benchmark, a JFR recording and a GC log exist for the service at 300 requests a second, with your predictions beside the results. A seeded leak (a static map that grows per request) is found from two heap dumps and fixed. A seeded pinned virtual thread (a blocking native call) is found from a thread dump and removed. p99 and GC pause meet the targets.
2Data and eventsThe modules are separated and a Spring Modulith test fails on a cross-module call. A schema change ships in two steps (expand, then contract) with no downtime in a test. Events go through an outbox table. With the broker killed 20 times during 10,000 orders, no order is lost and a duplicate delivery is ignored by an idempotent consumer. A slow database (500 ms added) trips a timeout, not a pile-up.
3Operated serviceThe service runs in kind with probes and resource limits, and a rolling deploy at 300 requests a second drops zero requests. Metrics and traces show every request. A service level objective (99.5% of orders under 300 ms) has an alert. A k6 run at 900 requests a second finds the saturation point, and the service sheds load past it with a 429 (too many requests) instead of failing.
4Changed and shippedThe service moves from the long-term support JDK to the latest feature JDK with a parallel run and a written compare of latency and memory. Architecture rules run as tests. The build produces a software bill of materials (SBOM), and a seeded vulnerable dependency is caught in CI. The runbook covers a stall, a leak and a broker outage. An upgrade policy memo says which JDK you run and when you move.
WWork version (Staff and Principal)On a real service: measure a baseline (p99, GC pause and heap over a week), make one change that removes a stall, leak or failure mode, and measure again. Pass: a target agreed in writing beforehand is met, or the worst stall falls at least 5 times on the same load. The ADR is signed by a named stakeholder, scored by someone other than you, and a recheck is booked three months later. Timebox: six weeks alongside Phases 3 and 4.

Evidence to hand in. A repository whose README opens with the problem and the numbers above; one ADR (architecture decision record) per decision, each naming a rejected option; raw measurement output and a test report; a one-page design doc; a diagram of the order path. Skeletons: Checkpoint 1 is one table (run, load, p99, GC pause p99, heap) plus the two seeded-fault runs. Checkpoint 2 is the 20-kill table (kill, orders sent, orders stored, events delivered, duplicates ignored). Checkpoint 4 is a before and after table for the JDK move.

LevelWhat you build and hand in
FresherCore scope: one service, constant limits, the four checkpoints, ADRs 1 to 4.
SeniorAdds the load test in CI, a seeded-fault suite and ADRs 5 to 7.
StaffAdds written decisions with numbers: service level objective, alert thresholds, cost of the instances and an upgrade policy.
PrincipalAdds a platform standard for many services, a review checklist other teams reuse, and a measured outcome from the work version.

Do this at work. A solo lab is not promotion evidence. Take a real Java service that has stalled, leaked or failed under load and run Checkpoint W on it. Start with a baseline over a week. Make one change. Measure again. A promotion committee accepts the real decision and who made it, the named stakeholder who relied on it, before and after numbers, and a result checked again three months later.

Solo version. With no real service, use a named open-source Spring application, read its code and write a one-page review of how it bounds threads, memory and retries. Ask a peer or mentor to check your measurements. Replay a recorded request file instead of live traffic.

Laptop scale. Everything runs on a laptop in containers. Peak 900 requests a second on a laptop is a scaled number: report what your laptop reaches and extrapolate in the memo. A cloud run is optional and its cost goes in the memo.

Honest sizing. A service that works takes an experienced engineer a few days. The checkpoints add seeded faults, measurement and a rolling-deploy test, which is where the hours go. Phase 2 is the heaviest for a fresher (modules, schema change and events together), so use the extra fresher week there.

Worked reference: what a "3 Proven" hand-in for Checkpoint 1 looks like

The numbers are illustrative. The shape (a blocking call that pins a carrier thread caps throughput far below the offered load) is what you should expect. Hand in your own measurements.

300 requests a second, 40 ms database call200 platform threadsVirtual threads, blocking call pinsVirtual threads, pin removed
Threads or carriers in use at peak200 of 2002 of 2 carriers stuck2 carriers, 14 mounted
p99260 ms4,100 ms110 ms
Heap after 10 minutes410 MB405 MB395 MB
GC pause p9918 ms17 ms17 ms

ADR-2 excerpt: Use virtual threads with a limit of 64 database connections. Little's law (requests in flight equal arrival rate times time in system) gives 300 x 0.04 = 12 database calls in flight at average load and 36 at the 900 peak. A pool of 64 leaves headroom of 1.8 times at peak. Platform threads would also work at this load, but the plan expects the service to grow and virtual threads keep the code simple. Rejected: an unbounded virtual-thread fan-out, because the database, not the JVM, is the limit. Revisit if the database call grows past 100 ms.