Phase 3 of 4 · Study weeks 9-12
Operating the service
This phase makes the third checkpoint possible: a service that runs in Kubernetes, shows what it is doing and sheds load before it falls over.
Topics every week
- Week 98 h
- Containers and the JVM
- Requests and limits
- Probes and graceful shutdown
- Rolling deploys
Hand inA rolling deploy at 300 requests a second with zero failed requests
- Week 108 h
- Micrometer metrics and histograms
- OpenTelemetry traces
- Structured logs
- Service level objectives and burn-rate alerts
Hand inA dashboard and one alert for the 99.5% under 300 ms objective
- Week 118 h
- Load testing with k6
- Finding the saturation point
- Capacity and cost per 1,000 orders
- Coordinated omission
Hand inA load report with the saturation point and a cost estimate
- Week 128 h
- Overload and load shedding
- Failure drills (a pod kill, a slow database, a full disk)
- Runbooks
Hand inCheckpoint 3, with the drill results and the runbook draft
Practice projects
- Practice 1, the clean shutdown (Week 9): send
SIGTERMduring load. Done when: in-flight requests finish, no request fails, and the pod exits within its grace period. - Practice 2, the burn-rate alert (Week 10): inject 300 ms of extra latency. Done when: the alert fires within its stated window and stays silent on a normal run.
- Practice 3, the saturation point (Week 11): ramp k6 from 100 to 1,200 requests a second. Done when: a chart shows where p99 breaks, and a note says what saturated first (CPU, pool or database).
- Optional, a cost model: price the instances for peak and for average. Done when: the memo gives a cost per 1,000 orders and the assumption that moves it most.
GitHub repos to study
Existence and last commit were checked on 22 Nov 2026 by shallow clone. Every repository below had a commit in the last 12 months.
| Repo | What to read | Week |
|---|---|---|
| micrometer-metrics/micrometer | Timers and histograms, and how percentiles are published | 10 |
| open-telemetry/opentelemetry-java | The SDK and the context propagation code | 10 |
| grafana/k6 | Scenarios, thresholds and arrival-rate executors | 11 |
Milestone checkpoint
Run Checkpoint 3. Write your prediction for the saturation point first. Run the rolling deploy, the three drills and the k6 ramp. Hand in the dashboard, the alert test, the load report, ADR-5 (what the service sheds and at what point) and the first draft of the runbook.
Check yourself
Answer each question in your head first, then open it to check.
Why must memory limits and JVM heap settings be set together?
The JVM sizes its heap from the container limit by default, and native memory comes on top. Set the heap to leave room, or the kernel kills the process.What does a readiness probe protect, and what does a liveness probe risk?
Readiness keeps traffic away from a pod that cannot serve. A liveness probe that fails on a slow dependency restarts healthy pods and makes an outage worse.Why does coordinated omission make a load test look better than it is?
A load generator that waits for a slow response also stops sending, so the slow period is under-counted. Use an open model with a fixed arrival rate.What does a burn-rate alert tell you that a fixed threshold does not?
How fast you are spending the error budget, so a short bad spike and a slow steady burn both page at the right urgency.Why shed load instead of queueing?
A queue adds latency for everyone and still fails when it fills. A fast 429 keeps the requests you accept inside the objective.