Phase 3 of 4 · Study weeks 9-12
Operating the service
This phase makes the third checkpoint possible: a service you can see into, load test and break on purpose.
Topics every week
- Week 98 h
- OpenTelemetry traces and context propagation
- Prometheus metrics and labels
- Cardinality
Hand inA trace from enqueue to delivery and a dashboard
- Week 108 h
- Load testing
- The saturation point
- Coordinated omission
- Service level objectives and burn-rate alerts
Hand inA load report with the saturation point and the delivery-delay alert
- Week 118 h
- Failure drills: a hanging endpoint, a slow database, a full disk
- Backpressure and load shedding
- Runbooks
Hand inDrill results and a first runbook
- Week 128 h
- Deterministic tests with
testing/synctest - Property tests
- Benchmarks with
testing.B - Profiling in production safely
Hand inCheckpoint 3, with predicted and measured numbers side by side
- Deterministic tests with
Practice projects
- Practice 1, the dashboard (Week 9): show enqueue rate, delivery delay and in-flight per endpoint. Done when: an injected slow endpoint is visible in under 1 minute, and the label set stays under 1,000 series.
- Practice 2, the burn-rate alert (Week 10): inject 5 s of delivery delay. Done when: the alert fires within its stated window and stays silent on a normal run.
- Practice 3, the hanging endpoint (Week 11): hang one of 400 endpoints for 30 s. Done when: the other 399 keep their delay and the hung endpoint's queue sheds at its bound.
- Optional, a production profile: capture a CPU profile from a running service. Done when: the capture adds under 5% overhead and exposes no payload data.
GitHub repos to study
Existence and last commit were checked on 22 Nov 2026 by shallow clone. Every repository below had a commit in the last 12 months.
| Repo | What to read | Week |
|---|---|---|
| open-telemetry/opentelemetry-go | The SDK, spans and context propagation | 9 |
| prometheus/prometheus | The client model: counters, histograms and labels | 9 |
| golang/go | testing/synctest and how it fakes the clock | 12 |
Milestone checkpoint
Run Checkpoint 3. Write your prediction for the saturation point first. Add the traces and metrics, run the load test and the hanging-endpoint drill, and rewrite the time-based tests with synctest. Hand in the dashboard, the alert test, the load report, ADR-5 (what the service sheds and when) and the first draft of the runbook.
Check yourself
Answer each question in your head first, then open it to check.
Why is a high-cardinality label a problem in Prometheus?
Each label value creates a series, so an unbounded label such as an event id grows memory and query cost without limit.What does coordinated omission do to a load test?
A generator that waits for a slow reply stops sending, so the slow period is under-counted. Use a fixed arrival rate.Why use a burn-rate alert for the delivery objective?
It tells you how fast you are spending the error budget, so a brief spike and a slow burn page at the right urgency.What does
synctestchange about a test that uses timers?It runs the test in a bubble with a fake clock, so timers fire when all goroutines are blocked and the test takes no real time.When is shedding load better than queueing?
When a queue only adds latency for everyone and still fills. A bounded queue with a clear rejection keeps accepted work inside the objective.