Phase 2 of 4 · Study weeks 5-8
A durable service
This phase makes the second checkpoint possible: a service that stores events, survives a kill and shuts down cleanly.
Topics every week
- Week 58 h
net/httpservers, handlers and timeoutslog/slog- JSON with
encoding/jsonand v2 differences - gRPC basics
Hand inAn HTTP intake with full timeouts and structured logs
- Week 68 h
pgxand pooling- Transactions
- Claiming work with leases and
FOR UPDATE SKIP LOCKED - Idempotency keys
Hand inA store that claims events with a lease, tested with two workers
- Week 78 h
- Retries, backoff and jitter
- Per-endpoint rate limits
- Signing with HMAC and replay protection
- Dead letters
Hand inA delivery loop with a retry budget and a signed payload
- Week 88 h
- Graceful shutdown
- Crash tests with
SIGKILL - At-least-once and what it means for receivers
- Failure injection
Hand inCheckpoint 2, with the 20-kill table
Practice projects
- Practice 1, the timeouts (Week 5): send a slow request body to the intake. Done when: the server closes the connection within the read timeout and a test proves it.
- Practice 2, the lease (Week 6): run two workers against one table. Done when: no event is claimed twice at the same time, and an expired lease lets the other worker take over.
- Practice 3, the retry storm (Week 7): make an endpoint fail for 60 s. Done when: retries use backoff with jitter, the total attempts stay inside the budget and the endpoint is not hit harder after it recovers.
- Optional, a dead-letter table: park events that fail 10 times. Done when: a query lists them with the last error and a replay endpoint re-queues one.
GitHub repos to study
Existence and last commit were checked on 22 Nov 2026 by shallow clone. Every repository below had a commit in the last 12 months.
| Repo | What to read | Week |
|---|---|---|
| jackc/pgx | The pool, and how it handles a cancelled context | 6 |
| grpc/grpc-go | Interceptors, deadlines and the server shutdown path | 5 |
| golang/go | net/http server, Shutdown and the timeouts | 5 |
Milestone checkpoint
Build Checkpoint 2. Write the 20-kill script and the tamper test first and watch them fail. Add the store with leases, the retry loop, the signature and the shutdown path. Hand in the 20-kill table, the shutdown test, ADR-3 (leases against a message broker) and ADR-4 (retry budget and idempotency key).
Check yourself
Answer each question in your head first, then open it to check.
Why does at-least-once delivery need an idempotency key?
A crash after delivery and before the record is updated means a retry. The receiver uses the key to ignore the second copy.What does
SKIP LOCKEDgive you, and what does it not?It lets workers skip rows another transaction has locked, so they do not wait. It does not stop a worker that holds a row from crashing, so you still need a lease with an expiry.Why add jitter to retries?
Without jitter, all clients retry at the same instants, which makes a recovering service fail again.Which
net/httptimeouts protect a server from slow clients?Read header, read, write and idle timeouts. The default server has none.In what order should the service shut down on
SIGTERM?Stop accepting, finish in-flight work within a deadline, release leases, close the database and flush logs, then exit.