knaxuslearn · build · ship

Roadmaps / Go / Phase 2

Phase 2 of 4 · Study weeks 5-8

A durable service

This phase makes the second checkpoint possible: a service that stores events, survives a kill and shuts down cleanly.

Topics every week

  1. Week 58 h
    • net/http servers, handlers and timeouts
    • log/slog
    • JSON with encoding/json and v2 differences
    • gRPC basics

    Hand inAn HTTP intake with full timeouts and structured logs

  2. Week 68 h
    • pgx and pooling
    • Transactions
    • Claiming work with leases and FOR UPDATE SKIP LOCKED
    • Idempotency keys

    Hand inA store that claims events with a lease, tested with two workers

  3. Week 78 h
    • Retries, backoff and jitter
    • Per-endpoint rate limits
    • Signing with HMAC and replay protection
    • Dead letters

    Hand inA delivery loop with a retry budget and a signed payload

  4. Week 88 h
    • Graceful shutdown
    • Crash tests with SIGKILL
    • At-least-once and what it means for receivers
    • Failure injection

    Hand inCheckpoint 2, with the 20-kill table

Practice projects

  • Practice 1, the timeouts (Week 5): send a slow request body to the intake. Done when: the server closes the connection within the read timeout and a test proves it.
  • Practice 2, the lease (Week 6): run two workers against one table. Done when: no event is claimed twice at the same time, and an expired lease lets the other worker take over.
  • Practice 3, the retry storm (Week 7): make an endpoint fail for 60 s. Done when: retries use backoff with jitter, the total attempts stay inside the budget and the endpoint is not hit harder after it recovers.
  • Optional, a dead-letter table: park events that fail 10 times. Done when: a query lists them with the last error and a replay endpoint re-queues one.

GitHub repos to study

Existence and last commit were checked on 22 Nov 2026 by shallow clone. Every repository below had a commit in the last 12 months.

RepoWhat to readWeek
jackc/pgxThe pool, and how it handles a cancelled context6
grpc/grpc-goInterceptors, deadlines and the server shutdown path5
golang/gonet/http server, Shutdown and the timeouts5

Milestone checkpoint

Build Checkpoint 2. Write the 20-kill script and the tamper test first and watch them fail. Add the store with leases, the retry loop, the signature and the shutdown path. Hand in the 20-kill table, the shutdown test, ADR-3 (leases against a message broker) and ADR-4 (retry budget and idempotency key).

Check yourself

Answer each question in your head first, then open it to check.

  1. Why does at-least-once delivery need an idempotency key?
    A crash after delivery and before the record is updated means a retry. The receiver uses the key to ignore the second copy.
  2. What does SKIP LOCKED give you, and what does it not?
    It lets workers skip rows another transaction has locked, so they do not wait. It does not stop a worker that holds a row from crashing, so you still need a lease with an expiry.
  3. Why add jitter to retries?
    Without jitter, all clients retry at the same instants, which makes a recovering service fail again.
  4. Which net/http timeouts protect a server from slow clients?
    Read header, read, write and idle timeouts. The default server has none.
  5. In what order should the service shut down on SIGTERM?
    Stop accepting, finish in-flight work within a deadline, release leases, close the database and flush logs, then exit.