knaxuslearn · build · ship

Roadmaps / Distributed Systems / Phase 1

Phase 1 of 4 · Study weeks 1-4

Time, failure and messages

This phase makes the first checkpoint possible: nodes that notice failure and handle repeated or late messages safely.

Topics every week

  1. Week 18 h
    • System models: crash, omission and Byzantine faults
    • The network is asynchronous
    • What a guarantee is
    • The Maelstrom workbench

    Hand inAn echo and a unique-id node passing the workbench tests

  2. Week 28 h
    • Time and order: physical clocks, skew and drift
    • Lamport and vector clocks
    • Happens-before
    • Why timeouts are guesses

    Hand inA note that orders eight events with a vector clock, checked by a program

  3. Week 38 h
    • Failure detection with heartbeats
    • Gossip and membership
    • Retries, duplicates and idempotent handlers
    • At-least-once against exactly-once

    Hand inThe failure detector and the duplicate-delivery test

  4. Week 48 h
    • Broadcast and dissemination
    • Eventual consistency and CRDTs
    • Conflict resolution
    • Predicted against measured

    Hand inCheckpoint 1: the timeout table and the seeded duplicate bug

Practice projects

  • Practice 1, the ordering note (Week 2): assign vector clocks to eight events across three nodes. Done when: a program confirms your ordering and you name one pair of concurrent events.
  • Practice 2, the broadcast (Week 3): build a broadcast node that survives partitions. Done when: the workbench broadcast test passes with partitions and you report messages per operation.
  • Practice 3, the idempotent handler (Week 4): deliver every message twice. Done when: the final state matches single delivery, and the seeded bug fails the test before the fix.
  • Optional, a counter CRDT: build a grow-only counter that merges. Done when: the workbench counter test passes after partitions heal.

GitHub repos to study

Existence and last commit were checked on 22 Nov 2026 by shallow clone. Every repository below had a commit in the last 12 months. Look at the README for archive notices before you rely on one.

RepoWhat to readWeek
jepsen-io/maelstromThe protocol doc and the workload descriptions1
hashicorp/memberlistA gossip membership and failure detection library3
jepsen-io/jepsenHow faults and checkers are written, and the published analyses4

Milestone checkpoint

Build Checkpoint 1. Write your predictions for detection time and false suspicions first. Build the heartbeat and failure detector, run three timeout settings against random kills, and fill the table. Add the duplicate-delivery test and the seeded bug. Hand in the table with your own numbers, the test output, ADR-1 (the system model and the guarantee) and ADR-2 (the timeout choice).

Check yourself

Answer each question in your head first, then open it to check.

  1. Why can a node not tell a crashed peer from a slow one?
    In an asynchronous network there is no bound on delay, so silence is consistent with both. A timeout is a guess that trades detection time for false alarms.
  2. What does a vector clock tell you that a Lamport clock does not?
    It shows whether two events are concurrent. A Lamport clock only gives an order consistent with causality, so a smaller number does not mean causally earlier.
  3. What is the difference between at-least-once delivery and exactly-once effect?
    At-least-once means a message may arrive more than once. Exactly-once effect is reached by making handlers idempotent, so the repeats change nothing.
  4. What does eventual consistency promise, and what does it not?
    If updates stop, replicas converge to the same value. It does not bound how long, or say what a reader sees in the meantime.
  5. Why add jitter to retries and timeouts?
    Without jitter many clients or nodes retry at the same moment, which causes synchronised load spikes and, in elections, split votes.