Roadmaps / Distributed Systems / Phase 3
Phase 3 of 4 · Study weeks 9-12
Storage, reads and scale
This phase makes the third checkpoint possible: a store that survives restarts, keeps its log small and serves reads with a stated guarantee.
Topics every week
- Week 98 h
- The write-ahead log and
fsync - Recovery after a crash
- Snapshots and log compaction
- Installing a snapshot on a slow follower
Hand inState that survives 20 kills with no lost write
- The write-ahead log and
- Week 108 h
- Linearizable reads: through the log, read index and leases
- Clock assumptions
- Stale reads on followers
Hand inA read path with its assumption written down and the clock-skew test
- Week 118 h
- Storage engines: LSM trees and B-trees
- Compaction
- Write amplification
- Sharding and routing
Hand inA table comparing an in-memory map and an LSM store for your state machine
- Week 128 h
- Throughput and tail latency
- Batching and pipelining
- Measuring with spread
- Predicted against measured
Hand inCheckpoint 3: the throughput report with spread and the 1-hour log size
Practice projects
- Practice 1, the recovery (Week 9): kill a node mid-write, 20 times. Done when: every acknowledged write is present after restart and the table shows recovery time.
- Practice 2, the lease read (Week 10): serve reads from the leader using a lease. Done when: a clock-skew test makes the lease unsafe, the checker catches a stale read and a skew bound fixes it.
- Practice 3, the batch size (Week 12): batch 1, 10, 100 and 1,000 writes per round. Done when: a chart shows throughput against p99 and you pick the size with a reason.
- Optional, two shards: split keys across two groups with a router. Done when: a test shows one group failing does not stop the other.
GitHub repos to study
Existence and last commit were checked on 22 Nov 2026 by shallow clone. Every repository below had a commit in the last 12 months.
| Repo | What to read | Week |
|---|---|---|
| facebook/rocksdb | The write path, compaction and the tuning guide | 11 |
| etcd-io/etcd | How a Raft log, snapshots and a key-value store are combined | 9 |
| tikv/tikv | A multi-group Raft store and its read paths | 10 |
| apple/foundationdb | Deterministic simulation testing | 12 |
Milestone checkpoint
Build Checkpoint 3. Write your throughput and log-size predictions first. Add the write-ahead log, snapshots and the read path. Run the 20 kills, the 1-hour log-size run and the throughput runs three times each. Seed a clock skew against the lease read. Hand in the throughput report with spread, the kill table, ADR-5 (the read path and its clock assumption) and ADR-6 (the snapshot policy).
Check yourself
Answer each question in your head first, then open it to check.
Why must a write be on disk before it is acknowledged?
If the node crashes after acknowledging but before storing, the write is lost, and the majority rule no longer protects it.What does a snapshot let you do, and what does it cost?
It lets you drop the old log and bring a slow follower up to date quickly. It costs the time and disk to write it and the work to keep it consistent with the log.What is the clock assumption behind a lease read?
Clocks on different nodes drift by at most a known bound, so a lease that has not expired on the leader has expired on no other node in a way that allows a second leader.Why does a larger batch raise throughput and latency together?
One round of replication and one flush cover more writes, so the fixed cost is shared, but each write waits for the batch to fill.What is write amplification in an LSM tree?
Each byte is rewritten several times as data moves down levels during compaction, so disk writes are larger than the data the application wrote.