Phase 3 of 4 · Study weeks 9-12
Change capture and streaming
This phase makes the third checkpoint possible: changes that flow from the database to the lake within seconds, survive a crash and handle late data.
Topics every week
- Week 98 h
- Kafka: topics, partitions, offsets and consumer groups
- Ordering and keys
- Retention and compaction
- Delivery guarantees
Hand inA producer and consumer with a measured throughput and a lag number
- Week 108 h
- Change data capture from the Postgres log
- Debezium
- Snapshots and schema changes
- Deletes and tombstones
Hand inOrders flowing from Postgres into a Kafka topic
- Week 118 h
- Streaming jobs: windows, watermarks and late data
- State and checkpoints
- Exactly-once effects with idempotent sinks
Hand inA streaming job writing to Iceberg with a late-data test
- Week 128 h
- Failure drills: kill, restart and replay
- Duplicate and missing row checks
- Lag at p95
- Predicted against measured
Hand inCheckpoint 3: the lag report and the kill-restart check
Practice projects
- Practice 1, the offsets (Week 9): stop a consumer, produce 1,000 messages and restart it. Done when: you show the committed offset, the lag and the point where a crash would cause a repeat.
- Practice 2, the delete (Week 10): delete an order in Postgres. Done when: the delete reaches the lake as a delete or a marked row and a test proves it.
- Practice 3, the late events (Week 11): send 1,000 events 90 minutes late. Done when: they land in the right day, and events 3 hours late are counted in a late-data table, with a test for each.
- Optional, a different engine: run the same stream in a streaming SQL engine. Done when: a table compares lag, state size and lines of code with your first.
GitHub repos to study
Existence and last commit were checked on 22 Nov 2026 by shallow clone. Every repository below had a commit in the last 12 months.
| Repo | What to read | Week |
|---|---|---|
| apache/kafka | The log, the consumer group protocol and the docs on delivery | 9 |
| debezium/debezium | The Postgres connector and how it handles snapshots | 10 |
| apache/flink | Watermarks, state and checkpointing | 11 |
| risingwavelabs/risingwave | A streaming database, as a comparison | 11 |
| airbytehq/airbyte | How a managed connector handles incremental sync | 10 |
Milestone checkpoint
Build Checkpoint 3. Write your lag prediction and the row-check query first. Stand up Postgres, Debezium, Kafka and the streaming job, run 10 minutes at 2,000 events a second and report lag. Send the late events. Kill the job in the middle of a run, restart it and run the check. Seed a duplicate-delivery bug and catch it with the check. Hand in the lag report, the late-data test, the kill-restart result, ADR-5 (the sink and its idempotency) and ADR-6 (watermarks and the late-data rule).
Check yourself
Answer each question in your head first, then open it to check.
What does a Kafka consumer offset mean, and what happens if you commit it too early?
It marks how far the consumer has read. Committing before the work is done means a crash loses those messages, and committing after means a crash repeats them.Why does change capture read the database log and not poll the table?
The log has every change in order, including deletes, at little load on the database. Polling misses deletes and intermediate updates and adds load.What is a watermark?
A moving estimate of how far event time has progressed, used to decide when a window can close and how late an event may be.How do you get an exactly-once effect from an at-least-once stream?
Make the sink idempotent: write by key with a merge, or write with a transaction that includes the offset, so a repeat has no effect.What happens to a delete in a log-based pipeline?
It arrives as an event with a delete marker (and often a tombstone), which the sink must turn into a delete or a flag, and the table format must remove the data on cleanup.