Roadmaps / Data / How the plan works
Read first
How the plan works
Time model. One study week is 8 hours: about 2 hours of reading, 5 of building and 1 of review. The core path is 16 weeks, 128 hours. Optional stretch work adds about 30 hours, so the full path is 158 hours. Fresher pacing: plan 20 weeks (160 hours), with one extra week in each phase for debugging and rerunning the checkpoint. At 5 hours a week, allow 26 weeks for the core path. Week 0 is an extra 10 hours and is optional. Practice projects are the building time of their weeks, not extra hours.
One stack for the core path. The core path uses Parquet files, Apache Iceberg tables, DuckDB or Spark for batch, dbt for transformations, Kafka with Debezium for change capture and Flink or a SQL streaming engine for the stream. Delta Lake and other engines appear as comparison exercises. Pick a different table format if you prefer; the checkpoints do not depend on the choice.
Every phase has the same parts. A weekly table, up to three practice projects, up to five repositories to read, one milestone checkpoint and five questions to check yourself. The checkpoint is the point of the phase. The weekly deliverables are steps toward it.
Scoring. Score each checkpoint yourself, then ask one other engineer to score it. Use six dimensions: specification and budgets; trade-offs and decisions; correctness under failure; measurement; security; communication and influence. Use four levels: 1 Works on your machine; 2 Sound (specified, tested, repeatable: a stranger can rerun your measurements); 3 Proven (trade-offs measured, holds at the limits, a seeded fault is caught); 4 Leading (a method other teams reuse, with a measured outcome). With no second engineer, post your hand-in for review in a community forum and cap your self-score at 2.
| Checkpoint | Judged mainly on |
|---|---|
| 1 Stored | Specification; measurement |
| 2 Batch correct | Correctness under failure; measurement |
| 3 Streaming | Correctness under failure; trade-offs |
| 4 Governed and cheap | Security; communication; trade-offs |
Three rules of the bench. Reconcile every output against its source before you trust it. Write your prediction before each measurement; the deliverables of Weeks 4, 8, 12 and 16 compare predicted and measured numbers. Every job you write must give the same result when run twice.