Roadmaps / Data
Data, AI and web · 16 study weeks · 4 phases
Data Roadmap Beta
Data work fails quietly. A number is wrong, nobody notices, and a decision gets made on it. These 16 study weeks teach how to store data in open formats and lakehouse tables, how to write batch jobs that can be re-run safely, how to move changes from a database in near real time, and how to govern and pay for a data platform. You build one system, eventlake, the data platform of an online shop, and every phase adds a checkpoint to it. At 8 hours a week the core path is 128 hours and takes 16 weeks. At 12 hours a week it takes about 11 weeks, and at 16 hours a week about 8 weeks. At the end you can show, with reconciliation numbers, that your tables are correct, re-runnable and defensible.
This roadmap is in beta and keeps improving over time. If you spot an error, please email your suggestions and corrections to [email protected].
Read first
Phase 1 · Study weeks 1-4
Storage, formats and tables
This phase makes the first checkpoint possible: data stored so that queries read little and schema changes do not break readers.
- 5 repos
- Milestone checkpoint
- Study week 1Row and column storage8 h
- Study week 2Partitioning and sorting8 h
- Study week 3Table formats: Iceberg and Delta8 h
- Study week 4Query engines: DuckDB, Spark and Trino8 h
Phase 2 · Study weeks 5-8
Batch you can trust
This phase makes the second checkpoint possible: daily tables that are correct, re-runnable and checked before anyone reads them.
- 5 repos
- Milestone checkpoint
- Study week 5Idempotent jobs: overwrite by partition and merge by key8 h
- Study week 6Modelling: facts, dimensions and slowly changing dimensions8 h
- Study week 7Data quality tests8 h
- Study week 8Orchestration: dependencies, retries and backfills8 h
Phase 3 · Study weeks 9-12
Change capture and streaming
This phase makes the third checkpoint possible: changes that flow from the database to the lake within seconds, survive a crash and handle late data.
- 5 repos
- Milestone checkpoint
- Study week 9Kafka: topics, partitions, offsets and consumer groups8 h
- Study week 10Change data capture from the Postgres log8 h
- Study week 11Streaming jobs: windows, watermarks and late data8 h
- Study week 12Failure drills: kill, restart and replay8 h
Phase 4 · Study weeks 13-16
Governance, cost and platform
This phase makes the last checkpoint possible: a platform with owners, contracts, lineage, deletion and a cost you can explain.
- 3 repos
- Milestone checkpoint
- Study week 13Data contracts and ownership8 h
- Study week 14Lineage with OpenLineage8 h
- Study week 15Access control and masking8 h
- Study week 16Cost: bytes scanned, storage and compute8 h