knaxuslearn · build · ship

Roadmaps / Data

Data, AI and web · 16 study weeks · 4 phases

Data Roadmap Beta

Data work fails quietly. A number is wrong, nobody notices, and a decision gets made on it. These 16 study weeks teach how to store data in open formats and lakehouse tables, how to write batch jobs that can be re-run safely, how to move changes from a database in near real time, and how to govern and pay for a data platform. You build one system, eventlake, the data platform of an online shop, and every phase adds a checkpoint to it. At 8 hours a week the core path is 128 hours and takes 16 weeks. At 12 hours a week it takes about 11 weeks, and at 16 hours a week about 8 weeks. At the end you can show, with reconciliation numbers, that your tables are correct, re-runnable and defensible.

This roadmap is in beta and keeps improving over time. If you spot an error, please email your suggestions and corrections to [email protected].

Read first

  1. Phase 1 · Study weeks 1-4

    Storage, formats and tables

    This phase makes the first checkpoint possible: data stored so that queries read little and schema changes do not break readers.

    • 5 repos
    • Milestone checkpoint
    1. Study week 1Row and column storage8 h
    2. Study week 2Partitioning and sorting8 h
    3. Study week 3Table formats: Iceberg and Delta8 h
    4. Study week 4Query engines: DuckDB, Spark and Trino8 h
  2. Phase 2 · Study weeks 5-8

    Batch you can trust

    This phase makes the second checkpoint possible: daily tables that are correct, re-runnable and checked before anyone reads them.

    • 5 repos
    • Milestone checkpoint
    1. Study week 5Idempotent jobs: overwrite by partition and merge by key8 h
    2. Study week 6Modelling: facts, dimensions and slowly changing dimensions8 h
    3. Study week 7Data quality tests8 h
    4. Study week 8Orchestration: dependencies, retries and backfills8 h
  3. Phase 3 · Study weeks 9-12

    Change capture and streaming

    This phase makes the third checkpoint possible: changes that flow from the database to the lake within seconds, survive a crash and handle late data.

    • 5 repos
    • Milestone checkpoint
    1. Study week 9Kafka: topics, partitions, offsets and consumer groups8 h
    2. Study week 10Change data capture from the Postgres log8 h
    3. Study week 11Streaming jobs: windows, watermarks and late data8 h
    4. Study week 12Failure drills: kill, restart and replay8 h
  4. Phase 4 · Study weeks 13-16

    Governance, cost and platform

    This phase makes the last checkpoint possible: a platform with owners, contracts, lineage, deletion and a cost you can explain.

    • 3 repos
    • Milestone checkpoint
    1. Study week 13Data contracts and ownership8 h
    2. Study week 14Lineage with OpenLineage8 h
    3. Study week 15Access control and masking8 h
    4. Study week 16Cost: bytes scanned, storage and compute8 h

Also in this roadmap