knaxuslearn · build · ship

Roadmaps / Data / Phase 1

Phase 1 of 4 · Study weeks 1-4

Storage, formats and tables

This phase makes the first checkpoint possible: data stored so that queries read little and schema changes do not break readers.

Topics every week

  1. Week 18 h
    • Row and column storage
    • Parquet: row groups, pages and encodings
    • Compression
    • Reading a file footer

    Hand inA note that labels the parts of one Parquet file, from a metadata dump

  2. Week 28 h
    • Partitioning and sorting
    • Statistics and pruning
    • Small files and compaction
    • Object storage behaviour

    Hand inThe three-way table with predictions

  3. Week 38 h
    • Table formats: Iceberg and Delta
    • Snapshots and time travel
    • Schema evolution
    • Hidden partitioning

    Hand inAn Iceberg table with a schema change and a time travel query

  4. Week 48 h
    • Query engines: DuckDB, Spark and Trino
    • Reading a query plan
    • Bytes scanned as a cost
    • Predicted against measured

    Hand inCheckpoint 1: the table, the small-files fix and the schema change test

Practice projects

  • Practice 1, the file footer (Week 1): dump the metadata of a Parquet file. Done when: you label row groups, column chunks and statistics and explain how one query would skip a row group.
  • Practice 2, the partition choice (Week 2): partition by day, by hour and by customer. Done when: a table shows files, bytes scanned and time for three queries for each, and you pick one with a reason.
  • Practice 3, time travel (Week 3): update a table twice. Done when: you query each snapshot, roll back one change and explain what is stored on disk for each.
  • Optional, the same table in Delta: write the table in Delta Lake. Done when: a table lists three differences you met (metadata layout, commit protocol, tooling).

GitHub repos to study

Existence and last commit were checked on 22 Nov 2026 by shallow clone. Every repository below had a commit in the last 12 months. Look at the README for archive notices before you rely on one.

RepoWhat to readWeek
apache/icebergThe table specification and the metadata file layout3
delta-io/deltaThe transaction log protocol3
apache/parquet-javaHow a file is written: row groups, pages and encodings1
apache/arrowThe columnar in-memory format1
duckdb/duckdbThe vectorised engine and its Parquet reader4

Milestone checkpoint

Build Checkpoint 1. Write your predictions for bytes scanned and time first. Generate 5 million orders, write them as CSV, plain Parquet and a partitioned and sorted Iceberg table, and run the same three queries on each. Create the tiny-files case, measure it and compact it. Add a nullable column and check an old reader. Hand in the table with your own numbers, the plans, ADR-1 (the table format) and ADR-2 (partitioning and compaction).

Check yourself

Answer each question in your head first, then open it to check.

  1. Why does a columnar format make analytic queries cheaper?
    A query reads only the columns it needs, and similar values in a column compress well.
  2. What is partition pruning, and what can defeat it?
    The engine skips partitions that cannot match the filter. A filter on a function of the partition column, or on a different column, can defeat it.
  3. Why do many small files hurt?
    Each file costs an open, a metadata read and a request, and object stores charge and delay per request, so overhead swamps the data read.
  4. What does a table format add on top of Parquet files?
    Atomic commits, snapshots, schema evolution and a record of which files make up the table, so readers see a consistent version.
  5. What does time travel cost you?
    Old data files must be kept until you expire the snapshots, so storage grows, and a deletion is not physical until you expire and clean up.