Phase 2 of 4 · Study weeks 5-8
Batch you can trust
This phase makes the second checkpoint possible: daily tables that are correct, re-runnable and checked before anyone reads them.
Topics every week
- Week 58 h
- Idempotent jobs: overwrite by partition and merge by key
- Late-arriving data
- Deterministic transforms
- Run dates against event dates
Hand inA daily job and the hash check that two runs match
- Week 68 h
- Modelling: facts, dimensions and slowly changing dimensions
- dbt models and tests
- Incremental models
Hand inRevenue and conversion models with tests
- Week 78 h
- Data quality tests
- Reconciliation against the source
- Quarantine for bad rows
- Alerts that mean something
Hand inThe reconciliation report and the three seeded bad-data tests
- Week 88 h
- Orchestration: dependencies, retries and backfills
- Backfilling safely
- Cost and runtime of a backfill
- Predicted against measured
Hand inCheckpoint 2: the 90-day backfill on a copy and its time
Practice projects
- Practice 1, the double run (Week 5): run the same day twice. Done when: a hash of the output matches, and a test fails if a run appends instead of replacing.
- Practice 2, the slowly changing dimension (Week 6): keep the history of a customer's country. Done when: a query returns the country as of 3 March and a test covers a change on that day.
- Practice 3, the bad row (Week 7): inject a negative price, a duplicate order and a missing key. Done when: each is caught by a test, quarantined with a reason and never reaches the report table.
- Optional, the same job in Spark: run the daily job on Spark in local mode. Done when: a table compares runtime and lines of code with your first engine.
GitHub repos to study
Existence and last commit were checked on 22 Nov 2026 by shallow clone. Every repository below had a commit in the last 12 months.
| Repo | What to read | Week |
|---|---|---|
| dbt-labs/dbt-core | How models, tests and incremental strategies are compiled | 6 |
| dagster-io/dagster | Assets, partitions and backfills | 8 |
| great-expectations/great_expectations | How checks are declared and run | 7 |
| apache/spark | The execution model and the adaptive query execution docs | 8 |
| trinodb/trino | The Iceberg connector and the query planner | 8 |
Milestone checkpoint
Build Checkpoint 2. Write the reconciliation query and the three bad-data tests first. Build the daily job and the models. Run each day twice and compare hashes. Run the 90-day backfill on a copy and time it. Hand in the hash comparison, the reconciliation report, the test results, ADR-3 (the write strategy: overwrite or merge) and ADR-4 (the quarantine and alert rules).
Check yourself
Answer each question in your head first, then open it to check.
What makes a job idempotent?
Running it again with the same inputs produces the same output, usually by overwriting a whole partition or merging on a key and never appending blindly.Why reconcile against the source, not only test the output?
Tests catch rules you thought of. A reconciliation catches silent loss and duplication that no test described.What is the difference between event time and processing time?
Event time is when the thing happened. Processing time is when your system saw it. Late data makes them differ, and reports usually need event time.How do you backfill without breaking readers?
Build into a copy or a branch of the table, check it against the old one, then swap in one atomic commit, with a snapshot to roll back to.When is an alert worse than none?
When it fires often enough that people ignore it, or fires with no clear action, so a real problem gets lost.