knaxuslearn · build · ship

Roadmaps / AI / Phase 2

Phase 2 of 4 · Study weeks 5-8

Evaluation

This phase makes the second checkpoint possible: a way to say whether a change made the system better, backed by an evaluation set and a judge you have checked.

Topics every week

  1. Week 58 h
    • Evaluation sets: sources, coverage and labels
    • Answerable and unanswerable cases
    • Splits and leakage
    • Labelling with a second person

    Hand inThe evaluation set (100 questions) with expected answers and source passages

  2. Week 68 h
    • Scorers: exact checks, rubrics and model judges
    • Judge bias and agreement with humans
    • Calibrating on 30 labels

    Hand inA scorer with a measured agreement with your human labels

  3. Week 78 h
    • Variance: temperature, seeds and sampling
    • Confidence intervals
    • How many runs
    • Paired comparisons

    Hand inThe score report with mean and spread over 5 runs

  4. Week 88 h
    • Regression tests in CI
    • Online signals and feedback
    • Benchmarks and why they mislead
    • Predicted against measured

    Hand inCheckpoint 2: the seeded regression caught in CI

Practice projects

  • Practice 1, the evaluation set (Week 5): write 100 questions from your documents, 20 of them unanswerable. Done when: every question has an expected answer or a refusal and a source passage, and a second person checked 20.
  • Practice 2, the judge check (Week 6): score 30 answers by hand and with the judge. Done when: you report agreement and list three cases where the judge was wrong.
  • Practice 3, the noise floor (Week 7): run the same system 5 times. Done when: you state the smallest score change you can trust and why.
  • Optional, a public benchmark: run a small public benchmark on your system. Done when: a note says what it measures and why it does not predict your results.

GitHub repos to study

Existence and last commit were checked on 22 Nov 2026 by shallow clone. Every repository below had a commit in the last 12 months.

RepoWhat to readWeek
UKGovernmentBEIS/inspect_aiHow tasks, solvers and scorers are structured6
promptfoo/promptfooEvaluation configs and CI use8
SWE-bench/SWE-benchHow a benchmark builds test cases and checks answers8

Milestone checkpoint

Build Checkpoint 2. Write the evaluation set and the labelling rules first. Build the scorer and check it against 30 human labels. Run the system 5 times and report mean and spread. Seed a prompt change that makes the assistant refuse too often, and show the CI evaluation failing on it. Hand in the set, the scorer, the raw scores, ADR-3 (the scoring rules) and ADR-4 (the noise floor and the pass rule for CI).

Check yourself

Answer each question in your head first, then open it to check.

  1. Why must an evaluation set include questions the system should refuse?
    Otherwise a system that always answers looks perfect, and the real risk (confident wrong answers) goes unmeasured.
  2. What can go wrong when a model judges another model?
    The judge may prefer longer or its own style of answer, may be wrong on facts it does not know and may drift with a model update. Check it against human labels and re-check after changes.
  3. Why report spread over several runs?
    A single run of a stochastic system can be lucky or unlucky, so you cannot tell a real change from noise.
  4. What is leakage between an evaluation set and a prompt?
    Examples from the set leak into the prompt or tuning data, so the score measures memorisation, not the ability to answer new questions.
  5. When does a public benchmark score mislead?
    The benchmark may differ from your task, may have leaked into model training and rewards narrow skills, so it does not predict performance on your data.