Phase 3 of 4 · Study weeks 9-12
Retrieval and grounding
This phase makes the third checkpoint possible: answers that cite their sources, and refusals when the sources are silent.
Topics every week
- Week 98 h
- Chunking: size, overlap and structure
- Embeddings and what they capture
- Vector indexes (exact, HNSW)
- Metadata filters
Hand inThe index over your documents and a recall at 5 measurement
- Week 108 h
- Keyword search and hybrid search
- Reranking
- Query rewriting
- Where retrieval fails
Hand inThe retrieval comparison table
- Week 118 h
- Grounded answers and citations
- Citation checking
- Refusal rules
- Handling conflicting sources
Hand inA citation checker and the refusal rate on unanswerable questions
- Week 128 h
- Freshness: updates, deletions and re-indexing
- Access control in retrieval
- Cost of the index
- Predicted against measured
Hand inCheckpoint 3: the table, the citation report and the stale-index test
Practice projects
- Practice 1, chunk sizes (Week 9): index with three chunk sizes. Done when: a table shows recall at 5 and index size for each and you pick one with a reason.
- Practice 2, the failure list (Week 10): list the 20 questions retrieval got wrong. Done when: each has a cause (wording, exact code, split across chunks) and you fix the top cause with a measured gain.
- Practice 3, the citation check (Week 11): verify every cited passage supports its claim. Done when: a report gives citation accuracy and shows five bad citations with the reason.
- Optional, per-user access: filter results by a user's allowed collections. Done when: a test shows a user never receives a passage from a collection they cannot read.
GitHub repos to study
Existence and last commit were checked on 22 Nov 2026 by shallow clone. Every repository below had a commit in the last 12 months.
| Repo | What to read | Week |
|---|---|---|
| pgvector/pgvector | The index types and their trade-offs | 9 |
| qdrant/qdrant | A dedicated vector database: filtering and payload design | 9 |
| langfuse/langfuse | How traces and scores are stored for retrieval debugging | 12 |
Milestone checkpoint
Build Checkpoint 3. Write your recall and refusal predictions first. Build the index, run vector, keyword and hybrid search with and without a reranker, and fill the table. Build the citation checker. Seed the stale-index bug (update a document and check whether the old text is still served) and fix it. Hand in the table, the citation report, ADR-5 (chunking) and ADR-6 (the search method and the rerank depth).
Check yourself
Answer each question in your head first, then open it to check.
Why do embeddings miss exact product names and error codes?
Embeddings capture meaning and blur rare exact strings, so keyword search or hybrid search handles them better.What does a reranker add, and what does it cost?
It reads the question and each candidate together and orders them better, at the cost of extra latency and compute for every query.How do you stop the model from inventing a source?
Give the model numbered passages, require it to cite by number, check every citation against the passage and refuse when no passage supports an answer.What happens to answers when a document is updated or deleted?
Without a re-index or a delete in the vector store, the old text can still be retrieved. Test updates and deletions explicitly.How do you keep one user's documents from reaching another user?
Filter in the retrieval query by the user's permissions before ranking, not after generation. Test with a second user.