Key takeaways
- ✓RAG searches your documents first, sends the top paragraphs to the model with rules (answer only from these, cite the source, say if you don't know), and returns the answer with its source.
- ✓It fixed invention: without documents, the model claimed a student discount, dark mode and Apple Pay the help center never mentions. With RAG, all four uncovered questions got "not enough information" in 36 of 36 runs.
- ✓It made answers checkable: every answer named its source. It didn't make them correct: checked against their sources, 40 of 54 answers to six covered questions were right.
- ✓Failure 1, the answer wasn't sent: with one paragraph, "We're 14 people" got Pro, every run; the 10-member limit was in another article. Sending three paragraphs fixed it.
- ✓Failure 2, the answer was sent and misread: "Can I pay without a credit card?" got "No" while listing PayPal, citing the right page. More paragraphs made it give up instead. Only checking each answer finds this.
RAG, retrieval-augmented generation, gives a model your documents before it answers. When a question comes in, the system searches the documents, sends the best paragraphs to the model with instructions to answer only from them, and returns the answer with its sources. It's the standard fix for a model making things up about your product. It fixes some things and not others: it makes answers checkable, not correct. This chapter covers how the pipeline works, what it fixed in a real test, how to check every answer, and the two ways answers still went wrong.
The examples use Lab 04's own RAG code (01_basic_rag.py: its search, prompt and rules) on the made-up help center from the search chapters: six articles and the getting-started guide, split into paragraphs. Ten questions were written before any run; every answer is a real recorded reply from gpt-5.4-nano (temperature 0), three runs each.
The pipeline
RAG runs in two phases:
- Indexing, once: split the documents into chunks, embed each chunk, store the vectors (Chapters 7–9).
- Querying, on every question: embed the question, retrieve the top k chunks, build a prompt from them, generate the answer, and return it with its citations.
The prompt carries the rules. Lab 04's system prompt:
You are a helpful assistant that answers questions using only the provided context.
Rules:
- Answer only from the context. Do not use prior knowledge.
- Cite the source document for each claim using [source_name] format.
- If the context does not contain enough information, say: "I don't have enough
information to answer that based on the available documents."
- Be concise. Do not pad your answer.
and each retrieved chunk goes in as [source] followed by its text, then the question.
When an answer is wrong, check the steps in order: what was retrieved, what was sent, then what the model did with it. Most bad answers start with retrieval.
What RAG fixed
Before RAG, the assistant had a plain help-center prompt and no documents. Four questions the help center doesn't cover:
| Question | No documents (3 runs) | RAG (1, 3 or 6 paragraphs; 9 runs) |
|---|---|---|
| Do you offer a student discount? | "Yes, we offer a student discount", every run | "I don't have enough information…" |
| Is there a dark mode? | Yes, with menu steps, every run | Not enough information |
| Can I pay with Apple Pay? | Yes, if available at checkout | Not enough information (and the methods it does list) |
| What's your support phone number? | An honest "I don't have one" | Not enough information |
With RAG, all four said so in 36 of 36 runs. RAG changed two things there: the paragraphs, and the rule to say when they aren't enough. And every answer it gave named its source. That's what RAG really buys: answers you can check.
Check every answer
So check them. Six questions the help center does answer, each with 1, 3 or 6 paragraphs sent, three runs each: 54 answers, each compared with what the help center says. 40 were right. 6 were wrong, 3 incomplete, and 5 said there wasn't enough information when there was.
Six questions, with 1, 3 or 6 paragraphs sent, three runs each: 40 of 54 right. Click any square to read the answer and what was sent.
| Question | 1 paragraph | 3 | 6 |
|---|---|---|---|
| Can I pay without a credit card? | |||
| We're 14 people. Which plan do we need and what does it cost? | |||
| If I cancel, how do I keep a copy of my projects? | |||
| Will the app work on a plane? | |||
| I was charged by mistake, can I get my money back? | |||
| Can it post updates into our Slack? |
Can I pay without a credit card? · 1 paragraph sent · run 1 · wrong
“No. We accept Visa, Mastercard, American Express, and PayPal, and we do not currently accept cryptocurrency or bank transfers. [billing]”
What was sent
- [billing] 0.48 · We accept Visa, Mastercard, American Express and PayPal. Cards are charged on the same day each month, or each year for annual plans. We do not currently accept cryptocurrency or bank transfers. To change the card on file, go to Account Settings, then Billing, then Payment method.
The failures came in two kinds, and they need different fixes.
Failure 1: the answer wasn't sent
"We're 14 people. Which plan do we need and what does it cost?" With one paragraph, search sent the pricing paragraph, and the model answered Pro, every run. Pro allows 10 members; that limit is in the teams article, which wasn't sent. With three paragraphs it answered Enterprise, citing both articles, every run.
"If I cancel, how do I keep a copy of my projects?" With one paragraph it explained exporting, but missed that data is deleted 30 days after the plan ends: incomplete, every run. With three, the 30-day paragraph was sent and the answer was complete.
The fix is in retrieval: send enough paragraphs, or retrieve better ones (Chapter 11). More isn't free: each paragraph costs tokens, and the lab notes that by k = 6 irrelevant text starts creeping in.
Failure 2: the answer was sent and misread
"Can I pay without a credit card?" Search put the billing paragraph first, well ahead of the rest (0.48 vs. 0.25). It says: "We accept Visa, Mastercard, American Express and PayPal."
| Paragraphs sent | Answers (3 runs) |
|---|---|
| 1 | "No. We accept Visa, Mastercard, American Express, and PayPal…" [billing], every run |
| 3 | "I don't have enough information…", every run |
| 6 | Not enough information twice; "You can pay with PayPal" once |
The answer cited the right page every time. It even listed PayPal while saying no. A citation tells you where the model looked, not that it read it right. More paragraphs didn't help. Retrieval can't fix this; only checking the answers finds it, and the fix is in the prompt, the model, or a rule for this kind of question.
How to check answers
- Write the rule for each test question, from the documents: "14 people needs Enterprise", "a complete answer mentions the 30-day deadline", "PayPal means yes". Simple text rules worked here; a model judge scales further, if you check it against your own labels first.
- Check the answer against what was sent, not just against the truth. Did the model have the answer? That separates failure 1 from failure 2.
- Check the citations: does every cited source appear in what was sent?
- Run more than once. One run is a sample: the card question gave three different outcomes across nine runs.
Lab 04 also measures faithfulness (is every claim supported by the retrieved text?) separately from correctness (is it complete and right?), with a model judge. An answer can be faithful and still incomplete, like the export answer without the 30-day deadline. And it applies a score cutoff (0.35) before the prompt, so weak matches never reach the model, as in Chapter 7.
The checklist
- Give the model rules: answer only from the context, cite sources, say when it isn't enough.
- Log what was retrieved and sent with every answer, so you can tell "not sent" from "misread".
- Test with questions the documents answer and questions they don't.
- Check each answer against its sources with written rules or a validated judge, over more than one run.
- Fix "not sent" in retrieval (k, chunking, search); fix "misread" in the prompt or the model, and re-check.
Practice: check the answers
Build the prompt the way the lab does, pull citations out of an answer and flag any that weren't sent, grade answers against written rules (right, incomplete, wrong, or "not enough information"), and tally a question's runs. The tests use the recorded answers to the card, 14-people and cancel-and-keep questions.
Quiz
1.What are the steps of RAG when a question comes in?
2.Without documents, the assistant said "Yes, we offer a student discount" in every run. What did RAG change for that question?
3.Every answer named its source. Does that mean the answers were right?
4."We're 14 people. Which plan do we need?" got "Pro" with one paragraph and "Enterprise" with three. Why?
5."Can I pay without a credit card?" The billing paragraph, listing PayPal, was first. What happened?
6.How should you check RAG answers?
Check your understanding
Try each question before opening its answer.
What happens at each step of RAG when a question comes in?
Embed the question, retrieve the top k chunks from the index, build a prompt with rules and the chunks, generate the answer, and return it with its citations. The chunks were split, embedded and indexed ahead of time.
What did RAG fix in the test?
Invention on uncovered questions: without documents the model claimed a student discount, dark mode and Apple Pay; with RAG, all four uncovered questions got "not enough information", 36 of 36. And every answer named its source, so it could be checked.
Why isn't a citation proof of a right answer?
It shows where the model looked, not how it read it. The card answer cited the billing page, which lists PayPal, and still said "No", every run with one paragraph.
How do you tell a retrieval failure from a reading failure?
Look at what was sent. If the answer wasn't in it (the 10-member limit with one paragraph), fix retrieval. If it was (PayPal, first), the model misread it: fix the prompt or model, and keep checking.
Why send more than one paragraph, and why not send a lot?
One paragraph missed the 10-member limit and the 30-day deadline; three fixed both. But each paragraph costs tokens, more text can distract the model, and the card question got worse with more.
What's the difference between faithfulness and correctness?
Faithful: every claim is supported by the retrieved text. Correct: the answer is complete and right. The export answer without the 30-day deadline was faithful but incomplete.
Next up: Chapter 11 tries to improve retrieval itself, with keyword search, hybrid search and a reranker, and measures each.