Key takeaways
- ✓A search test is a list of questions, each labelled with the article that should answer it. Check two things separately: did the right article rank first, and did the cutoff let it through?
- ✓Questions written while reading the docs borrow their words. Eight such questions found the right article 8/8 at scores of 0.51 to 0.78; the same needs in customer words found it 7/8, and 7 of 8 scored lower.
- ✓A cutoff set from the easy questions (0.50) handed 5 of 8 customer-worded questions to a person, and 4 of those had the right article first. "Will it work on a plane?" fell from 0.78 to 0.18 and still ranked right.
- ✓Set the cutoff from realistic questions plus questions nothing should answer: off-topic ones scored about 0.08, so 0.15 answered all eight and stopped both. Ten questions are a first guess.
- ✓Precision@k asks how much of the top k is right; recall@k asks how many of the right documents made it in. Report recall with a position metric like MRR, and keep the labels reviewed: a label is a decision.
A search test is a list of questions, each labelled with what should answer it. Run them, and you can see whether a change to the model, the chunking or the cutoff made search better or worse. The test is only as good as its questions, though. Written while reading the documents, they reuse the documents' words and match easily; real users don't know those words. This chapter covers what to check, why the wording of test questions matters so much, how to set a cutoff from a test, and the standard metrics, precision@k and recall@k.
The main example is a made-up help center written for the video: six articles (billing, cancelling, account, data, the app, teams), four paragraphs each, searched one paragraph at a time with Lab 03's paragraph chunker and FAISS search. Eight test questions, the same eight needs in customer words, and two off-topic questions were written before any search ran; every score is a real text-embedding-3-small result. The precision and recall numbers come from the lab's own golden set (05_evaluation.py), seven articles about AI.
What a search test checks
Each test question gets a label: the article (or passage) that answers it. Then check each stage on its own:
| Check | Question | What a failure means |
|---|---|---|
| Ranking | Is the labelled article the first result? | Search found the wrong thing |
| Cutoff | Is the top score high enough to answer? | A right result was thrown away, or a wrong one served |
| Answer | Does the returned text hold the whole answer? | The chunk is right but incomplete (Chapter 9) |
A result can pass one and fail another, and each failure has a different fix. Report them separately.
Test wording vs. customer wording
The eight test questions were written while reading the articles: "How do I reset my password?", "What payment methods do you accept?", "Does the mobile app work offline?". All eight ranked the labelled article first, with top scores from 0.51 to 0.78. A cutoff just below the lowest, 0.50, looks safe.
Then the same eight needs, the way customers ask: "I can't get into my account, what do I do?", "Can I pay without a credit card?", "Will it work on a plane?".
| Test wording | Customer wording | |
|---|---|---|
| Labelled article first | 8/8 | 7/8 |
| Top scores | 0.51 – 0.78 | 0.18 – 0.52; lower for 7 of 8 |
| Words shared with the answering paragraph (all eight) | 18 | 2 |
The ranking barely moved; the scores did. "Will it work on a plane?" still found the offline paragraph, at 0.18 instead of 0.78. The test questions borrowed the articles' words, so they matched easily. (The customer versions were also written for the video, not collected, which is the next thing to fix in a real test.)
Each pair asks the same thing. The test wording borrows the article's vocabulary; the customer wording asks for the same thing in other words. The line marks the cutoff on both bars; the outcome shown is for the customer wording. A correct rank can still be handed off.
Test: “How do I reset my password?” · Customer: “I can't get into my account, what do I do?”
Test: “What payment methods do you accept?” · Customer: “Can I pay without a credit card?”
Test: “How long is my data kept after I cancel my subscription?” · Customer: “If I quit, how long before my stuff is gone?”
Test: “How do I export my data?” · Customer: “How do I get a copy of everything I've made?”
Test: “Does the mobile app work offline?” · Customer: “Will it work on a plane?”
Test: “How many members can a Pro workspace have?” · Customer: “We're 14 people, can we all use Pro?”
Test: “What is your refund policy?” · Customer: “I was charged by mistake, can I get my money back?”
Test: “How do I turn on two-factor authentication?” · Customer: “How do I make my login more secure?”
At 0.50, four correct top articles are blocked. At 0.15, all eight customer questions pass, including one wrong top article. Two off-topic questions near 0.08 remain below that line.
The cutoff, checked on both
At 0.50, five of the eight customer questions fall below the cutoff and go to a person:
- Four are gate misses: the right article was already first. The cutoff threw those answers away.
- One is a ranking miss: "If I quit, how long before my stuff is gone?" ranked the data article's delete-account paragraph just ahead of the labelled cancelling article, by less than 0.001. No cutoff can reorder that.
So with 0.50, only 3 of 8 customers get a right answer, although search ranked the right article first for 7.
To set a better cutoff you also need questions nothing should answer. "How do I cook pasta?" and "What's the weather in Paris tomorrow?" scored about 0.08. A cutoff of 0.15 answers all eight customer questions (one with the wrong article) and stops both off-topic ones. Ten authored questions make that a first guess from the right kind of data, not a production threshold.
Labels are decisions
The quit question shows another point: how long before my stuff is gone could reasonably be answered by the cancelling article (data is kept 30 days) or the data article (deleting your account). A label is a person's judgement, not a fact in the text. Review labels, allow more than one right answer where it's fair, and look at close calls before counting them as failures.
Precision@k and recall@k
The checks above look at the first result. When several results go to a model (as in RAG), look at the top k:
| Metric | Question | Formula |
|---|---|---|
| Precision@k | Of the top k, how many are right? | right results in top k ÷ k |
| Recall@k | Of the right documents, how many made it into the top k? | right documents found ÷ all right documents |
For "What is the self-attention mechanism?", labelled transformers, the top 3 chunks came from transformers, transformers and fine-tuning: precision@3 = 2/3, recall@3 = 1. For a question labelled with two articles, a top 3 that only contains one of them has recall@3 = 1/2, however precise it looks.
In the lab's own run (eight questions, k = 3):
| Chunks | Precision@3 | Recall@3 |
|---|---|---|
| Paragraphs | 0.708 (17 of 24) | 0.938 |
| Fixed 300 characters, 50 overlap | 0.792 (19 of 24) | 0.938 |
Fixed chunks look more precise, but the two extra hits were neighbouring chunks of the same article, which share text. Read what a metric counts before trusting a gap, especially a 2-of-24 one. And when the same eight questions were rewritten in plain words without the articles' key terms, both dropped (precision@3 0.542 vs. 0.625, recall@3 0.812 vs. 0.938): the same lesson as above, on different data.
Precision and recall ignore position. MRR (how high the first right result is) and Hit@k (is any right result in the top k) add it, and nDCG scores the whole order. A good default: recall@k together with MRR or Hit@1.
The checklist
- Write test questions in users' words, ideally from real messages, and keep each one's exact text.
- Label the answering article and passage, review the labels, and allow more than one right answer where it's fair.
- Include questions nothing should answer. A cutoff needs both kinds.
- Check ranking, cutoff and answer separately, and report each.
- Report recall@k with MRR or Hit@1, and read what each metric counts.
- Record the documents, embedding model, chunker and cutoff with every run, and confirm a threshold on held-back questions before release.
Practice: score a search run
Score the eight recorded customer-worded results at two cutoffs (ranked right, answered, answered right, handed off), count off-topic rejections, then write precision@k and recall@k and check them on the lab's examples, including the case where precision looks perfect and recall shows what's missing.
Quiz
1.Eight test questions all ranked the right article first. Why didn't that mean search worked for customers?
2.With the cutoff at 0.50, what happened to the eight customer-worded questions?
3."Will it work on a plane?" ranked the offline article first at 0.18; "Does the mobile app work offline?" scored 0.78. What does that show?
4.At a cutoff of 0.15, all eight customer questions are answered and both off-topic ones stop. Is search fixed?
5.The top 3 for a question are chunks from transformers, transformers, fine_tuning, and only transformers is relevant. Precision@3 and recall@3?
6.In Lab 03, fixed-size chunks scored higher precision@3 than paragraphs (0.792 vs. 0.708) with recall tied. Does that make them better?
Check your understanding
Try each question before opening its answer.
What does a search test need?
Questions in the words users actually use, each labelled with the article or passage that answers it, plus questions nothing should answer. Then separate checks for ranking, the cutoff, and whether the returned text holds the answer.
Why did the 8/8 test give false comfort?
Its questions were written while reading the articles and reused their words (18 shared words vs. 2 for customer wording). The same needs in customer words scored lower for 7 of 8, and one ranked the wrong article first.
What's the difference between a ranking miss and a gate miss?
A ranking miss puts the wrong article first; changing the cutoff can't fix it. A gate miss is a right result thrown away by the cutoff (or a wrong one served). At 0.50, four of the five handoffs were gate misses.
How do you choose a cutoff?
From realistic answerable questions and questions nothing should answer: here off-topic scored about 0.08 and the lowest customer question 0.18, so 0.15 separated them. Then confirm it on held-back questions; ten questions are a first guess.
What do precision@k and recall@k each miss?
Precision@k doesn't see right documents that never showed up; recall@k doesn't see junk or where the right result sits. Neither looks at position, which is what MRR, Hit@k and nDCG add.
Why read what a metric counts before trusting it?
In the lab, fixed-size chunks had higher precision@3 (0.792 vs. 0.708), but the extra hits were neighbouring chunks of the same article sharing text: more chunks from one right article, not more information.
Next up: Chapter 9 splits a long guide into chunks, and adds the check this test misses: does the returned text hold the whole answer?