Key takeaways
- ✓Meaning search (embeddings) put the right paragraph first for 10 of 12 questions. Keyword search (BM25) managed 7, and missed one of the six exact-term questions it's supposed to be good at.
- ✓Keywords match words, not meaning: "Will it work on a plane?" matched "will" and "work" in other paragraphs, and the right one, which says "works", ranked 23rd of 34.
- ✓Hybrid search merges both rankings by position (reciprocal rank fusion). It scored 9: it fixed one question, pushed two right answers down, and dropped the plane paragraph from its top 10, because only each search's top 20 counts.
- ✓A reranker has a model read the question and the candidates together and reorder them. It moved a right answer from 6th to 1st and lifted hybrid to 11 of 12, at about 825 tokens and a second per question, but it can't bring back a paragraph search already dropped.
- ✓On meaning search, the reranker scored 12, 12 and 11 over three runs. On this help center keywords didn't earn their place; measure every upgrade on your own questions, and check search on its own before reranking.
Retrieval decides what the model gets to read, so when RAG answers are wrong because the right paragraph wasn't sent (Chapter 10), the fix is better retrieval. Three upgrades are standard advice: add keyword search, merge it with meaning search into hybrid search, and add a reranker that reorders the results with a model. Each one helps on some questions and hurts on others. This chapter explains how each works and measures all of them on the same labelled questions.
The code is Lab 04's own (02_advanced_retrieval.py: its dense search, BM25, reciprocal rank fusion, hybrid_search and llm_rerank), run on the made-up help center from the search chapters: 34 paragraphs. Twelve customer questions were written before any run, each labelled with the one paragraph that answers it: six use exact product terms (Okta, VAT number, TLS 1.3, iOS 15, CSV, a 15-minute lockout) and six use everyday words. Every ranking is real; the reranker (gpt-5.4-nano) ran three times.
The scoreboard
"Right" means the labelled paragraph came first.
| Method | Right (of 12) | Exact terms (of 6) | Everyday words (of 6) |
|---|---|---|---|
| Meaning search (embeddings) | 10 | 6 | 4 |
| Keyword search (BM25) | 7 | 5 | 2 |
| Hybrid (both, merged) | 9 | 6 | 3 |
| Hybrid + reranker | 11, 11, 11 | 6 | 5 |
| Meaning + reranker | 11, 12, 12 | 6 | 5–6 |
Right paragraph first, of 12. Each square shows where the right paragraph ranked.
Will it work on a plane?
Meaning
#1
Keywords (BM25)
#23
Hybrid
—
Hybrid + reranker
— — —
Meaning + reranker
#1 #1 #1
Right paragraph [app]: The mobile app works offline for anything you opened recently. You can read projects, tasks and comments without a connection. Creating or editing needs the internet, and the app shows a banner while you are offline.
The reranker put “Comments keep discussion next to the work. Type the at sign followed by a teammate's name to mention them, and they get a notification right…” first: the offline paragraph wasn't in the list it was given.
Meaning search
The search from Chapters 7–9: embed the question and every paragraph, rank by cosine similarity. It handled all six exact-term questions, and missed two everyday ones by one place: "If I quit, how long before my stuff is gone?" (the delete-account paragraph first) and "How do I make my login more secure?" (the lockout paragraph first). In both, the right paragraph was second.
Keyword search (BM25)
BM25 is the classic keyword ranking: a paragraph scores higher the more of the question's words it contains, with rare words counting more and long paragraphs counting a little less. It's exact where embeddings blur: product codes, error messages, function names.
Here it scored 7. On exact terms, 5 of 6: "Does it run on iOS 15?" put the right paragraph second. On everyday words it struggled. "Will it work on a plane?" matched "will" in the password paragraph and "work" in the automations and comments paragraphs; the offline paragraph says "works", with an s, and ranked 23rd of 34. Keywords match words, not meaning, and typos hurt them the same way.
Hybrid search
Hybrid search runs both searches and merges the two rankings. The usual merge is reciprocal rank fusion (RRF): each list gives a paragraph 1 / (60 + rank) points, and the points add up. It only uses positions, so cosine scores and BM25 scores never have to be put on one scale. The 60 stops the very top positions from dominating.
The effect: a paragraph that shows up in both lists, even far down, can beat one that's first in only one. Here that cut both ways. Hybrid fixed the quit question (the right paragraph was second for meaning, first for keywords) and pushed down two right answers: "How do I get a copy of everything I've made?" from 1st to 2nd, and the plane paragraph out of the top 10.
Why the plane paragraph fell out is an implementation detail: the lab's hybrid_search merges only each search's top 20 (twice the k it returns). At keyword rank 23 the offline paragraph got no keyword points, only 1/61 from meaning search. Paragraphs found by both searches, like the comments paragraph (meaning #6, keywords #3), outscored it, and it landed 12th. Merging the full lists would have put it 6th. Read the merge code before trusting the method's reputation.
Reranking
Embedding search encodes the question and each paragraph separately: that's what makes it fast, since paragraph vectors are computed ahead of time. But the model never reads the question and a paragraph together, so it can't tell apart paragraphs on the same topic that answer different questions. A reranker is a second stage that does:
- First stage: fast search returns a pool of candidates (here, the top 10).
- Second stage: a slower model reads the question with the candidates and reorders them.
| Reranker | How it scores | Trade-off |
|---|---|---|
| Cross-encoder (e.g. a MiniLM or BGE reranker) | A small model scores each question–paragraph pair | Fast and cheap, runs locally. The usual choice |
| Hosted reranker (Cohere, Voyage, Jina) | The same, behind an API | Stronger, nothing to run, costs per request |
| LLM reranker (used here) | Ask an LLM to score the candidates | Easy to try, slowest and most expensive |
For "How do I make my login more secure?", hybrid had the two-factor paragraph 6th. The reranker gave it 10 out of 10 and moved it to 1st, in every run. Hybrid plus the reranker scored 11 of 12 in all three runs, one better than meaning search alone. The miss was the plane question: the reranker put the comments paragraph first, because the offline paragraph wasn't in the pool it was given. A reranker reorders; it can't bring back what search dropped.
It cost about 825 tokens and 1.1 seconds per question with an LLM; a cross-encoder would be cheaper and faster.
What won here
Meaning search had every right paragraph in its top two, so a reranker had something to work with: meaning search plus the reranker scored 12, 12 and 11 over three runs. Keyword search didn't earn its place: meaning search already handled every exact term asked about.
That's a result for this help center, not a rule. Guides recommend keywords for rare codes, part numbers and error strings, and these questions had product names and versions but none of those. Twelve questions is a small test.
| Add | When |
|---|---|
| Keyword search / hybrid | Your questions contain rare exact strings (codes, part numbers, error messages) that meaning search misses, and you've checked the merge depth |
| A reranker | The right passage is usually in the pool but not first, order matters (few passages sent), and you can pay the latency |
| Neither | Meaning search already ranks the right passage first; measure before adding cost |
Lab 04 covers two more upgrades not measured here. Query expansion has a model rewrite the question several ways and merges the searches. HyDE has a model write a made-up answer and searches with that, since it reads more like a document. Both cost extra model calls per question. The lab also scores rankings with MRR and nDCG, which reward the right result being near the top (Chapter 8).
The checklist
- Label each test question with the passage that answers it, with both exact-term and everyday wording.
- Measure every upgrade before and after, per question, not just the total: hybrid fixed one and broke two here.
- Check the first stage on its own: is the right passage in the pool? A reranker can't fix recall.
- Read the implementation: merge depth, pool size and k decide results.
- Record the cost (tokens, latency) of each stage, and re-test after changing documents or models.
Practice: merge and rerank
Write reciprocal rank fusion with a merge depth, a rank lookup, and a reranker that reorders a pool. The tests use the recorded meaning and keyword rankings for all 12 questions: they reproduce the 10, 7 and 9 right, the plane paragraph's 1st / 23rd / 12th place (6th with full lists), and the login question's move from 6th to 1st.
Quiz
1.How does BM25 score a paragraph?
2."Will it work on a plane?" Meaning search ranked the offline paragraph 1st. Where did keyword search rank it, and why?
3.How does reciprocal rank fusion merge two rankings?
4.Why did hybrid search drop the plane paragraph from its top 10, though meaning search had it first?
5.What can a reranker fix, and what can't it?
6.On this help center, which setup won?
Check your understanding
Try each question before opening its answer.
What is keyword search good at, and what is it bad at?
Exact, rare strings: codes, names, error messages. It's bad at rewording, plurals and typos, because it matches words, not meaning: "work" didn't match "works", so the plane answer ranked 23rd.
How does reciprocal rank fusion work, and why use positions?
Each list gives a paragraph 1 / (60 + rank); points add up across lists. BM25 and cosine scores are on different scales, so adding scores would let one dominate. Positions avoid that.
Why did hybrid search lose the plane paragraph?
The lab merges only each search's top 20. At keyword rank 23 it got no keyword points, so paragraphs found by both searches outscored it and it fell to 12th. With full lists it would have been 6th.
Why does a reranker rank better than embedding search, and why can't it replace it?
It reads the question and each candidate together, so it can tell apart passages on the same topic. But nothing can be computed ahead of time, so scoring a whole collection per question is too slow: it only reorders a pool from fast search.
What did the reranker cost here?
About 825 tokens and 1.1 seconds per question with an LLM reranker. It moved the login answer from 6th to 1st and lifted hybrid from 9 to 11 of 12.
When would you add keyword search?
When your questions contain rare exact strings that meaning search misses, like part numbers and error codes. Here meaning search already got every exact term, so keywords only added risk. Measure on your own questions.
Next up: Chapter 12 answers questions from a database instead of documents: text-to-SQL, and what the model can't see.