Key takeaways
- ✓An embedding turns text into a fixed-length vector (1,536 numbers for text-embedding-3-small) so that texts with similar meanings land close together. Search embeds the question and ranks stored texts by cosine similarity.
- ✓A score says how close two texts are, not whether one answers the other. A same-need rewording scored 0.58 and a different question 0.55: read scores next to each other, never against a fixed number.
- ✓Search always returns something, so add a cutoff that hands weak matches to a person. Set it from your own answerable and unanswerable questions: 0.40 sent a right answer at 0.38 away; 0.30 kept all five and stopped both off-topic questions.
- ✓A cutoff only stops weak matches. A wrong match scored 0.66, higher than three of five right ones, so no cutoff that keeps the right ones can stop it.
- ✓Context narrows the candidates: filtering to billing FAQs removed the wrong data answer, though the refund policy was still second. Filtering changes what can rank; a cutoff only decides whether to serve the top result.
Search by meaning finds text that means the same thing as a question, even when it shares none of its words. It works by turning text into vectors and ranking stored texts by how close their vectors are to the question's. The ranking is useful. The score that comes with it is easy to misread: it says how close two texts are, not whether one answers the other. This chapter covers how embedding search works, what a similarity score tells you, how to set a cutoff for questions nothing answers, and what a cutoff can't fix.
The examples are from ai-eng-labs Lab 03, 01_embeddings_basics.py and 04_faq_retrieval.py: a made-up help center of 15 FAQs in four categories (billing, data, technical, support), and made-up customer questions. Every score is a real result from OpenAI's text-embedding-3-small; no chat model is involved.
Embeddings
An embedding is a fixed-length list of numbers, a vector, that represents the meaning of a text. text-embedding-3-small turns any text into 1,536 numbers. No single number means anything you could name. Together they place the text in a space where texts with similar meanings land near each other: "I can't log in" and "my account is inaccessible" end up close, with no word in common.
Two practical rules follow:
- Embed questions and documents with the same model. Vectors from different models live in different spaces and can't be compared.
- A flat picture is only a sketch. Squashing 1,536 directions onto a page (with PCA or UMAP) shows rough clusters, but distances on the page don't match the real ones.
Similarity scores
To compare two embeddings, the usual measure is cosine similarity: whether two vectors point the same way. 1 means the same direction, 0 means unrelated.
score = dot(a, b) / (norm(a) * norm(b))
OpenAI's vectors already have length 1, so the dot product gives the same number. Euclidean distance moves the opposite way (closer means a smaller distance) and, for vectors of length 1, ranks results in the same order.
Scored against "How do I reset my password?":
| Text | Relationship | Score |
|---|---|---|
| I forgot my password, how do I reset it? | Same meaning, other words | 0.83 |
| Steps to recover access to my account | The same need | 0.58 |
| How do I update my email address? | A different question | 0.55 |
| What is the refund policy? | A different topic | 0.22 |
| The transformer architecture uses self-attention. | Unrelated | −0.01 |
Two things to notice. A near-identical rewording scores 0.83, not 1. And the same need (0.58) and a different question (0.55) are almost tied. A score only means something next to other scores, from the same model and collection. 0.5 isn't good or bad on its own.
Search: score, then sort
Search is that score, computed against every stored text, then sorted:
- Store: embed every FAQ question and keep the vector with its question and answer.
- Compare: embed the customer's question with the same model and score it against every stored vector.
- Rank: sort by score. The first result is the closest candidate in this collection, not necessarily a right one.
The lab normalizes the vectors and uses FAISS IndexFlatIP (inner product), which gives cosine rankings. That's exact search: every vector is compared. It's fine up to around 100,000 vectors; beyond that, approximate indexes (HNSW, IVF) skip most comparisons, run much faster, and occasionally miss a match.
The strength of search by meaning: "Does it work without internet?" finds "Does the app work offline?" first at 0.72, with the next result around 0.30. Keyword search can't connect "internet" with "offline".
A cutoff for "nothing fits"
Search always returns something. "How do I cook pasta?" gets the password-reset FAQ first, at 0.22. So add a cutoff: if the top score is below it, hand the question to a person. A cutoff doesn't change the ranking; it only decides whether to serve the first result.
Set it from your own questions, some the help center can answer and some it can't. Move it across the lab's seven:
Each bar is the best result search found. The vertical line is the cutoff. Move it and watch which useful matches disappear.
What the line cannot fix
A separate refund question ranked the wrong data policy first at 0.66. A right refund FAQ in this test scored 0.38. Any cutoff that lets 0.38 through also lets 0.66 through. Inspect the ranked result and use context, not just a score gate.
| Cutoff | Answerable questions kept (of 5) | Off-topic questions stopped (of 2) |
|---|---|---|
| 0.40 | 4: "I need to get my money back" found the refund policy at 0.38, and was handed off | 2 |
| 0.30 | 5 | 2 |
| 0.20 | 5 | 1: pasta (0.22) gets through |
0.40 looks like a reasonable number from a tutorial, and it throws away a right answer. 0.30 works on these seven. That's a first guess: a different model, collection or way of asking needs its own calibration, on more questions (Chapter 8).
What a cutoff can't fix
"Can I get my money back if I cancel?" returns "What happens to my data if I cancel?" first, at 0.66: about cancelling, not money. That's higher than three of the five right answers above (0.47, 0.43, 0.38). Any cutoff low enough to keep them keeps this wrong match too. A cutoff stops weak matches. It can't stop a confident wrong one.
The top three all mention cancelling; the refund policy is only fourth, at 0.52. What helps is changing the candidates, not the threshold:
- Filter by context. The customer was on the billing page, so search only billing FAQs. Now "How do I cancel my subscription?" is first (0.56) and the refund policy second (0.52). The data answer is gone, though the refund policy still isn't first. (The lab filters after searching, post-filtering; a vector database can also pre-filter, searching only the matching items.)
- Show more than one result, or ask a clarifying question when the top results are close.
- Test for confident wrong matches on purpose: label which FAQ should answer each question and check the top result, not just its score (Chapter 8).
The checklist
- Use the same embedding model for questions and documents, and record it with every result.
- Read scores relative to each other, never against a number from somewhere else.
- Set the cutoff from your own data: questions that should be answered and questions that shouldn't. Check what it wrongly sends away.
- Check the top result itself, not just its score, for confident wrong matches.
- Use trustworthy context (page, product area, category) to narrow the candidates, then test the ranking again.
- Switch to approximate search only when exact search gets too slow, and measure what it misses.
Practice: rank, then decide
Implement cosine similarity (with a clear error for mismatched vectors), a stable ranking, the answer-or-hand-off decision, and a report that calibrates a cutoff on the seven recorded top scores. The tests include the right refund match at 0.38 and the wrong data match at 0.66.
Quiz
1.What does text-embedding-3-small return for one sentence?
2."Steps to recover access to my account" (the same need as a password reset) scored 0.58; "How do I update my email address?" (a different question) scored 0.55. What follows?
3."How do I cook pasta?" returns the password-reset FAQ at 0.22. Why?
4.A cutoff of 0.40, copied from a tutorial, was tried on five answerable and two off-topic questions. What went wrong?
5."Can I get my money back if I cancel?" returned "What happens to my data if I cancel?" at 0.66. Can a cutoff stop it while keeping the right refund match at 0.38?
6.The lab searches with FAISS IndexFlatIP on normalized vectors. What does that mean?
Check your understanding
Try each question before opening its answer.
What is an embedding, and why does search by meaning work?
A fixed-length vector (here 1,536 numbers) that places a text by its meaning. Texts with similar meanings get nearby vectors, so a question can find an answer that shares none of its words, like "without internet" and "offline".
What does a similarity score tell you?
How closely two vectors point the same way in this model's space. It ranks candidates; it doesn't say the top one is right. A same-need rewording scored 0.58 and a different question 0.55.
Why does search need a cutoff, and how do you set it?
Search always returns something, even for "how do I cook pasta?". Below the cutoff, hand the question to a person. Set it from your own answerable and unanswerable questions, and check which right answers it sends away: 0.40 lost one at 0.38; 0.30 kept all five here.
Why can't a cutoff fix "Can I get my money back if I cancel?"
Its wrong match scored 0.66, higher than three right answers. Any threshold that keeps 0.38 also keeps 0.66. It needs better candidates: context, more results, or a clarifying question.
What did filtering to billing change, and what didn't it?
It removed the data FAQ from the candidates, so cancel subscription (0.56) and the refund policy (0.52) came first and second. The refund policy still wasn't first, so a serving flow might show both or ask.
When would you use approximate search?
When exact search, which compares every vector, gets too slow: around 100,000 vectors and up. Approximate indexes like HNSW or IVF skip most comparisons and occasionally miss a match, so measure recall before and after switching.
Next up: Chapter 8 builds a search test, and shows how test questions written from the docs make search look better than it is.