Key takeaways
- ✓A chunk is what search stores one embedding for and hands back. Too big, and one vector covers too many topics; cut in the wrong place, and the answer is split between two chunks.
- ✓Stored whole, a ten-topic guide ranked first for 5 of 8 questions and returned all 2,365 characters. Split at paragraphs, the first result held the whole answer for 8 of 8, at about 230 characters.
- ✓Cutting every 300 characters kept the whole answer first for 5 of 8; the Slack answer was cut at "The Sl". Adding 50 characters of overlap made it 3 of 8: overlap moves the cuts, and here they landed on more answers.
- ✓Across 16 chunk sizes from 250 to 400, 50 characters of overlap kept more answers whole at 11 sizes, fewer at 2 and the same at 3. Measure size and overlap on your own documents; neither is a safe default.
- ✓"Right article first" said 8/8 for every split but the whole guide. Add a check that the returned text holds the whole answer, and keep "exists in some chunk" and "comes back first" apart.
Search doesn't store whole documents. It stores chunks, one embedding each, and hands back a chunk as its result. So the way you split a document decides what search can find and what text it returns. Make a chunk too big, and its single vector covers too many topics to match any question strongly. Cut in the wrong place, and the answer ends up split between two chunks, neither of which holds all of it. This chapter compares the common ways to split, what each one did on a real example, and the test that catches an answer cut in half.
The example is a made-up getting-started guide written for the video: 2,365 characters, ten topics, one paragraph each (templates, reminders, shortcuts, integrations, automations…), with eight customer-style questions, each labelled with the one guide sentence that answers it. It's searched together with the six help-center articles from Chapter 8, so a wrong result can come from another article. The splitting and search code is Lab 03's (chunk_paragraph, chunk_fixed and exact FAISS search); every score is a real text-embedding-3-small result.
Two ways chunking fails
- Dilution. One chunk covers many topics, so its embedding is a bit about everything and matches no question strongly. A shorter, focused chunk from another document can outrank it.
- Boundary loss. The right chunk ranks first, but a cut runs through the answer, so the rest of it sits in another chunk.
They need different fixes, and neither shows up in a "right article first" check.
The common splits
| Split | How it cuts | Trade-off |
|---|---|---|
| Whole document | One chunk per document | Simple; dilutes, and returns far more text than needed |
| Paragraph | At blank lines | Keeps the author's structure; sizes vary |
| Fixed size | Every N characters, wherever that falls | Predictable sizes; cuts through words and sentences |
| Fixed size + overlap | Each chunk starts some characters before the last one ended | Text near a cut appears twice; more chunks to embed, and every cut moves |
| Sentence / recursive | N sentences, or paragraphs, then sentences, then words until small enough | Handles messier documents; more complex |
What each did on the guide
Choose a split. Each tile is one customer question: green returns its whole answer first, amber finds the guide but misses the full sentence, and gray ranks another article first.
The Slack integration posts task updates to a channel you choose.
Complete labelled answer: “The Slack integration posts task updates to a channel you choose.”
The complete answer exists in at least one stored guide chunk for 8/8 questions. It still has to rank first to reach the user.
| Split | Guide ranked first | Whole answer in the first result | Whole answer exists in some chunk | Text returned |
|---|---|---|---|---|
| Whole guide, one chunk | 5/8 | 5/8 | 8/8 | 2,365 characters |
| Paragraphs (10 chunks) | 8/8 | 8/8 | 8/8 | about 230 |
| Every 300 characters (8 chunks) | 8/8 | 5/8 | 6/8 | about 295 |
| 300, with 50 overlap (10 chunks) | 8/8 | 3/8 | 3/8 | about 299 |
| 300, with 150 overlap (16 chunks) | 8/8 | 4/8 | 8/8 | about 295 |
Whole guide. Each question scored 0.41 to 0.68 against its own paragraph, but only 0.29 to 0.41 against the whole guide. Three questions went to shorter articles on other subjects; "Can it post updates into our Slack?" went to the app article at 0.29, with the guide a close second at 0.28. When the guide did win, search handed back all 2,365 characters for a one-sentence answer.
Paragraphs. Ten focused chunks, and every first result held the whole answer. One caveat: the guide was written one topic per paragraph, which suits this split. Long multi-topic paragraphs, tables or headings change that.
Fixed size. At 300 characters, the first Slack result ends "…Integrations connect your workspace to the tools you already use. The Sl". The guide still ranks first, so a right-article check passes, but the answer is cut in two.
Overlap. 50 characters of overlap is a common fix. Here it made things worse: 3 of 8. Overlap shifts every cut by 50 characters per chunk, and the new cuts landed on more answers. It rescued neither answer that was already cut and cut three more; the Slack chunk now ends "…posts task updates to a channel yo".
Was 300 just unlucky? Across 16 sizes from 250 to 400, 50 characters of overlap kept more answers whole somewhere at 11 sizes, fewer at 2 (270 and 300) and the same at 3. (That sweep counts whether each answer survives whole in any chunk; it doesn't rerun search.) So overlap isn't a guarantee either way. With fixed sizes, where the cuts land decides what survives.
With 150 characters of overlap, every answer existed whole in some chunk, but only 4 came back whole first: the cut copies competed with the whole ones and often won.
Test the returned text
The right-article check from Chapter 8 said 8/8 for every split except the whole guide. It can't see a cut answer. Add two checks:
- Whole answer first: does the first result contain the entire labelled answer?
- Answer exists: does any chunk contain it whole? If not, no ranking can fix it: change the split.
Exact text matching works here because the answer sentences were labelled in advance. On real documents, label answer passages, and review cases where a paraphrase should count.
The checklist
- Split where the meaning changes: sections and paragraphs first. Use recursive splitting for long or messy ones.
- Label answer passages, not just documents, for a set of realistic questions.
- Measure "answer exists" and "whole answer first" separately, plus how much text comes back.
- Choose size and overlap by measuring them on your own documents and questions, not because a value is common. Inspect the cuts on failed questions.
- Re-run after changing documents, the chunker or the embedding model. Eight questions on a tidy guide are a start, not a benchmark.
Practice: split, then check the text
Write a fixed-size splitter with overlap (and an error for a step that would never advance), plus the two checks: does any chunk hold the answer, and does the first result hold it whole. The tests run on the real guide: they find the chunk ending "The Sl" and reproduce the 6, 3 and 8 answers that survive at 0, 50 and 150 characters of overlap.
Quiz
1.What is a chunk?
2.The guide stored as one chunk ranked first for only 5 of 8 questions, though it contained every answer. Why?
3.At 300 characters, the first result for "Can it post updates into our Slack?" ended with "The Sl". What kind of failure is that?
4.Adding 50 characters of overlap at 300 dropped whole-answer-first from 5 of 8 to 3 of 8. What's the lesson?
5.With 150 characters of overlap, every answer existed whole in some chunk, but only 4 came back whole in the first result. What does that show?
6.Paragraphs won 8 of 8 here. Should paragraph splitting be the default everywhere?
Check your understanding
Try each question before opening its answer.
Why does chunking affect answer quality, not just storage?
Search stores one embedding per chunk and returns the chunk's text. A chunk that's too broad matches weakly; a cut through the answer means the first result can't hold all of it.
Why did the whole guide lose three questions it contained the answers to?
One embedding for ten topics is a bit about everything: questions scored 0.29 to 0.41 against it, vs. 0.41 to 0.68 against their own paragraph, so shorter articles won. When it did win, it returned 2,365 characters.
Does overlap help?
Sometimes. It repeats text near cuts, but it also moves every cut. At 300 characters, 50 of overlap took whole-answer-first from 5 to 3 of 8; across 16 sizes it helped at 11 and hurt at 2. Measure it.
What's the difference between "answer exists" and "whole answer first"?
Exists: some chunk holds the whole answer. First: the top result does. With 150 overlap, all eight existed but only four came back whole first, because cut copies outranked the whole ones.
Why isn't "right article first" enough?
It said 8/8 for every split except the whole guide, including the ones that cut the Slack answer at "The Sl". Check the returned text, not just its source.
Should paragraph splitting be the default?
It won here because the guide has one topic per paragraph. Split where the meaning changes, then measure on your own documents; long paragraphs, tables and sections may need recursive splitting.
Next up: Chapter 10 puts search to work: RAG sends the retrieved paragraphs to a model, and every answer gets checked against them.