Key takeaways
- ✓RAG over scans has two steps that fail differently: find the page (search an index), then read it (the model answers from what it's handed). Measure each step on its own.
- ✓Indexing the text stored in each file works only for files that have any: 16 of 24 pages (scans and screenshots) had none, so only the 8 invoice questions could be answered. CLIP, which matches the question against each page's picture without reading it, put the right page first for 3 of 24.
- ✓Having a vision model write out every page once, when it arrives, made every page findable: the right page was in the top 3 for 24 of 24 questions in all 3 runs. Answers from those transcripts were right 23, 23 and 22 times of 24.
- ✓One misread name was the costly error: "Project Tern" was written down as "Project Term" in every run. Search still ranked it first, the card had the $91,000, and the answer was NOT FOUND every time. The lab's own earlier run wrote Plover as "Mover", and that memo fell out of the top 3.
- ✓Handing the model the page images fixed Tern but took Osprey's budget for Plover in 2 of 3 runs. Images and transcripts together got 24 of 24 in every run, for 58¢ per 1,000 questions against 6¢ for transcripts alone. Check the transcripts against names you already know.
Search over documents (Chapters 7 to 11) assumes every page has text to search. Scanned paper and screenshots don't: a scan stored as a PDF is a picture of paper, and a dashboard screenshot is pixels. RAG over them still has the same two steps, find the pages and read them to answer, but each step now has to deal with images, and each can go wrong on its own. This chapter covers three ways to index image-only pages, three ways to answer from what search found, and the error that matters most: a misread name, written into the index once and met by every question after.
The code is Lab 08's 04_multimodal_rag.py, with CLIP as Script 3 (03_image_search.py) introduces it: its archive, prompts, top-3 search and scoring. The archive has 24 pages made by code, so every answer is known: 8 digital invoices (PDFs with text inside), 8 scanned memos (one per project: Heron, Kestrel, Osprey…) and 8 dashboard screenshots. Each question's answer is on exactly one page. The transcripts and answers are real recorded replies from gpt-5.4-nano, in 3 runs; each run rebuilt the index from scratch. Search uses text-embedding-3-small and sends the top 3 pages.
Two steps, two places to go wrong
Think of an index as a filing clerk who writes an index card for every page when it arrives.
- Find: compare the question with every card and pull the 3 closest pages. That's retrieval, scored by recall@k: is the right page in the top k (Chapter 8)?
- Read: the model answers from what it's handed, with the lab's instruction: "Answer from these pages only. Reply with the number only, or NOT FOUND."
For scans, the question at step 1 is what goes on the card. At step 2 it's what the model gets to read: the cards, the page images, or both.
Step 1: what goes on the card
| Index | Right page first | Right page in the top 3 |
|---|---|---|
| The file's own text (the usual pipeline) | 8 of 24 | 8 of 24 |
| CLIP: match the question against each page's picture | 3 of 24 | 5 of 24 |
| A vision model's transcript of each page | 23, 22, 24 (3 runs) | 24 of 24, every run |
The file's own text. The standard pipeline pulls the text stored in each file and embeds it (Chapter 26 covers what that loses on a single page). An invoice PDF has text, so its card is full. All 8 scans and all 8 screenshots came out as (no text): 16 blank cards. Only the 8 invoice questions found their page, and only those were answered right, in every run.
CLIP. CLIP was trained on hundreds of millions of image–caption pairs, so a picture and a sentence describing it land close together in one embedding space. That lets you skip text entirely: embed the question and each page image, and search by similarity. It costs nothing per page and runs on a laptop. Here it put the right page first 3 times. For every memo question, its first pick was an invoice. For 7 of 8 dashboard questions its first pick was a dashboard, but the right one only once. It doesn't seem to read the names, which is all that tells one memo from another. Models trained on document pages, like ColPali, read small print far better; they want a GPU, and the lab names them but doesn't run them.
A vision model's transcript. When a page arrives, a vision model writes out all its text once, with the lab's prompt: "Transcribe this page for a search index: write out all its text, keeping labels next to their values. For charts, say what they show." The transcript becomes the card, and search runs on text as usual. Every page now had a card, and the right page was in the top 3 for every question, in every run. Writing the 24 cards cost about 28¢ per 1,000 pages, paid once.
Pick a way, then click any square to see the question, what search pulled, the right page's card, and the recorded answer:
Step 1 · find
A vision model writes out each page once; the transcript is the card.
Right page in the top 3: 24 · 24 · 24 of 24
First: 23 · 22 · 24 (runs 1 · 2 · 3)
Step 2 · read
The model reads the 3 cards.
Right answers: 23 · 23 · 22 of 24 (runs 1 · 2 · 3)
~263 input tokens per question · $0.06 per 1,000 questions
Invoices (PDF text)
Scanned memos
Dashboard screenshots
What budget was approved for Project Tern?
run 1 · right answer 91,000
Search pulled
- 1. memo · Tern ← the right page
- 2. memo · Kestrel
- 3. memo · Heron
This page's card: the vision model's transcript
MEMO Re: Project Term Project Term is approved with a budget of $91,000. The kickoff is in room 2.19. Please send questions to the project office.
Model answered: NOT FOUND · NOT FOUND
Step 2: what the model reads
All three readers below use the transcript index for step 1, so they get the same 3 pages.
| The model reads | Right of 24 (3 runs) | Input tokens per question | Per 1,000 questions |
|---|---|---|---|
| The file's own text (its own index) | 8, 8, 8 | ~275 | $0.06 |
| The 3 transcripts (cards) | 23, 23, 22 | ~263 | $0.06 |
| The 3 page images | 23, 24, 22 | ~2,627 | $0.53 |
| Page images + their transcripts | 24, 24, 24 | ~2,869 | $0.58 |
From the cards. Most answers were right. The miss in every run was Tern. Its memo says "Project Tern is approved with a budget of $91,000." Its transcript said "Project Term", with an m, in all 3 runs. Three facts sit side by side:
- Search still ranked Tern's card first, every run: Term was close enough to Tern.
- The card has the budget: $91,000.
- The question says Tern, the card says Term, and the model answered NOT FOUND, every run.
Run 3 had a second slip of the same kind: the Adatum Cleaning invoice was written out as "Datum Cleaning", search ranked it first, and the answer was NOT FOUND. In both, the near-miss name was close enough for search, and the answer was NOT FOUND.
From the page images. The model reads the pages itself, so the transcript's slip doesn't matter: Tern came back right, 91,000, every run. But a new miss appeared. Asked Plover's budget, it said 83,000 in 2 of 3 runs. Plover's budget is 66,000; 83,000 is Osprey's, and Osprey's memo was the second of the 3 pages it was handed, every run. The memos share one template, so they look almost alike. Plover's own page was first each time. Run 3 also said NOT FOUND for Merlin.
Pages and cards. The model got the 3 images and their transcripts, with a note that the transcripts may contain reading errors. 24 of 24, in all 3 runs. The test doesn't show why. A likely reading: the image covers the transcript's slips (Tern), and the transcript covers the image's (Plover).
The slip that happens before anyone asks
Tern shows the cheaper kind of index error: the page was still found, and a reader with the image recovered it. The lab's own earlier run of the same code shows the expensive kind. Its transcript wrote Plover as "Project Mover", and for the Plover question that memo ranked 4th, just outside the 3 pages the model reads. When a page isn't retrieved, no reader can recover it, text or image.
An index-time error is made once and stays. Every question about that page meets it, until someone re-indexes. So check the cards when they're written. A cheap check: compare every name on a card with the names you already know (vendors, projects, services). Here, that flags every name slip in all 4 runs, including Mover:
2 of 24 cards name something that isn't on our lists. Answers from the cards went wrong on 1; the check flagged 1 of 1.
memo · expected Ibis · card says IbIs
Answer from the cards: 52000 · right
memo · expected Tern · card says Term
Answer from the cards: NOT FOUND · NOT FOUND
The check reads "Project …" on memos, the first line on invoices, and "…-service" on dashboards. It's case-sensitive, so "IbIs" is flagged too, though that answer was right.
Across the 3 runs, answers from the cards went wrong 4 times (Tern 3 times, Adatum once), and the check flagged all 4. It also flagged "IbIs" for Ibis, which was harmless: that answer was right. A flagged card can be re-transcribed at a higher resolution or with a stronger model, or sent to a person. The lab also suggests fuzzy keyword search next to embeddings, so near-miss names still match.
Where the cost goes
There are two bills, paid at different times:
- At index time, once per page: writing the transcripts, about 28¢ per 1,000 pages.
- At answer time, on every question: about 6¢ per 1,000 questions from the cards, 53¢ from the page images, 58¢ from both. Three images per question are what cost the tokens (Chapter 26 shows how image size drives them).
So the usual plan: transcribe every scan and screenshot when it arrives, search the transcripts, and answer from the cards by default. Hand over the pages and their cards when the exact number matters.
Caveats
- Made-up pages, one template per kind. All 8 memos share one layout and differ only in the project name and numbers, which is what made Osprey's budget easy to mistake for Plover's. Real archives vary more, in ways that help and hurt.
- Small numbers. 24 questions, one per page, 3 runs. One question more or less changes a score by 4 points. Plover went wrong in 2 of 3 runs, so a single run could have missed it.
- The lab's README reports a different run. Its single run had the Mover slip, so its numbers are lower: the right page in the top 3 for 23 of 24, transcripts 22 of 24, page images 22 of 24 (missing Wren), both 23 of 24. This chapter uses the 3 fresh runs recorded for the video, where Plover was written out right every time.
- CLIP is a general model. It was trained on photos and captions, not documents; a document-trained retriever was not tested.
- The causes aren't tested. That Term caused the NOT FOUND, and that the two readers cover each other's slips, are the likely readings, not measured results.
- The scoring is loose. The lab's
is_rightcounts an answer right if any number in it equals the right one. With "number only" answers that's fine, but a reply listing several numbers could pass.
The checklist
- Measure finding and reading separately: recall@k per kind of page, then answer accuracy.
- Don't index scans and screenshots by their file text. Transcribe each page once with a vision model, or use a document-trained retriever, and measure it.
- Check every transcript when it's written, against names you already know. Re-transcribe or review what's flagged.
- Hand over the page images, with their transcripts, when the exact number matters. Record the cost per question.
- Test every kind of page, over more than one run. One run would have missed Plover.
Practice: check the index
Count how often search found the right page, using the recorded ranks for all 24 questions; write the lab's answer check; flag the names on a transcript that aren't on your lists, using the real "Term", "Datum Cleaning", "IbIs" and "Mover" cards; and work out what each way of answering costs per 1,000 questions.
Quiz
1.A scanned memo is a PDF. Why can't standard text-based RAG find it?
2.CLIP put the right page first for 3 of 24 questions. What did it do with the memo questions?
3.With a vision model's transcripts as the index, every right page was in the top 3. Why was Tern still answered NOT FOUND?
4.Why is a misread name at index time worse than a misread at answer time?
5.Given the 3 page images, the model answered Plover's budget as 83,000 in 2 of 3 runs. Where did that number come from?
6.Pages and their transcripts together got 24 of 24 in every run. Why not always do that?
Check your understanding
Try each question before opening its answer.
What are the two steps of RAG, and why measure them separately?
Find: search pulls the closest pages (scored with recall@k). Read: the model answers from what it's handed (scored by answer accuracy). They fail differently: Tern's page was found first every run and still answered NOT FOUND, so only measuring both tells you which step to fix.
Why did indexing the file's own text answer only 8 of 24?
Only the 8 invoice PDFs had text stored in the file. The 8 scans and 8 screenshots came out as "(no text)", so their cards were blank and search couldn't find them.
What does CLIP do, and why did it do badly here?
It puts images and text in one embedding space, so a question can be matched against page pictures without reading them. It's free and local, but it doesn't read small print: for every memo question its first pick was an invoice, and it put the right page first for 3 of 24.
Why was "Project Term" found but not answered?
Search only needs a close match, so the card ranked first every run, and it had the $91,000. But the question asked about Tern and the card said Term, and the model answered NOT FOUND every time. Given the page image instead, it answered 91,000.
How would you catch index-time slips like Term and Mover?
Check each transcript when it's written: compare the names on it with the names you know (projects, vendors, services). Here that flagged every name slip, including all 4 wrong answers from the cards and the lab's "Mover". Re-transcribe or review flagged pages, and consider fuzzy keyword search next to embeddings.
When is it worth handing the model the page images?
When the exact number matters and the page is a scan, screenshot or form. Pages plus transcripts got 24 of 24 in every run, but cost about 58¢ per 1,000 questions against 6¢ for transcripts alone, paid on every question.
Next up: Chapter 28 moves from pages to voice: a phone agent that has to start speaking within a second, timed step by step.