Text extraction vs. vision models

A PDF can reach a language model in two ways. Text extraction pulls the words out of the file and sends them as text: it's cheap, exact, and the usual first step of a document pipeline. Or you can render the page and send a page image, a picture of the page, to a model that can read images (a vision model, also called a multimodal model). The difference is the layout. Extraction keeps the words and loses everything that isn't a word: marked boxes, chart bars, which number sits in which row, and every word on a scan. The page image keeps all of that, and here cost about 11 times as many tokens.

The source is Lab 08 (Multimodal & Voice), script 02_documents_as_images.py: six documents generated by code, so every right answer is known (an invoice, a price list, a claim form with checkboxes, a report with a bar chart, and two scans that are pictures of paper with no text inside), its questions, its instruction and its grader. The replies were recorded for the video, three fresh runs per question, from gpt-5.4-nano reading the extracted text, gpt-5.4-nano reading the page image, and gpt-5.4-mini reading the page image. One of the lab's 21 questions is left out ("Which product has the fewest units in stock?" expects the Filing cabinet, but the page says a blank cell means back order, so the Office chair arguably has fewer), which leaves 20 questions and 60 answers per setup.

Two ways to hand over a page

Every question went out with the same instruction: "Answer the question from the document. Reply with the answer only. If the document doesn't contain the answer, reply NOT FOUND."

What extraction leaves out

Here's what PyMuPDF's get_text() gave for each document:

• Both scans: 0 characters. A scanner stores a picture, not text.
• The claim form: every word, no marks. "Type of damage: Fire Water Theft Storm", "Police report filed? Yes No", "Preferred contact: Email Phone Post". A box and the cross in it are drawn lines, so nothing in the text says which one is marked.
• The report: the caption, not the chart. "Figure 1: defect rate by month (% of units shipped)", and not one bar.
• The price list: every value, in column order. All the SKUs, then all the product names, then all the prices, then the stock numbers.
• The invoice: everything. Clean digital text in a simple table.

Pick a document to compare its extracted text with the page, then click any square to read that reply:

Two ways to fail

When the answer isn't in the text, there are two ways the model can fail, and they're not equally bad.

NOT FOUND is the safe failure. On the scans, the chart questions and the damage type, the model followed its instruction: 27 of the 60 text answers. You know you don't know, so a fallback or a person can take over.

A confident wrong answer is the dangerous one. 9 of the 60, on three questions, the same answer in all 3 runs:

The instruction said to reply NOT FOUND when the answer isn't there. It didn't help, because the text looked like it held the answer: the right labels were all there. Note one pattern, an observation rather than a proven cause: both form answers are the first option listed in the text (Yes comes before No, Email comes first). Yet the third form question, which type of damage is marked, got NOT FOUND every run, not "Fire".

A wrong answer looks exactly like a right one, so it flows into the next step unnoticed. Count NOT FOUND and wrong answers separately when you test.

Every number there, still wrong

The price list is the sneakiest case, because nothing is missing from its text. The lab built it the way some reporting tools export PDFs: the table looks normal on the page, but the file stores it one column at a time. And one cell is blank: the Office chair is on back order.

A blank cell writes nothing. So the stock column comes out with 7 numbers for 8 products, and nothing in the text says which row is the blank one. Pair them up in order, and every number from the chair down lands one row too high:

Pairing in order gives exactly the 18 the model answered in all 3 runs, which makes it the likely reason, though not a proven one. The text even contains the note "A blank stock cell means the item is on back order", and that didn't help either. Only the layout said which number belonged to which product, and extraction threw the layout away.

A cheap guard for this case: before trusting a table's text, check that every column has the same number of cells.

The scoreboard

Right answers out of 3 runs per question:

The page image fixed every form, chart and price-list answer. All four of nano's misses were on the memo scan: the project lead's name, misspelled in all 3 runs ("Marcus Oyevaran", "Marcus OyeEaran", "Marcus Oyeoranan" for Oyelaran), and one finish date read as June 18 instead of July 18. The bigger gpt-5.4-mini got all 60.

What a page image costs

A model reads and bills in tokens, small chunks of a few characters each. A vision model cuts an image into patches and turns those into tokens too, billed as input next to the text. So more pixels cost more tokens. In the lab's first script, a chart cost 172 tokens at 512 pixels wide, 652 at 1,024 and 2,611 at 2,048, and nano's "detail" setting changed nothing: only the size did. Older models price images differently, so measure on yours.

That's about 11× the input tokens on average. The text average includes the scans, whose text is empty; on the four pages that do have text, it's 136 against 1,174 tokens, about 8.6×. The prices are the lab's list prices. Either way it's small money per question, and large at volume.

How sharp a scan needs to be

Scans add one more dial: resolution, in dots per inch (DPI). The lab's scans so far were at 100 DPI. The script made both again at 50, 75 and 150 and asked the six scan questions three more times each:

Tokens roughly double with each step. At 50 DPI the print turned to mush: the budget came back as $849,500 or $94,500, and the lead's name as "Marcus O'Bryan" or close to it. The name was still wrong at 75 and 100 DPI, and right in all 3 runs only at 150, for about 8× the tokens of 50 DPI (2,648 / 343). The one miss at 150 was a NOT FOUND on a shipping rate.

Pick the lowest resolution that still reads your smallest print, and find it by measuring on a labelled sample of your own documents. Re-check when the model changes, because models cut images into tokens differently.

Choosing per page

There's no single right way; route each page:

"Read twice and compare" catches a misread that varies between reads, like the lead's name, which came back different in all 3 runs. It doesn't catch the text's Yes or 18, which were the same every time; for those, compare the text reading with the image reading. OCR (optical character recognition) gives a scan a text layer cheaply and locally, but it still drops marks and chart values and often the table structure, so it sits between the two.

Lab 08 covers more than this chapter. Script 01 asks the same question about charts: on eight questions about two chart images (one run each), nano got 6 of 8 and mini 8 of 8, while nano given the numbers as text got 5 of 8, its misses being counting and comparing mistakes, not reading ones. Scripts 03–04 search over images and scans (Chapter 27), script 05 measures speech-to-text (covered in the Voice AI track), and scripts 06–07 time a voice agent (Chapter 28).

Caveats

• Made-up documents, a best case. Every page was drawn by code, and the scans were made to look scanned (rotated, blurred, speckled, saved as low-quality JPEG). Real scans, handwriting and photos are messier.
• Small numbers. 20 questions, 3 runs each. The size of the gap (24 vs 56 of 60) is clear; small differences between setups wouldn't be.
• The lab's README shows its own single cached run (21 questions: 9, 20 and 21 right; ~113 text tokens; scans scored per question). This chapter uses the three fresh runs recorded for the video. Both point the same way.
• The "first option" pattern behind Yes and Email is an observation from two questions, not a tested cause. Likewise the column pairing explains 18 exactly but isn't proven to be what the model did.
• The 100-DPI row is a separate set of runs from the main scoreboard, on the same image. Both got 14 of 18 on the scans.
• Prices are the lab's list prices (nano $0.20 / $1.25, mini $0.75 / $4.50 per 1M input / output tokens). Check current pricing.

The checklist

1. Look at what extraction gives you for each kind of page before building on it: scans, forms and charts often give nothing, or not the part you need.
2. Route per page: clean digital text to extraction; scans, forms, charts and tricky tables to the page image.
3. Score NOT FOUND and wrong answers separately. The wrong ones are the ones that hurt.
4. Guard extracted tables: check every column has the same number of cells before pairing values with labels.
5. Pick scan resolution by measuring: the lowest DPI that still reads your smallest print, on your own documents.
6. Check the names and numbers that matter with a second, different reading: a bigger model, or the image against the text.

Practice: grade the readings

Write the lab's grader and sort the recorded replies into right, NOT FOUND and wrong (24, 27 and 9 from the text). Split the price list's real extracted text into columns, pair them in order to get the footrest's 18, and write the check that flags the short column. Then compute the cost per 1,000 questions from the recorded token counts, and pick the lowest scan resolution that reaches a target score.

Quiz

Check your understanding

Try each question before opening its answer.

Next up: Chapter 27 searches an archive of scans and screenshots, where every page is read once, long before anyone asks a question.