Prompt caching and the Batch API

Many API calls repeat most of what they send. This chapter measures two ways to pay less for them: prompt caching, where the provider reuses a repeated start of the prompt, and the Batch API, half price for work that can wait. It also shows what quietly breaks the cache.

The source is Lab 12 (Inference & Serving), scripts 03_prompt_caching.py and 04_batch_api.py. The video reran the lab's own code (its prompt, questions, call settings and tickets) on 2 November 2026, with real calls to OpenAI's gpt-5.4-nano (snapshot gpt-5.4-nano-2026-03-17), and kept every call: three runs of each caching test, one batch of 20 tickets, and the same tickets twice as normal calls. All of it cost $0.025. The support assistant is a sample for a made-up software company, Ledgerly, and its "manual" is filler text standing in for a real one; costs are at list price. Chapter 32 covered streaming and slow calls; chapter 30 showed the same idea as prompt caching on your own server (prefix caching).

A few words first

• The prompt is everything the app sends. It starts with the system prompt, fixed instructions sent every time (here, a manual that is nearly all of the ~2,860-token prompt), then the customer's question.
• Tokens are pieces of words. You pay per token: input tokens for what you send, output tokens for what comes back.
• On gpt-5.4-nano, per million tokens: input $0.20, cached input $0.02, output $1.25. An output token costs about six times an input token.

What prompt caching is

Think of a barista who knows a regular's usual. The moment the order starts the usual way, she starts making it. If it starts any other way, she starts from scratch.

The provider does the same: it remembers the start of a prompt it just read, so it doesn't have to read it again. When the next prompt starts exactly the same way, that part is read from the cache, and billed at a tenth of the price on this model. It's automatic, but best-effort: it can miss. Every response says how many input tokens came from the cache, in usage.prompt_tokens_details.cached_tokens, so you can check.

The lab's calls also send a prompt_cache_key. OpenAI runs many machines, each with its own cache; this optional label sends calls with the same start to the same one, which makes hits more likely. Each test series used a fresh prompt and key, so its first call always started cold.

Five calls, measured

Five different questions, each behind the same ~2,860-token prompt. Run 2 of 3:

From the second call on, each call cost about half as much. Why only half, if cached tokens cost a tenth? Because only part of the prompt was cached: 1,792 tokens, 63%, the same amount on every hit. The rest is paid in full:

The runs don't show why the cache stopped at 1,792 tokens. (The lab's notes say OpenAI's cache works in steps, the first 1,024 tokens and then 128 at a time; the recorded runs only show where it stopped.)

And a cache can miss. Across the three runs, 11 of 12 repeat calls hit. The one that didn't, call 3 of run 1, had the same start and paid full price. So check cached_tokens; don't assume.

Switch the prompt, the run, and the call:

What breaks it

Some apps put the current time at the top of the system prompt. In this test it's the exact time, down to the microsecond, so it's new on every call:

With the time first, every prompt is different from its first line, and the cache has nothing to reuse. Moved to the end, the long manual is the same start again, and calls were cached from the second or third one, the same 1,792 tokens as before. The cache can only reuse a prompt's start, never past the first difference.

So the rule is: stable parts first, changing parts last. Instructions, tool definitions and reference documents go first; the date, the user's details and the question go last. A date that changes once a day should hurt far less than a time that changes every call, but that wasn't tested here.

Watch for hidden changes at the top, too: a list of tools (the actions your app lets the model call) built in a different order each run, for example from a Python set, or JSON whose keys come out in a different order. The start changes, and the cache misses, with nothing in the code that looks like a timestamp.

Too short to cache

The same five questions with one short line of rules instead of the manual, 56–63 tokens: 0 of 15 calls cached. OpenAI's docs say it only caches prompts of about 1,024 tokens or more. (The lab didn't measure the cutoff itself.)

So caching pays off for long starts that repeat: big instructions, tool lists, or a document you ask many questions about. A short prompt is cheap anyway: 8.6–8.8¢ per 1,000 calls here.

The Batch API

The second discount is for work nobody is waiting on, like sorting yesterday's support tickets or a nightly test run. A batch is many requests sent together as one file, one request per line, each with its own name, a custom_id:

{"custom_id": "ticket-000", "method": "POST", "url": "/v1/chat/completions",
 "body": {"model": "gpt-5.4-nano", "reasoning_effort": "none", "max_completion_tokens": 60, "messages": [...]}}

Twenty support tickets went in, each to be summed up in at most 8 words and given a category (billing, bug, account, feature_request, security):

• All 20 completed, none failed, in 2.9 minutes from creation to completion. That's not a promise: the only promise is within 24 hours.
• Half the list price for the same tokens. This batch's tokens (1,039 in, 254 out) come to 2.6¢ per 1,000 tickets at list price, 1.3¢ as a batch. The half is OpenAI's documented batch price applied to these tokens, not a bill.
• Results can come back in any order. These came back in order this time, but OpenAI doesn't promise it, so match each result by its custom_id, never by its position.

Are batch answers the same as normal calls? Not word for word. But neither are normal calls:

Word for word, the two rounds of normal calls matched on 0 of 20. Language models vary from run to run, batch or not. The disagreements were on tickets between two categories: slow search (bug or feature request?), a profile picture that won't change (bug or account?), switching to annual billing mid-cycle (billing or feature request?).

Also in the lab

The lab's README and checklist answers record the lab's own earlier run of the same scripts, with slightly different counts:

• Fewer calls per test. The scripts send 5 calls with the same start, and 3 each for the timestamp and short-prompt tests (the video's rerun used 5 everywhere, three times). In that run, too, 1,792 of about 2,860 tokens were cached from the second call; the timestamp at the end was cached from the third call; a 61-token prompt was never cached.
• The batch. The lab's run finished in about 2.5 minutes, $0.00028 instead of $0.00057 for normal calls. The script estimates the normal price from the batch's own tokens and says "same answers" without checking; the video's rerun sent the tickets as normal calls twice to check, which is where 18 of 20 comes from.
• Failures. A request in a batch can fail on its own. Look at the batch's error file and resubmit just the failed custom_ids (the practice below does the "which ones" part).
• Other providers. The checklist answers note that Anthropic's caching needs explicit cache_control markers and charges extra to write the cache; the stable-first rule is the same. Not tested in the lab.

What to do

• Order every prompt: stable parts first, changing parts last. Instructions, tools and documents, then the date, the user and the question.
• Keep the stable part byte-for-byte identical: tools in a fixed order, JSON with sorted keys, no IDs or times at the top.
• Log cached_tokens on every call. Caching is best-effort; a drop to zero is a sign something at the top started changing.
• Don't expect caching on short prompts. Under about 1,024 tokens, there's nothing to cache.
• Send work that can wait as a batch: nightly tests, labeling, re-scoring old conversations. Not anything a user is waiting on.
• Give every batch request a custom_id and match results by it; resubmit the ones with no result.

Caveats

• One small model, one day. gpt-5.4-nano on 2 November 2026, at list price. Other models have other cache prices and steps; Anthropic's caching works differently.
• Filler text. The manual stands in for a real one. A real manual of the same length should cache the same way, but that wasn't tested.
• Small samples. Three runs of five calls per test, one batch of 20. "11 of 12" means the cache usually hit, not that it hits 92% of the time.
• A cache key helps. Every caching call sent a prompt_cache_key, which makes hits more likely. Without one, expect more misses.
• Speed wasn't measured. Cached and uncached calls took similar, mixed times (cached 0.69–3.12 s); the lab doesn't claim caching made calls faster.
• The batch price is OpenAI's documented discount applied to the tokens, not a billed amount, and 2.9 minutes was one batch on one day.

The checklist

1. Is everything that changes per call at the end of the prompt?
2. Are tool lists and JSON in a fixed order?
3. Is the shared start over about 1,024 tokens? If not, don't count on caching.
4. Do you log cached_tokens, and alert when it drops?
5. Which jobs can wait a day? Send those as batches.
6. Does every batch request have a custom_id, and do you match and retry by it?
7. Did you compare batch answers with a second round of normal calls, not just one, before deciding they differ?

Practice: price a call, find the break, match a batch

Price a call with and without the cache, count hits after the first call, find how much of a prompt can be reused, and match a batch's results by custom_id. The tests use the recorded runs and expect the video's numbers: 64.7¢ and 32.4¢ per 1,000 calls, 11 of 12 hits, 0 with the time first and 10 with it last, 1.3¢ for the batch, and 18 of 20 three ways.

Quiz

Check your understanding

Try each question before opening its answer.

Next up: Chapter 34, semantic caching and cascades: reusing a whole answer for a question that means the same thing, and letting a cheaper model answer first.