Semantic caching and cascades: shortcuts around the model

Every call to a paid model costs time and money. This chapter tests two shortcuts around that call: a semantic cache, which reuses an answer already given when a new question means the same, and a model cascade, which lets a cheap model answer first and asks a bigger one only when the cheap one is unsure. Both work by guessing when the expensive call isn't needed, and both can guess wrong.

The source is Lab 12 (Inference & Serving), scripts 05_semantic_cache.py and 06_cascade.py. The video ran the lab's own code once, live, on November 2, 2026: real calls to OpenAI's text-embedding-3-small, gpt-5.4-nano and gpt-5.4-mini, 216 calls for about 2¢. The numbers below are from that recording. The questions and tickets are made up, for a software company's help desk, and each has a known right answer. The lab's README reports its own earlier run; where it differs, it's noted. Chapter 32 timed how fast the API answers, and chapter 33 cut its cost with OpenAI's prompt cache, where the model still runs every time. Today, on a hit, it doesn't.

A binder of answers

Think of a help desk with a binder of answers already given. Same question again? Copy the answer out of the binder. A semantic cache is that binder, for a model. Semantic means it matches by meaning, not by exact words. Reusing a saved answer is a hit; on a hit, the model isn't called.

What does a hit save? The lab timed ten of each on "How do I reset my password?":

Every model call was slower than every hit. A miss pays for both: the lookup, then the model. (The README's run: 0.19 s and 1.72 s.)

Measuring meaning

A program can't read, so each question becomes an embedding: a long list of numbers (1,536 of them here). The model that makes them is trained so that similar meanings get similar numbers. Think of it as a spot on a map of meaning; questions about the same thing land close together. The similarity score (cosine similarity) says how close two spots are: 1 is the same spot.

"How do I reset my password?" and "I forgot my password, how can I change it?" mean the same and score 0.77.

Same meaning vs. look-alikes

The lab scored 12 pairs of "a question already answered" and "a new question". In 6, the new question means the same, so reusing the answer is right. In the other 6 it only looks alike ("cancel my subscription" vs. "renew my subscription"), so reusing the answer is wrong.

• Same meaning: 0.72 to 0.84.
• Look-alikes: 0.68 to 0.96: below that range, inside it, and above it.
• The highest of all, 0.96, is "Can I get a refund within 30 days?" vs. "Can I get a refund after 30 days?". One word flips the answer, and the texts stay almost identical.

An embedding measures how alike two texts are, not whether one answer fits both.

The cutoff

The cache needs a cutoff: at or above it, reuse the saved answer; below it, call the model. Drag it and watch both rows:

Every cutoff that reuses even one good pair also reuses the refund pair. The only cutoff with no wrong answers reuses nothing, so the cache saves nothing. Careful, though: these 12 pairs were picked to show the risk. Real traffic may have far more true repeats and fewer look-alikes; the way to know is to score your own questions.

A model cascade

The second shortcut. Back at the help desk, a junior agent answers first; when they aren't sure, they ask a senior colleague. A cascade does that with models: a small, cheap model answers every request, and if it's unsure, a bigger, pricier model answers instead. Here the small one is gpt-5.4-nano and the big one gpt-5.4-mini; call them nano and mini.

It differs from a router, which picks a model before anything answers. A cascade decides after the cheap model has answered, so it knows more, but a request sent up pays for two calls and takes about twice as long (one call was about 0.6 s for either model).

The task has one right answer, so it's easy to grade: give each support ticket a priority. 1 is urgent: a breach, an outage for many users or a whole team, lost data. 2 is one person completely blocked, or a billing error over $500. 3 is everything else. 24 tickets with known answers, each run 3 times:

• mini: 24 of 24, every run, at $0.142 per 1,000 tickets;
• nano: 22, 23 and 23 of 24, at $0.038 per 1,000. Mini costs 3.7× as much.

How sure was nano?

The API can return a logprob for each token: the log of how likely the model thought it was. Nano answers with one digit, so turned back into a plain probability, that's its confidence: 0.89 means about 89% sure.

• 21 of the 24 answers were more than 95% sure, and right, in every run.
• "Please remove my former employee's access" (priority 3): nano was only about half sure (0.45–0.53), every run, and said 2 in one run of three.
• "Nobody on our team can upload files" (an outage, so priority 1): nano said 2, every run, about 0.89–0.91 sure. Wrong, but confident.
• "3 months of project history disappeared" (priority 1): right, and just as sure, 0.89–0.90.

So confidence can't tell those last two apart. A cutoff high enough to catch the outage every run (above 0.912) also sends that right answer up to mini. (The README's run had nano's least-sure mistake at 0.41.)

Choosing the cascade's cutoff

Below the cutoff, the ticket goes to mini. To choose it fairly, the lab splits the tickets: 16 dev tickets to try cutoffs on, and 8 holdout tickets kept aside for one final check. Its rule: keep the cheapest cutoff whose cascade scores as well as mini on dev.

• On dev, nano got all 16 right, every run. So a cutoff of 0 already matched mini, and it's the cheapest: never ask mini.
• On holdout, nano alone got 6, 7 and 7 of 8. The upload outage was in the holdout, nano missed it every time, and with a cutoff of 0 it never went up.

Pick a run, then try the cutoffs on dev and on holdout:

On the holdout, in all three runs:

Any cutoff from 0.95 to 0.99 got all eight right, every run, for about half to two thirds of mini's cost. But that's known only by peeking at the holdout; choose the cutoff there and it's no longer a fair check. (The README's run put 0.99 at about three quarters of mini's cost.)

The lesson is about the dev set: to tune a cascade, it needs tickets the cheap model gets wrong. These 16 had none, so the tuning had nothing to learn from. And with 8 holdout tickets, one ticket is an eighth of the score. This is a demonstration, not a result.

Also in the lab

• A safety check on hits (untested here). The lab suggests asking nano, on each hit, whether the saved answer fully answers the new question. The video recorded it but didn't use it: nano wrote the saved answers with no product context, so they mostly ask "which service do you mean?", and the check's yes/no says nothing about the design.
• Other ways to make a cache safer, from the lab's checklist answers: cache only question types where a near miss is harmless, require exact matches on key details (plan names, numbers, dates), and expire saved answers when the facts behind them change, such as a new refund policy.
• Free-text confidence. Logprobs work for a one-token answer. For free text, the lab's answers list sampling several answers and checking they agree, or a separate check, each at the cost of extra calls.

What to do

• Put a semantic cache only where a wrong reuse does no harm, or add checks on the details that flip an answer.
• Measure your repeat rate first, from your logs. A cache saves only on questions that truly repeat.
• Score your own pairs, including look-alikes, and count wrong reuses at each cutoff, not just savings.
• For a cascade, build a dev set that includes the cheap model's mistakes, mixed with ones it gets right, then tune the cutoff on it.
• Check the cutoff once on a holdout you didn't tune on, and use enough tickets that one ticket doesn't swing the score.
• Count both calls when a request is sent up: in the cost, and in the wait.

Caveats

• Picked pairs. The 12 pairs were chosen to show the risk; they say nothing about how often look-alikes occur in real traffic.
• Tiny samples. 12 pairs, 24 tickets, 3 runs. With 8 holdout tickets, one ticket is 12.5 percentage points.
• Made-up questions and tickets, for a software help desk, each with one right answer chosen by the lab.
• One day, one network. Times include the internet trip from one connection on November 2, 2026; costs are list prices.
• Confidence comes from one token. It works for a priority digit, not for a paragraph.

The checklist

1. Log real questions and measure how many truly repeat before building a cache.
2. Label pairs as "reuse is right" or "reuse is wrong", including look-alikes that differ by one detail.
3. Pick the cache cutoff by counting wrong answers, and weigh what one wrong answer costs.
4. Give saved answers an expiry, or clear them when the facts change.
5. For a cascade, label examples the cheap model gets wrong and put them in the dev set.
6. Choose the cutoff on dev, check it once on a holdout, and don't re-tune on the holdout.
7. Watch the confidently wrong: the cascade can't catch an answer the cheap model is sure of.

Practice: the cache and the cascade in code

Write cosine similarity, the cache's reuse decision, a confidence from a logprob, the cascade's cost and accuracy, and script 06's rule for choosing its cutoff. The tests use the recorded run: the 12 pair similarities (5 good and 3 wrong at 0.75, none of either at 0.96), nano's 22 of 24 and 3.7× cheaper, "never ask mini" from the dev set, 6 of 8 on holdout, and 0.95 and 0.99 at $0.074 and $0.092 per 1,000. The last test adds the outage to the dev set and watches the rule choose a real cutoff.

Quiz

Check your understanding

Try each question before opening its answer.

Next up: Chapter 35 covers speculative decoding: a small model guesses the next few words and a big one checks them, which made one model almost 1.5× faster, and at 4 bits made it slower.