Silent failures: no error, still wrong

An AI tool can answer every request with no error and still be wrong. Nothing crashes, the logs are clean, and the answers look fine. This chapter breaks two tools on purpose to see what that looks like: a question-answering assistant whose search returns the wrong thing, and a reply writer whose replies break the rules in its instructions. It covers what each failure looks like to the person asking, the signal that gives it away, and what fixes it.

The source is Lab 13 (Resilience), scripts 04_chaos_rag.py (breaking search in a RAG pipeline) and 03_reask.py (checking replies and asking again), recorded in October 2026. gpt-5.4-nano answers in both, and gpt-5.4-mini writes the questions and grades the answers. The articles, questions and support emails are test data. Retries, fallbacks and crashes, the failures you can see, were chapters 32 to 34. The lab's scripts on deadlines and tricky inputs aren't covered here.

How the assistant answers

The assistant is a RAG pipeline (retrieval-augmented generation): it looks things up, then answers from what it found. The documents are 7 short articles about AI engineering, split into 28 paragraphs. For each question:

1. Search returns the 3 paragraphs closest to the question, each with a match score: the higher, the closer search thinks it is.
2. The model answers from those paragraphs only. Its instructions say: cite the source of every claim, and "if the context does not contain the answer, reply with exactly: I don't know based on the provided documents."

The test set has 72 questions: 56 that the articles answer (written by gpt-5.4-mini, two per paragraph, each with a short gold answer and its source) and 16 written by hand that they don't ("How much does Pinecone cost per month?"). mini grades each answer against the gold answer: correct, refused or wrong.

With search working, 55 of the 56 were right, and all 16 unanswerable questions got "I don't know".

Five ways to break search

The lab wraps search in a ChaosRetriever that breaks it one way at a time, the same way on every run. Two pipelines get the same broken search:

• Naive: whatever search returns goes into the prompt, and an exception reaches the person as an error.
• Defended: a timeout is retried once and then answered with "Search is unavailable right now"; no results means "I couldn't find anything about that", without calling the model; and the prompt adds a line saying the passages may be unrelated.

The video walks through three of these: working, wrong passages and the timeout. Pick a break and a pipeline to see what the assistant did with one question, and with all 56:

Wrong passages, confident scores

The quiet one is wrong passages: for every question, search returns paragraphs from other articles, and each carries the score of the right paragraph it replaced, like an index that got mixed up. For "What is an AI agent?", the top paragraph is about prompt engineering instead of agents, with the same 0.825.

Watch the usual warning signs and nothing shows up. The scores look normal, so a minimum-score check passes. Nothing raises an error. Right answers fell from 55 to 3.

The worry was made-up answers, pieced together from the wrong paragraphs. That isn't what happened, for this model and prompt:

• "I don't know": 51 of 56. With search working, once.
• Wrong: 2, pieced together from the wrong paragraphs.
• Right: 3, but none from the documents. The paragraph with the answer wasn't in what the model saw, so those came from its own training. A right answer the documents didn't supply passes an answer-only test while the pipeline's grounding is broken. To catch that, check retrieval itself: is the right paragraph found? (Chapter 8 measures exactly that.)

The alarm: count "I don't know"

So the broken search showed up as a jump in refusals, from 1 to 51 of 56. A head chef can't taste every dish, but she can count how often the kitchen says "can't make that". If that count jumps, the pantry broke.

That's a signal you can monitor in production, where there are no gold answers:

• Track the refusal rate, say per hour, against its normal level, and alert when it jumps.
• Run a canary: a few questions with known answers every few minutes, so a broken search fails a check you own.

Every break in the table shows up in the refusal count: empty, wrong, timeout (defended), cut short, best results lost. The refusal rate isn't a perfect alarm. The break that gave the most wrong answers was passages cut short: 5 wrong naive, 8 defended, because almost enough context invites a guess. Those still came with 27 to 30 refusals, so the alarm would still fire.

A timeout: the break that errors

One break does raise an error: search times out, and nothing comes back. As built, that error went straight to the person: every one of the 72 questions got an error page, including the 16 the articles can't answer.

The fix is a few lines of code in the pipeline, not the prompt:

# simplified from 04_chaos_rag.py
def defended(question, retriever):
    try:
        found = retry(lambda: retriever(question), attempts=2, backoff=Backoff(base=0.1))
    except (GaveUp, SearchTimeout):
        return "Search is unavailable right now, so I can't answer from the documents. Please try again in a minute."
    if not found:
        return "I couldn't find anything about that in the documents."
    return answer(question, found)   # with a prompt line: the passages may be unrelated

Error pages went from 72 to 0. An honest "unavailable" is better than an answer from the model's own knowledge, even a right one: the person can't tell a grounded answer from a guess, and an honest failure can be retried.

The extra prompt line ("the passages may be unrelated, check them before use") changed results by a few answers either way, within noise. The defenses in code mattered; the prompt line didn't.

Replies with four rules

The second tool is the support desk's reply writer. The same small model reads a customer's email, labels it (what it's about, the order number, the product, how urgent), and writes the reply, all as one JSON object. Its instructions give four rules for the reply:

1. At most 40 words.
2. In the customer's language.
3. Quote the order number, if there is one.
4. No placeholders, like "[Your Name]": the reply is sent as is.

The test set is 300 emails written by a model for testing, with known labels: 240 in English, 60 in French, German or Spanish. One of them, from Fatima: she got a water filter in the wrong color, order A-3235. The first reply reads fine, and came back with no error.

But code can count the words: 44, four over. The lab's validate() checks all four rules in code, using only the email and the reply. It needs no right answer and no second AI to judge, so it can run on every live reply, like a head chef weighing every plate before it leaves the kitchen. It returns each problem as a sentence worded to send back to the model.

Of 300 first replies:

The JSON never broke, and every reply to a French, German or Spanish email was in the customer's language. Almost every break was one rule: too long, and 113 of the 210 were five words over or less. The small model seems bad at counting words as it writes, like a cook who eyeballs a portion instead of weighing it. A rule in the prompt is a request, not a guarantee.

Try the check on Fatima's replies, or edit one and watch the rules:

Send it back, with a note

When the check finds a problem, the code sends the reply back with a note saying what was wrong:

That reply is not valid: reply has 44 words; the limit is 40. Reply again with only the corrected JSON.

Fatima's second reply: 34 words, and every rule kept. Does the note matter, or would asking again work as well? Up to two more tries for each of the 300 emails:

Without a note, a fresh try mostly made the same mistake: of the 210 replies that were too long, 170 were too long again on the next try (123 with the note). The note gained 30 points in its first round; a fresh sample gained 14. Re-asking with the note didn't hurt the labels either: all four were right on 72% of emails before and after.

Two costs come with it:

• Each try is another call. One call per email costs $0.28 per 1,000; with re-asks, $0.64, more than double. On average each email also took twice as long (1.6 s → 3.2 s).
• One in five still broke a rule after two more tries (62 of 300). Those need a fallback, such as a person.

Fix the cause

When one rule breaks most of the time, re-asking treats the symptom. The lab tried two fixes for the cause.

A strict schema. The model answers in JSON, a simple text format with named fields. A strict schema is a setting on the AI service (OpenAI's structured outputs, strict: true) that forces that shape: the right fields, and values from a list. It checks the shape of the plate, not the portion: it can't easily say "at most 40 words". With the schema, 34% kept every rule, and 197 replies were still too long. The rules a schema can express never broke here. Adding re-asks on top reached 82%, about the same as re-asking alone.

A bigger model. gpt-5.4-mini, with no schema, kept every rule 95% of the time on its first try, at $0.89 per 1,000: about 3× nano's single call, and a bit more than nano with re-asks.

Pick a setup to compare:

Read why a reply is invalid before deciding that asking again will help. In a run that isn't in the video, the lab capped replies at 60 tokens: all 50 were cut off mid-answer, and two re-asks each still fixed none, because the limit cut off every new reply too.

What to do

• Watch how often the assistant says "I don't know". A jump means something upstream broke, even when nothing errored. Add a canary with known answers.
• Check retrieval, not only answers. A right answer from the model's own training hides broken grounding.
• When a step fails, give the person an honest message, never an error page and never a guess.
• Check every reply in code, and send it back with a note that names the rule and the value.
• When one rule breaks most of the time, fix the cause: a model that follows it, a rule the model can follow (fewer sentences, not fewer words), or enforcing it in code.

Caveats

• One model, one prompt. That nano refused instead of making things up with wrong passages is a result for this model and this instruction, not a law. Another model, or a prompt without the "say you don't know" rule, may guess.
• Test data. The questions and emails were written by models for testing. mini also graded the answers, so "right" and "wrong" are a model's judgement against short gold answers.
• The breaks are simulated. ChaosRetriever makes each break total: every question gets the same failure. Real outages are often partial, so the jump in refusals would be smaller.
• Small sets. 56 answerable questions and 300 emails give wide intervals: 79% is 74–84%.
• List prices. Costs are per 1,000 emails at the October 2026 list prices of the recorded calls.
• A valid reply can still be wrong. The check proves the rules, not that the reply is right. That needs evals on labeled data.

The checklist

1. Break each dependency on purpose: empty, wrong, slow, cut short. Watch what the person gets.
2. Count refusals per period and alert when the rate jumps from its normal level.
3. Run a canary of known questions, and check retrieval directly: is the right passage found?
4. Catch every error a step can raise, retry once if it's worth it, then answer with an honest message.
5. Write the output rules as code, and check every reply before it's sent.
6. Re-ask with the problems, worded so the model can fix them, and cap the tries. Send what still fails to a fallback.
7. Look at which rule breaks. If one breaks most of the time, fix the cause and measure it again.

Practice: catch what doesn't error

Write the lab's checks: validate a reply in code and word the note, ask until valid (with and without the note), recognize refusals and raise the alarm when their rate jumps, and the defended pipeline that turns a search timeout into an honest message. The tests use Fatima's two recorded replies (44 and 34 words) and the 56 answers with wrong passages (51 refusals).

Quiz

Check your understanding

Try each question before opening its answer.

Next up: Chapter 36 starts on observability. Every failure here was broken on purpose, so we knew where to look; in production you don't, so the next chapters record each step of every request to find out afterward what happened and why.