Timeouts, circuit breakers and fallbacks

Retries fix a short blip (chapter 32). They can't fix an outage: when the model is down for 25 seconds, every attempt in those 25 seconds fails together. This chapter adds somewhere else to send the request (a fallback chain), measures what that costs in quality, adds a circuit breaker to stop calling the broken model, and finds the case where the breaker barely helped: a model that hangs instead of erroring.

The source is Lab 13 (Resilience), script 02_fallbacks.py, recorded in October 2026 against the real OpenAI API. The workload is the support-email triage from the fine-tuning labs: 120 made-up emails with known right answers, 2 a second, each sorted by gpt-5.4-mini into four fields (what the customer wants, the order number, the product, and how urgent it is). From 15 s to 40 s, mini is broken on purpose, under the real SDK: either every call gets a 503 at once (a busy tone) or every call hangs (a phone that rings with no answer). Everything else is a real API call. The lab's other scripts are in the next chapters: crash recovery in chapter 34, and checking outputs and broken search in chapter 35. Its deadline script (one time budget across a four-step pipeline) isn't in the video.

A fallback chain

A fallback chain tries the next option when one fails:

gpt-5.4-mini ──fails──▶ gpt-5.4-nano ──fails──▶ canned reply: "a person will get back to you"

The last step never fails and is always safe: it doesn't triage at all, it hands the email to a person. The lab's version is a few lines:

def fallback(options, canned=None):
    errors = []
    for name, fn in options:
        try:
            return {"served_by": name, "result": fn(), "errors": errors}
        except Exception as e:          # every failure moves to the next option
            errors.append((name, type(e).__name__))
            last = e
    if canned is not None:
        return {"served_by": "canned", "result": canned, "errors": errors}
    raise last

Against the busy tone, with mini alone, 48 emails got an error: almost every one of the 50 sent during the outage. Marcus's email (he wants to return a sleeping pad, sent at 17 s) got its error after 1.3 s. With the chain, all 48 became answers, from nano, and not one error. Marcus got his after 2.4 s. The canned reply was never needed: nano never failed.

A model gateway (LiteLLM, Portkey, OpenRouter, or a cloud provider's own) runs this same chain outside your code, often across providers.

The backup is worse

Marcus writes “URGENT” in capitals. The shop's rules say that word alone isn't enough: he has no deadline and no safety risk, so the right urgency is normal. mini says normal. nano says high.

On the same 120 emails with no outage, all four fields right:

Answered by	All 4 right
gpt-5.4-mini	97% (92–99%)
gpt-5.4-nano	74% (66–81%)
canned reply	0%: it doesn't triage

Most of nano's misses (28 of 31) were the same field: how urgent. So an outage is a quality incident too, for as long as the backup is answering. Decide before the outage which requests can take the cheaper model's answer.

A circuit breaker

During the outage, every email still phones the broken model first, and the OpenAI SDK retries twice on its own: 148 calls to a model that's down. A circuit breaker sits in front of mini and counts failures, like the switch in a fuse box:

closed ──(5 failed emails in 10 s)──▶ open ──(5 s later)──▶ half-open ──(test call works)──▶ closed
                                        ▲                        │
                                        └───(test call fails)────┘

• Closed: calls go through.
• Open: calls fail at once with CircuitOpen, without touching the service. In the chain, the email goes straight to nano.
• Half-open: after the cooldown, one test call (a probe) goes through. If it works, the breaker closes; if not, it opens again for another 5 s.

An email counts as failed only after all its retries, and only for errors that can go away on their own (a timeout, a 5xx). A 400 is the caller's fault and doesn't count.

Against the busy tone, the breaker tripped 3.3 s into the outage. Waits were short either way, since an error comes back fast, but the typical wait during the outage still fell from 2.4 s to 1.0 s. The real win: the broken model got 30 calls instead of 148, and was left alone to recover.

When the model hangs

Now the other outage: mini picks up and says nothing. The client waits for its time limit, here 5 s, then gives up, and the SDK retries twice.

• With mini alone, Marcus's email failed after 16.3 s.
• With the fallback, nano answered him, after 18.0 s. More than half of the 50 emails sent during the outage (28) waited over 17 s.
• With the fallback and the breaker: 17.6 s. Almost the same.

The breaker did trip, at 33.4 s: with 6.6 s of the outage left. Most emails sent before the trip still waited about 17 s; only the ones sent after it got fast answers. Emails sent late in the outage and answered by mini waited it out: their last retry came after mini recovered.

Why the breaker was late

A breaker can only count failures that have happened. Here, one failure takes:

3 attempts × 5 s time limit  +  the SDK's backoff between them (0.5 s, then 1 s, minus jitter)  ≈ 16.3 s

The first failure arrives 16 s into the outage. The breaker needs five; emails arrive 2 a second, so the fifth lands about 2 s after the first: 15 + 2 + 16.3 ≈ 33.3 s. It opened at 33.4 s, and the fifth failure was most likely Marcus's own email. A breaker learns only as fast as failures arrive. The same arithmetic gives 18.3 s for the busy tone, where each attempt errors at once and only the backoff is left; the breaker opened at 18.3 s.

Change the time limit, the retries and the trip count, and see when the breaker would have tripped:

Fail fast

So mini was made to fail fast: a 3 s time limit and no retries. nano keeps its 5 s and its two retries, so the backup is still robust. Now a failure arrives in about 3 s, and the breaker tripped 4.5 s into the hang. Marcus waited 4.3 s instead of 17.6, and the typical wait during the outage fell from 17.2 s to 1.2 s.

All eight runs, email by email. Each row is one email, from sent to answered:

Outage	Code	mini / nano / error	All 4 right	Typical wait, outage	Marcus	Calls to mini, outage
busy tone	mini only	72 / 0 / 48	58%	1.4 s	error, 1.3 s	148
busy tone	fallback	72 / 48 / 0	93%	2.4 s	2.4 s	148
busy tone	fallback + breaker	60 / 60 / 0	89%	1.0 s	3.0 s	30
busy tone	fail fast + breaker	58 / 62 / 0	88%	1.0 s	1.1 s	10
hang	mini only	92 / 0 / 28	72%	16.2 s	error, 16.3 s	118
hang	fallback	92 / 28 / 0	93%	17.3 s	18.0 s	118
hang	fallback + breaker	76 / 44 / 0	91%	17.2 s	17.6 s	102
hang	fail fast + breaker	55 / 65 / 0	88%	1.2 s	4.3 s	12

The price of failing fast

• More emails on the weaker model: nano answered 65, not 44. Without failing fast, slow retries outlasted the outage and mini answered late but well. Some of nano's answers came even after mini had recovered, while the breaker was still open or testing (13 of the 14 that came after 40 s, before 46.7 s).
• A slower return: mini answered again 6.7 s after it recovered, against 2.7 s with the plain breaker.
• Three seconds is tight. One healthy call, sent just before the outage, was slow, got cut off at 3 s, and counted as one of the five failures. Without it, the fifth real failure was Marcus's, at 20.0 s.
• Quality: across all 120 emails, all 4 right went from 91% to 88%. On a test this small that could be chance (p = 0.51).

So pick per request type: 4 s answers from the backup, or 18 s answers from the main model.

Caveats

• One run per setting, 120 emails, one injected outage. The outage starts and ends sharply; real ones flap, slow down gradually, or hit some requests and not others.
• The failures are injected at the HTTP layer, under the real SDK, so retries, the chain and the breaker all see real exceptions. Everything not broken on purpose is a real call to the API.
• The emails are made up, with known right answers, so quality can be scored. A real shop's mix of urgent emails decides how much the backup costs.
• The timing arithmetic is an average. The SDK's backoff has jitter, and calls in flight when the outage starts finish at different times; it matched the measured trips to within half a second.
• Not tested here: a gateway, hedged requests (send a second request if the first is slow), and a breaker that treats rate limits (429) differently from outages. They're the lab's exercise ideas.

The checklist

1. Give every model call a fallback chain ending in a reply that's always safe.
2. Measure the backup's quality before you need it, on the same items as the main model.
3. Choose which requests never take the backup (a safety report, a double charge): queue them for the main model or a person, and tell the customer.
4. Set every time limit from what the customer can wait. The OpenAI SDK's default is 600 s per attempt, with 2 retries.
5. Put a breaker in front of a service that can go down, counting only errors that can go away on their own.
6. In front of a breaker, fail fast: a short limit and no retries on the main model, retries on the backup. Then the breaker hears about trouble in seconds.
7. Test with both kinds of outage, an instant error and a hang. A design that works for one can fail the other.

Practice: build the chain and the breaker

Write the lab's fallback chain and circuit breaker (closed, open, half-open, driven by a fake clock), then the arithmetic of why it tripped late: how long one failure takes to be counted, and when the fifth one lands. The last test checks the recorded waits from the hang outage: a typical 17.2 s with the breaker, 1.2 s failing fast, and Marcus's 18.0, 17.6 and 4.3 s.

Quiz

Check your understanding

Try each question before opening its answer.

What does a fallback chain buy you, and what does it cost?

It keeps answering: against instant errors it turned 48 errors into answers. The cost is quality, for as long as the backup answers: with no outage, mini got 97% of emails fully right and nano 74%.

What does a circuit breaker add when the fallback already removes the errors?

It stops calling a service that's known to be down. Against instant errors, calls to the broken model fell from 148 to 30, and the typical wait from 2.4 s to 1.0 s, since emails skip the doomed first try.

Why did the breaker barely help when mini hung?

Each failure took about 16.3 s to be counted: 3 attempts × a 5 s time limit plus the SDK's backoff. The breaker needs 5 failures, so it opened at 33.4 s, with 6.6 s of the 25-second outage left. Marcus waited 17.6 s instead of 18.0.

What does "fail fast" mean here, and what did it change?

The main model gets a 3 s limit and no retries; the backup keeps its retries. A failure now arrives in about 3 s, the breaker tripped 4.5 s into the hang, Marcus waited 4.3 s, and the typical wait fell from 17.2 s to 1.2 s.

What did failing fast cost?

nano answered 65 emails instead of 44, mini came back 6.7 s after recovery instead of 2.7 s, and one slow but healthy call was cut off and counted as a failure. All 4 right fell from 91% to 88%, which on 120 emails could be chance (p = 0.51).

Which requests should never go to the backup model?

The ones where a wrong answer is expensive, such as a safety report or a double charge. Queue them for the main model or a person, and tell the customer it will take longer.

Next up: Chapter 34 kills an AI agent in the middle of a refund. Does the customer get paid twice?