Retries and jitter: when trying again makes it worse

A retry is calling again after a model call fails. Done right, it turns a passing blip into an answer. Done wrong, it multiplies load on a service that's already down, keeps customers waiting through an outage, or sends everyone back at the same instant so a rate limit turns into minutes of waiting. This chapter covers which failures to retry, what retries buy and what they don't, retries stacked on retries, and the waves 200 customers made, until jitter broke them up.

The source is Lab 13 (Resilience), script 01_retries.py and resilience.py, recorded in October 2026 with openai 3.35.1 on one laptop. The failures are injected on purpose: a small switch in the lab's own code fails chosen calls before they leave the laptop, and the rest go to the real model (gpt-5.4-nano). The rush of 200 customers runs against a local test server, not the real API. Its numbers are from the video's rerun of that exercise; the lab's own run is quoted alongside. Lab 13's later scripts are the next chapters: fallbacks and the circuit breaker (33), crash and resume (34), and checked outputs, broken search and deadlines (35). Its tricky-inputs suite isn't in this series.

What to retry

A failed call comes back with a number that says why. Retry only what can go away on its own:

A 400 fails the same way every time, so retrying only wastes time and money. A timeout needs care: "timed out" means "unknown". The server may still do the work, so retrying a write (a refund, an order) can do it twice. That's what idempotency keys are for (chapter 34).

The lab's is_retryable() is that table in code:

RETRYABLE_STATUS = {408, 409, 429, 500, 502, 503, 504}

def is_retryable(exc):
    if isinstance(exc, (openai.APIConnectionError, httpx.TransportError, TimeoutError)):
        return True  # includes openai.APITimeoutError
    return getattr(exc, "status_code", None) in RETRYABLE_STATUS

When retries work: coin flips

Retries work best when failures are random and independent. If each call fails with chance p, n tries all fail with chance pⁿ, so at least one gets through with chance 1 − pⁿ. The lab failed calls at random with a 503, 100 customers per cell:

At 50%, coin-flip math predicts 50%, 87.5% and 96.9%; every measured count is inside its 95% interval. Five failures in a row happen 1 time in 32, so about 3 in 100 should still fail; 4 did.

The price: at 50% failures, each customer cost 2.01 calls. These failures were injected on the laptop, but if they were real overload, that's twice the load on a server that's already struggling. The typical time barely moved (0.63–0.80 s); the waits land in the slow end.

When they don't: an outage

Real failures aren't always coin flips. The lab made every call fail for 6 seconds, as if the model were down, while customers kept writing: 80 calls, one every 0.25 s, 24 of them during the outage.

Every retry during those seconds fails too. Only the setup that kept trying until the outage was over got everyone through, and its slowest customer waited over 8 seconds. Retries outlast an outage only by making people wait as long as it lasts. For an outage, you need a backup: chapter 33.

Retries on top of retries

The OpenAI SDK, the library that places each call, already retries on its own: 2 retries, 3 tries in all, with backoff and jitter. It honors Retry-After and never retries a 400. The lab measured it against a service that always fails (none of these reached the API):

The video skips one result from the lab: the SDK's default read timeout is 600 s. A server that accepts the connection and never answers would hold a call for 10 minutes per try, half an hour in all (computed, not waited out). Set your own time limit on every call.

Now wrap that in your own loop. Every one of your tries becomes 3 calls inside the SDK:

client = OpenAI(max_retries=0, timeout=10)   # the SDK doesn't retry; our retry() does

Add a gateway that also retries 3 times, and one user message sends 3 × 3 × 3 = 27 calls with default-ish settings. Retry in one place, usually the layer closest to the user, which knows how long they can wait, and turn the others off.

The rush: backoff without jitter

Exponential backoff waits longer after each failure, to give a busy server room to recover: here 0.5 s, 1, 2, 4, then 8 s at most. The lab's test:

• 200 customers send one call each at the same moment;
• a local test server lets in 20 calls a second, with at most 5 spare saved up, and turns the rest away with a 429;
• each customer retries up to 30 times with that backoff. The fastest possible finish is 10 s (200 at 20 a second).

Almost everyone is turned away at once, so almost everyone waits the same 0.5 s and calls back at the same moment. And again, and again: the calls arrive in waves.

One wave, from the server's log: at 15 s, 113 customers called within 0.37 s, and 12 got in. Why only 12? The server can't save up its quiet seconds: it holds at most 5 spare, and adds 20 a second, so in 0.37 s it had room for about 5 + 20 × 0.37 ≈ 12. The other 101 waited 8 s; for 7.8 s nobody called, with the server sitting idle; then they all called back together. Wave after wave, the last customer got an answer after 152 seconds, after 1,643 calls.

Jitter: a random wait

Jitter makes each wait random. With full jitter, each customer waits anywhere from 0 up to the backoff:

def delay(n, base=0.5, cap=8.0):
    return random.uniform(0, min(cap, base * 2 ** n))   # full jitter

Same backoff, same server, same 200 customers: the last answer came after 12.7 s, with fewer calls (987). The calls spread out, so the server stays busy instead of idle.

Waves depend on which customers win each race, which depends on thread timing, so the lab's run and the video's three reruns differ: last answer 192.1, 152.0, 144.1 and 144.0 s without jitter, 13.9, 12.7, 13.1 and 12.4 s with. Jitter won every pairing by more than 10×. The lab's own run (192 s, 2,640 calls vs 13.9 s, 984) is the README's table.

When the server says when

Some servers say when to call back: a Retry-After header, "try again in 1 second". Obey it: wait at least that long. But obeying it doesn't break the waves. Every customer waits the longer of the server's 1 s and the backoff, so the waits are still the same for everyone, and they still return together: 160.5 s. The server's wait plus a random extra of up to 1 s, with the jittered backoff: 17.1 s.

wait = max(delay(n), retry_after)   # Retry-After is a floor
wait += random.uniform(0, 1)        # and still add jitter

Switch between the four setups. Each bar is one second; green calls got in, red were turned away:

Caveats

• Injected failures. The coin flips, the outage and the stacked retries fail calls on the laptop before they leave it, so the "load on the server" they'd cause is an inference, not a measurement.
• A test server, not the real API. The rush runs 200 simulated customers (threads) against a local server set to 20 a second with 5 spare. A real provider's limits, and its other customers, are different.
• One laptop, a few runs. The waves vary with thread timing: 144–192 s across 4 runs. The jitter results were steady (12–14 s).
• Full jitter isn't the only kind. "Equal" jitter (half the backoff, plus a random half) guarantees a minimum wait, which is kinder to a service that needs time to recover. Both beat no jitter by far; the lab measured only full jitter.
• Retries add load. If a service fails because it's overloaded, retrying at 2 calls per customer keeps it down. A circuit breaker stops adding load; that's chapter 33.

The checklist

1. Retry only what can pass: 429, 5xx, timeouts, dropped connections. Raise a 400, 401 or 403 at once.
2. Treat a timed-out write as "unknown": retry it only with an idempotency key.
3. Retry in one layer. Count the tries in your code, the SDK and any gateway; turn all but one off (max_retries=0).
4. Set a time limit on every call. The SDK's default is 600 s per try.
5. Back off exponentially, with a cap, and jitter every wait, including waits the server asked for.
6. Obey Retry-After as a floor, not as the whole wait.
7. Don't count on retries for an outage. Bound them with a deadline, and plan a fallback.

Practice: retry, and count

Write the lab's pieces from scratch: which failures to retry, the backoff schedule with full and equal jitter, the coin-flip prediction and its Wilson interval, how many calls stacked layers send to a dead service, a retry loop that honors Retry-After, and the test server's token bucket. The tests use the recorded run: 57 / 87 / 96 of 100 inside their intervals, 15 calls for 5 × 3 tries, and the 113-call wave at 15 s, replayed through the bucket, letting exactly 12 in.

Quiz

Check your understanding

Try each question before opening its answer.

Next up: Chapter 33: the main model goes down for 25 seconds. A backup model keeps answering, and a circuit breaker stops calling a dead one, but only as fast as failures arrive.