Key takeaways
- ✓"Timed out" doesn't mean it didn't happen. The harness waited 1 s; the payment provider took 2 s and kept working after the harness hung up. The model heard "timed out", asked again, and one torn $79.50 pad was refunded twice: $159, in 3 of 3 runs.
- ✓An idempotency key lets the shop skip a write it has already seen. A key from the tool call's ID didn't help: the model's retry is a new call with a new ID, still $159 in 3 of 3. A key from the call's content (the tool and its inputs) gave one refund in 3 of 3: the shop refused the copy while the first was still running.
- ✓Even then, in 2 of 3 runs the agent handed the refund to a person while it was still going through. A timeout means unknown: wait for the call to finish, then check, or retry with a key that stays the same.
- ✓Checkpoints save the run after every step, but a crash can land after a write and before its save. Crashed and restarted, 8 tasks sent their write again with no key: the shop's own rules refused 6, and 2 happened twice (the second pad, an address change). With the call-ID key, the replay carried its saved ID and was skipped: 8 of 8 passed.
- ✓Refunds over $100 and every cancellation pause the run until a person answers, on the same save-stop-resume machinery as a crash. 8 writes waited across 30 tasks; the person said no 3 times, to the refunds the agent made against the rules before, and 28 of 30 passed.
The last chapter fixed the agent by changing the code around the model. This one is about that code: the harness, which runs the agent's tool calls for it. Think of the model as a manager and the harness as an assistant who makes the phone calls: the manager says "call the bank and refund this", and the assistant dials, waits, and reports back. This chapter covers what the harness must do when a tool is slow, when the program crashes, and when an action is too risky to leave to the agent.
The code is Lab 09's 03_harness.py, Exercises 2, 4 and 5, on the same made-up shop and tasks as the last chapter, with gpt-5.4-nano. Every run replays the model's real recorded answers from the lab's cache (no new model calls), and each replay matches the lab's recorded result. The slow payment provider and the crash are both set in code on purpose, and the timings are real: the provider sleeps 2.0 s per refund, and the harness waits 1.0 s. The lab's other two harness exercises, streaming and how quickly a stop button stops the loop, are live timings and aren't covered here.
What the harness does
- Runs each tool call with a time limit, so one slow tool can't hang the agent.
- Sends an idempotency key with every write. A write is any call that changes something: a refund, a cancellation, an address change.
- Saves the run after every step (a checkpoint), so a restarted run can continue.
- Pauses risky writes until a person answers.
Timed out doesn't mean it didn't happen
"One of the two sleeping pads from my tent order arrived torn. Please refund that pad." One pad: $79.50.
- The agent calls
issue_refund. The request reaches the payment provider, which takes 2 seconds. - At 1 second, the harness stops waiting and tells the model: "issue_refund timed out after 1s."
- Stopping waiting didn't cancel the request. The provider kept working, and at 2 seconds the refund went through.
- The model only heard "timed out", so it asked again. The same thing happened, and a second refund went through.
- It handed the case to a person, "to complete the refund".
The shop's database: two refunds, $159, in all 3 runs. Neither call said it worked. A time limit can't stop the tool: Python can't kill the thread running it, and a payment provider can't un-send a request. So "timed out" means you don't know.
T07 · “This is dana.whitfield@example.com. One of the two sleeping pads from my tent order arrived torn. Please refund that pad.”
The payment provider takes 2 s per refund; the harness waits 1 s for any tool call.
The refund calls in run 1, and the key each one carried
key: t07-0:…Bnkgcu33
→ {"error": "issue_refund timed out after 1s."}
key: t07-0:…uNAMsnKh
→ {"error": "issue_refund timed out after 1s."}
Each tool call gets a fresh ID, so the retry carries a new key. To the shop it's a new refund.
Which key?
The usual fix is an idempotency key. Think of an order number on a form: if the clerk has already handled that number, they don't process the form again. Every write carries a key, and if the shop has seen the key, it returns the first result and changes nothing. If the first write with that key is still running, the shop refuses the copy.
But which ID makes a good key? The model gives every tool call its own ID, so that's the obvious choice.
| The key | What the agent saw | Refunds | Task |
|---|---|---|---|
| None | timed out → timed out | 2 ($159), 3 of 3 runs | Failed |
| The tool call's ID | timed out → timed out | 2 ($159), 3 of 3 runs | Failed |
| The call's content: the tool and its inputs | timed out → still being processed | 1 ($79.50), 3 of 3 runs | Passed |
- The call's ID: each call gets a fresh ID, like a ticket number. Ask twice, and you get two tickets. The retry (
…uNAMsnKh, after…Bnkgcu33) carried a new key, so to the shop it was a new refund. - The call's content: both calls asked for the same thing (the same order, item, amount and reason), so both got the same key. The shop answered the copy "An identical request is still being processed. Check before retrying."
One more thing from the content-key runs: in 2 of the 3, the agent checked for refunds too early (none yet), then handed the refund to a person while the first one was still going through. If the person did it too, that's two refunds again. So wait for the call to finish, then check, or retry with a key that stays the same.
Crash and resume
Now a different failure: the program crashes, on purpose, right after a write. The harness saves the run after every step, and a restarted run continues from the last save. But the crash came after the write and before the save, so the restarted run doesn't know the write happened. It sends it again.
| 8 crashed tasks | What the shop did with the repeated write | Passed |
|---|---|---|
| No key | Refused 6 by its own rules ("refund more than is left", "can't cancel a cancelled order"); 2 happened twice | 6 of 8 |
| Key = the tool call's ID | Skipped all 8 as duplicates | 8 of 8 |
The two that happened twice: the torn-pad order had two pads, so a second $79.50 refund still fit; and an address change can always run again. The shop's own rules are a second line of defense, but don't count on having them.
Why does the call-ID key work now, and not before? After a crash, the harness replays the saved call, with its saved ID, so the key matches. After a timeout, the model made a brand-new call, with a new ID.
By how it's built, a content key would block the crash replay too (this wasn't tested). But it can also block a real repeat: say a customer has two torn pads, and the agent refunds them one at a time with the same details. The second looks like a copy. The lab's answer is to scope the key to the run and the item, or have the system issue an ID for each action when it's planned. So pick the key for the repeat you fear.
Ask a person
Last: writes too risky to leave to the agent. Refunds up to $100 go straight through. Bigger refunds, and every cancellation, pause the run until a person answers. Pausing works like a crash, on purpose: the run saves a checkpoint, stops with needs_human, and resumes with the person's answer.
In this test the person is a script: it approves only the kind of action each task expects. Across 30 tasks, 8 writes waited for an answer, and it said no 3 times:
| Waited for a person | Answer |
|---|---|
| $329 tent (refunded against the rules in the last chapter) | Denied |
| $249 sleeping bag (likewise) | Denied |
| $249 sleeping bag bought more than 30 days ago | Denied |
| Two refunds of $119 and $175, three cancellations | Approved |
28 of 30 tasks passed. Without the check, the agent would have sent those three refunds itself. The two misses: after the 30-day denial, the reply didn't tell the customer about the 30-day limit; and the discount request from the last chapter, which should go to a person, never reached the check, since only refunds and cancellations ask first.
The checklist
- Treat a timeout as unknown. Wait for the call to finish and check, or retry with a key that stays the same.
- If the model might retry a write, key it on what the write does (the tool and its inputs, scoped to the run).
- Checkpoint every step, and key each write on its call's ID, so a restarted run's replay is skipped.
- Give tools their own rules too, like "never refund more than was paid", as a second line of defense.
- Put a person in front of the writes that are too risky to take back, and check that the reply explains a "no".
Practice: keys and the shop
Write the harness's key, the shop's duplicate and still-running checks, and the approval rule. The tests replay the recorded timeline with the model's real call IDs and inputs: $159 with no key or a call-ID key, one refund with a content key, the crash replay skipped, a real repeat blocked by a content key, and the 8 writes that waited for a person.
Quiz
1.The refund call timed out after 1 second. What do you know about the refund?
2.Every write carried a key built from the tool call's ID. Why was the pad still refunded twice?
3.With a key built from the call's content, what happened to the retry?
4.The program crashed after a refund ran and before the checkpoint was saved. With no key, what happened when it restarted?
5.The call-ID key failed against the timeout but worked after the crash: 8 of 8. Why?
6.Refunds over $100 and cancellations waited for a person. The person said no 3 times. Which ones?
Check your understanding
Try each question before opening its answer.
What does "timed out" mean for a write?
That you don't know. The harness stopped waiting at 1 second, but the request had reached the payment provider, which finished at 2 seconds. The model asked again, and the pad was refunded twice.
What is an idempotency key?
An ID sent with every write. If the shop has already seen it, it returns the first result and changes nothing; if the first write with that key is still running, it refuses the copy.
Why didn't a key from the tool call's ID stop the double refund?
The model's retry was a new tool call with a new ID, so it carried a new key. A key from the call's content (the tool and its inputs) was the same for both calls, and gave one refund in 3 of 3 runs.
Why did the call-ID key work after a crash?
The restarted run replays the saved call with its saved ID, so the key matches and the shop skips it: 8 of 8 passed. With no key, the write was sent again in all 8, and 2 happened twice.
Why not always use a content key?
It also blocks a real repeat: two identical refunds in one run, like two torn pads refunded one at a time, look like one. Pick the key for the repeat you fear, or scope it to the item.
How does asking a person work, and what did it catch?
Refunds over $100 and cancellations pause the run: it saves a checkpoint, stops, and resumes with the answer. It denied the $329 tent, a $249 bag, and a bag past 30 days; 28 of 30 tasks passed.