Durable agents: kill it mid-refund, then resume it

When an agent dies in the middle of a task, something has to start it again. How it starts again decides whether the customer gets one refund or two. This chapter kills a support agent on purpose with SIGKILL, at chosen moments and at random ones, and checks every restart against a run that was never killed: were any writes done twice, and did the customer get the same answer?

The source is Lab 13 (Resilience), script 06_crash_resume.py, recorded in October 2026 on one laptop (MacBook Pro M4 Max, Python 3.14.7, DBOS 3.3.0 on SQLite). The agent is Lab 09's support agent for a made-up shop, Lantern Outfitters, on gpt-5.4-nano. Its replies were recorded once and are replayed as a stream, so a restart that reaches the same point gets the same reply, and "the same answer" has an exact meaning. What's tested is the harness around the model, not the API. The lab's other parts, failures of the model's service (retries in chapter 36, fallbacks and circuit breakers in chapter 37) and failures that raise no error (chapter 39), have their own chapters.

Steps, and the steps that change things

The agent works in steps. The model decides the next step, and a tool does it. Dana writes in: one of the two sleeping pads from her order arrived torn. Her task takes six model calls and five tools:

1. look up the customer, her orders, the order, and its refunds (four reads),
2. refund $79.50,
3. write the answer to Dana.

A step that changes the shop is a write: a refund, a cancellation, an address change, an email, a hand-off to a person. The shop runs in its own SQLite database, and each tool call is one transaction. So a refund the shop committed stays committed, whatever happens to the agent afterwards.

It's the moment you know from paying a bill online. You click Pay, the bank moves the money, and your power cuts out before you see "Paid". Did it go through? Click again and you might pay twice. The shop is the bank: its records survive the agent.

The test

A hook sends the agent's process SIGKILL (like kill -9): it stops mid-step, and no exception handler or cleanup runs. Then the parent starts the task again and compares the result with a run that was never killed. 14 support tasks, 12 of which commit a write, each killed at three chosen moments and three random ones, with five ways of recovering: 14 × 5 × 6 = 420 kill-and-resume runs. Every killed run finished; the question is how.

The three chosen moments:

• during the first reply: halfway through streaming the model's first reply, before anything happened in the shop,
• during the answer: halfway through streaming the final answer, after every tool ran,
• right after a write: inside the write tool, after the shop committed it and before the result reached the agent.

Start over, or keep a journal

The simplest recovery is to start over: run the whole task again from the top. Killed during the answer, that redoes every step, the refund included, though the shop already has one. Across the 14 tasks, starting over redid 75 model calls, made 5 extra writes (Dana's second refund among them), and gave the same answer in only 2 of 14.

The fix everyone reaches for is a journal (also called a checkpoint): after each step, save the conversation so far, like ticking off a checklist. The save is atomic (write a temporary file, then rename it), so a kill mid-save never leaves half a journal. On a restart, the agent reads the journal and carries on after the last tick. Killed during the answer, it redoes just the answer: one model call per run, 14 of 14 answers the same as a run never killed. The half-written answer was never saved, so it's thrown away and written again.

Pick a moment and a way to recover. The strip is Dana's task; the numbers below it are all the tasks killed at that moment:

The gap after a write

Now the worst moment. The shop has just committed Dana's refund. The agent hasn't saved it yet. Kill it right there.

The journal says the refund was asked for, not that it happened. So the restart sends it again. All 12 tasks that get as far as a write sent their write a second time. (The other 2 never commit one.)

How wide is that gap? A separate timing run with the lab's own shop and journal code (no model) measured it from the shop's commit to the saved journal: a median of 0.13 ms (at most 0.35 ms) over 200 runs. That's with the shop on the same machine. With a real payment service, the gap includes the reply's trip back over the network, so it's much wider. A trigger in the code kills the agent right inside it.

What the repeat did

• Done twice: 2. Dana ordered two pads, so a second $79.50 refund was still within what was left to refund. Owen's address was changed twice, to the same address.
• Refused by the shop's rules: 10. You can't refund more than what's left (7 refunds), or cancel an order that's already cancelled (3).
• Refused isn't harmless. When the repeat was refused, the conversation changed, and the answer with it. Grace's $68 refund went through once. On the restart, the agent tried it again, the shop refused twice, and the agent told her the refund was blocked and passed her to a person. In 4 of the 10 refusals the agent went on to make a different write (3 hand-offs to a person, 1 extra email).
• Same answer: 1 of 12. Only Owen's.

A note on counting: the lab's README table says the journal had "6 writes done twice". That column counts extra writes of any kind against the run never killed: the 2 repeats that went through plus the 4 different writes made after a refusal. The shop's own logs give the split above.

DBOS: the same gap

Maybe a hand-written journal is too simple. DBOS is a library for durable workflows: you mark functions as steps of a workflow, and it records each step's result in a database (SQLite here), so a restarted process finishes the workflow from its last recorded step. Temporal, Restate and Inngest do the same job.

@DBOS.step()
def tool_dbos_step(name: str, raw_args: str, keys: bool) -> dict:
    key = f"{DBOS.workflow_id}:{DBOS.step_id}" if keys else None
    result, _ = _dbos["shop"].call(name, raw_args, key)
    return result

Same 12 kills, same result as the journal, task by task. DBOS records a step after the step returns. The refund step never returned, so on recovery it ran again. That's the promise of durable execution: every step will finish, even if that means running it twice (at least once, not exactly once).

Idempotency keys

A good bank gives every payment a reference number. Send the same reference twice and it pays once. For a write, that number is an idempotency key: an ID sent along with the write. The service runs a key it hasn't seen and remembers the result; the same key again just gets that first result back.

The key must stay the same when the step is repeated, so build it from which run and which step:

• the journal: run id + tool call id, such as T07:call_YUjP…,
• DBOS: workflow id + step number, such as T07:10.

A new random ID made inside the step would be new after the restart too, and look like a brand-new refund.

With keys, on the journal and on DBOS: 0 writes done twice, 12 of 12 answers the same, and no model call redone. The journal keeps the work; the key makes a repeat harmless. Neither does the other's job.

If a service doesn't accept keys, the lab's answer key suggests checking before writing (does a refund for this order and item already exist?) and recording the intent before the call, so a resumed run can look the result up. Don't count on the service's own rules to catch it.

Random kills miss it

The lab also killed the agent at random moments: 3 per task per strategy, 210 in all, each landing anywhere between 5% and 95% of the run.

In the replay, each tool pauses 0.15 s, like a real service would, and the model's reply streams in. So most of a run is waiting, and 176 of the 210 kills landed while the agent waited on the model or a tool (8 of them inside the refund tool, before its commit). With a journal or DBOS, no random kill repeated a write: 42 of 42 answers the same for each saving strategy. Starting over made 3 extra writes and matched 32 of 42.

None hit the gap. At 0.13 ms in a 2-second run, you'd expect about one hit in 13,000 kills. That's why crash bugs survive random testing: aim the kill at the gap on purpose.

Random kills still earn their place. In the lab's first run, one landed between two tool calls the model had asked for at once, after the first result was saved. The resume looked for unfinished calls only in the last message (which was that first result), skipped the second call, and the API rejected the conversation: a tool call with no result. Lab 09's harness had the same bug, now fixed in both. No chosen kill point would have found it.

Half an answer

Killed while the answer streamed, every strategy threw the half-written answer away and asked the model again, and a user watching would see the answer start over. That's only safe because nothing acted on the half answer. Don't store, send or act on model output until it's complete.

Caveats

• A made-up shop, recorded replies, kills on purpose. 14 tasks on one laptop; the replay makes "same answer" exact, and a live model would vary on its own.
• The gap was measured with the shop in the same process. 0.13 ms is the journal path with a local SQLite shop. A real payment service's gap includes a network round trip.
• The waits are staged. Each tool pauses a fixed 0.15 s, and the 2.1 s run is a replay. Where random kills land depends on that.
• The DBOS gap wasn't timed separately. Its outcome matched the journal task by task.
• One worker on SQLite. In production, DBOS or Temporal need a shared database (Postgres) and workers that can take over each other's runs; the lab didn't test that.
• The shop's rules did some of the work. Without them, more of the 10 refused repeats would have gone through.

The checklist

1. Give every write an idempotency key that stays the same when the step is repeated: run + step, never a fresh random ID.
2. Save progress after every step, atomically, with your own journal or a durable-workflow library. The journal keeps the work; the key makes a repeat harmless.
3. Know what your workflow engine promises. Steps run at least once. "Exactly once" holds only for services that deduplicate by key.
4. Kill the agent on purpose right after a write, not only at random, and compare with a run that was never killed: writes done twice, shop state, answer.
5. Keep random kills too. They find what you didn't think to aim at, like a crash between two parallel tool calls.
6. Don't act on half an answer. Store, send or act on output only once it's complete.
7. When a service has no keys, check before writing and record the intent before the call.

Practice: kill it, then resume it

Write the three pieces the chapter is about: a shop that runs each idempotency key once, a stable key from the run and the step, and a journaled runner that skips finished steps, saves after each one, and can be killed in the gap right after a write. Then the random-kill arithmetic. The tests replay the lab's cases: Dana refunded twice by starting over and by the journal, Grace refunded once and told it was blocked, the repeat made harmless by a stable key and not by a random one, and about one hit in 13,000 random kills.

Quiz

Check your understanding

Try each question before opening its answer.

Next up: Chapter 39 is about failures that never raise an error: an answer that breaks the rules, search that quietly returns the wrong pages, and a request that's too slow.