Replays: the model, chance, or the page?

An auditor points at one answer the assistant gave last month, and asks about it. Asked again, the assistant says something else. This chapter works out why, with three suspects (the model, chance, and the documents), and what an audit log has to keep so you can tell them apart.

The source is Lab 16 (Compliance), script 04_versions_replay.py, recorded in October 2026; the video uses the recorded run and the answers behind it, no new calls. The company, Lantern Outfitters, its people and documents are made up; every answer is a recorded call to gpt-5.4-nano. The edit that turns 30 days into 45 was made by the lab on purpose, with two others, and the logged and changed answers ran the same day. This is the last chapter on compliance: chapter 49 covered who may see what, chapter 50 which requests to refuse, and chapter 51 a log nobody can quietly change. The lab's scripts 05 (explanations) and 06 (the evidence pack) aren't in the video; their numbers are in "Also in the lab" below.

The audit log entry

The assistant answers employees' questions: it searches the company documents the person may read, and a model writes the answer from the pages it finds. Every answer goes into an audit log: who asked what, and what came back. The lab's entry keeps more than that, because "what came back" isn't enough to explain it later:

Asked the same question today, with the same model, the assistant says 45 days. Step through the suspects:

Suspect 1: the model

The code asks for gpt-5.4-nano. That's a name, like ordering "the house wine": the restaurant can change which wine it is, and a model provider can point a name at a new version. But every reply says which dated version answered.

In the run that produced the log, all 3,008 calls asked for a name and got a dated version back: 2,256 gpt-5.4-nano calls named gpt-5.4-nano-2026-03-17, and 752 gpt-5.4-mini calls named gpt-5.4-mini-2026-03-17. Across the whole lab, all 2,950 gpt-5.4-nano replies named the same version. The field for the provider's backend build (system_fingerprint) was empty on these models, so the dated name is the only version you get.

Ruled out: the version that answered Ana is the version answering today. Log the name from the reply, not the request: when the provider moves the name, last month's entries still say what answered them.

Suspect 2: chance

A model picks its words with a little randomness. To see how much, the lab took 60 logged requests and asked each one 5 more times, with the same model, the same prompt and the same pages: 300 replays.

The tent warranty question came back in six different wordings, the logged one and five replays, and every one said 3 years. The returns question, replayed for three people, said 30 days all 15 times.

Ruled out: chance changed the words, never the fact. But notice what that means for the log: 177 of 300 replays used different words from what the user was given. Asking again tells you the fact; the exact words exist only in the log. A replay checks the outcome; it can't recreate the answer, so the log stores the answer.

Suspect 3: the documents

Ana's entry kept a fingerprint of each page the model was shown: a short code computed from the page. The lab's is a SHA-256 hash of the page's title, text and who may read it, cut to 16 characters:

# audit.py
def doc_hash(doc: dict) -> str:
    return hashlib.sha256(canonical({"title": doc["title"], "text": doc["text"], "groups": doc["groups"]})).hexdigest()[:16]

Change one word and the whole code changes. Today's fingerprint of the returns page doesn't match the logged one. The page was edited: "within 30 days of delivery" became "within 45 days of delivery". The other three pages Ana was shown still match. Try it on the page itself:

Found it: the model was the same; the page wasn't. Without the fingerprints, "the assistant now says 45 days" looks like the model changed its mind. With them, the log points at the page.

A fingerprint shows that a page changed, not what it said before. To show an auditor the old text, keep old versions of the pages too, or an index version that names them.

How many answers three edits touched

The lab edited three pages: returns (30 to 45 days), parental leave (16 to 20 weeks), and the holiday closure (December 23 to December 22). Then it checked every logged answer's fingerprints against today's pages:

• 152 of 376 logged answers had been shown at least one edited page.
• Asked again, 22 gave the new, edited fact: returns 8, parental leave 7, holidays 7.
• Every one of the 22 was a question about an edited fact. The other 130 were shown an edited page but asked about something else. Two of them did ask about an edited fact: Theo, the contractor, who can't read the leave or holiday pages, and said he didn't have the information then and now.

That's the use of fingerprints at scale: when a page changes, you can list every logged answer that was built on the old version, before anyone asks.

Also in the lab

These parts of Lab 16 aren't in the video. From the lab's README and result files (its own run, October 2026):

• Temperature 0 on a different model. gpt-5.4-nano was run at its default setting, as everywhere in the lab. Replayed on gpt-4o-mini at temperature 0, 293 of 300 replays matched its own first answer (98%) and 56 of 60 requests repeated all 5 times; the fact held 300 of 300. More repeatable, not guaranteed, and it's a different model too.
• Explanations (script 05). 80 requests, 40 answered and 40 refused, each with three explanations: a template filled from the record ("Based on: …"), the model's own, and one written by gpt-5.4-mini from the full record, including what was filtered out. Removing the source an explanation named lost the fact in 38 of 39 (template), 39 of 40 (the model's own) and 11 of 12 (from the record), a counterfactual test. 28 of the 40 from-record explanations named no document at all, so that test can't check them.
• Refusals that hint. No explanation named a withheld document or gave away its contents. But 2 of the 40 from-record explanations hinted: "without access to any more detailed private records" for "Why is Daniel Brooks out of office?". The template and the model's own explanation can't do that: neither saw what was filtered out.
• An AI judge's clarity scores (gpt-5.4-mini, 1–5, no people): template 1.49, the model's own 4.96, from the record 4.72, at 7, 37 and 41 words on average. The judge marked down the one kind that can't make up a source.
• The evidence pack (script 06). Ten safeguards rebuilt from the result files, each mapped to SOC 2, GDPR and (as it would apply to patient records) HIPAA: 7 pass, 1 partial (explanations: no human review), 2 gaps (the search step isn't routed by data region; audit metadata stays readable after an erasure). Then the questions no test answers, such as the provider's data processing agreement, key management and retention. It's evidence for an auditor, not legal advice.

What to do

• Log the dated model version from the reply, with the prompt version, the policy verdict and its version.
• Log the answer itself. You can't get the same words back by asking again.
• Fingerprint every page shown, and keep old versions of pages if you'll need to show what they said.
• Replay to check the outcome, not the text: the fact, the citations, the refusal.
• When a page changes, list the logged answers built on it, by fingerprint, before anyone asks.
• When an answer changes, check the suspects in order: the model version, chance (replay), the pages, then the prompt and the policy versions.

Caveats

• Made-up company, one day. The documents, people and edits are the lab's; the logged answers and the replays after the edit ran the same day, so the model couldn't have moved in between. In real use, months pass and the name may well move.
• 60 requests replayed. 123 of 300 is about 41% (36–47%); a different sample, or a longer answer style, would match less often.
• The fact check is a word match. "Same fact" means the expected phrase was there, or wasn't, in both. 26 of the 60 requests never gave the expected fact at all (refusals and nearby answers), so they count as the same every time.
• One small model. Other models, and settings like temperature, repeat more or less; the lesson, store the answer, holds either way.
• A fingerprint isn't the old page. It proves a change, not its content.

The checklist

1. Record the reply's model name, not the requested one, on every call.
2. Record the prompt version and the policy version with every answer.
3. Store the answer text in the log.
4. Fingerprint each page shown, and version the pages.
5. Keep a replay script that checks the fact, citations and refusals of logged answers.
6. On every document change, list the affected logged answers.
7. On every model version change, rerun your regression set before trusting it.

Practice: three suspects in code

Write the lab's page fingerprint, count what the replays repeat, find the logged answers an edit touched, and diagnose Ana's answer. The tests use the returns page as logged, the 60 replayed requests, the pages all 376 logged answers were shown, and the recorded answers after the edits, and expect the lab's numbers: 03e25881f6a16970 then eb2aed3366582a2c, 123 and 14 and 300, 152, and 22 (8, 7, 7).

Quiz

Check your understanding

Try each question before opening its answer.

Next up: That's the end of the Compliance section. Chapter 53 starts Governance with an inventory of every AI system: what AI the company runs, who owns each one, and what every source missed.