Debate, voting, or a bigger model

The previous chapter (multi-agent systems, part 1) found that a team of agents splitting up the work didn't beat one agent. This one tries the other common use of extra agents: spending extra calls to double-check an answer. Think of a second opinion from another doctor. It helps when the second doctor knows something the first one doesn't. Two doctors who learned the same wrong fact will just agree with each other.

The scope is Lab 10's scripts 06 (debate), which the video covers, and 05 (code review), which was cut from the video but is in the lab and gets its own section at the end. Everything runs on Lab 09's made-up outdoor shop, with gpt-5.4-nano and gpt-5.4-mini, no thinking (reasoning_effort="none"). The numbers are the lab's recorded runs.

A debate that agreed on the wrong answer

A customer wants a refund on a $249 sleeping bag (case P05). It was delivered on 2026-09-01, and today is 2026-10-15. The policy:

• Refunds only within 30 days of delivery.
• A refund over $200 must go to a person.
• If the carrier marks a parcel lost, refund it in full.

Two agents debated it. Both wrote that the bag was "within 30 days" and "not over $200", and refunded it. Each read the other's answer; they still agreed. The judge agreed too. Five model calls, one wrong refund. It was 44 days, so the right answer is refuse: past 30 days there's no refund at all, so the $200 rule never comes into it.

Five ways to spend calls

The test: 40 made-up refund cases sitting on the edges of the policy (day 30 vs. day 31, exactly $200, a lost parcel, an item already refunded). Code works out the right answer for each one, so grading is exact: the right decision, and for a refund, the right amount to the cent.

Setup	What it does	Calls per decision	Right	Cost per 1,000 decisions
Ask once	one nano answer	1	183/240 = 76%	$0.15
Vote of 3	the most common of 3 nano answers	3	61/80 = 76%	$0.44
Vote of 5	the most common of 5	5	31/40 = 78%	$0.74
Debate + judge	2 nano agents answer, each reads the other's answer and tries again, a nano judge picks	5	95/120 = 79%	$0.76
Bigger model	one gpt-5.4-mini answer	1 (pricier)	102/120 = 85%	$0.50

How the rows were built: each case was asked of the small model 6 times. The votes and the debates reuse those 6 answers (two votes of 3 from answers 1–3 and 4–6, one vote of 5 from answers 1–5, three debates from the pairs 1–2, 3–4, 5–6). The bigger model was asked 3 times per case.

• Votes and debate were hardly better than asking once. 76 to 79%, with p from 0.53 to 1.00 against asking once.
• The bigger model scored best, but even its lead is borderline with 40 cases: p = 0.05 against asking once, 0.24 against the debate. It also cost about a third less than the debate.
• So count the cost, not just the accuracy. A debate is five calls, and it should be compared with what five calls buy elsewhere, not with one cheap call.

Where the errors are

The average hides where the mistakes sit. Put the cases on a line of days since delivery, with the 30-day window drawn in, and switch between the setups. Each dot is one recorded answer.

• Past 30 days, every setup was wrong most of the time. 7 cases: right 12% asking once, 7% and 14% for the votes, 14% for the debate, and 14% for the bigger model too. For a backpack delivered 60 days earlier, a typical answer read: "The order was delivered on 2026-08-16, and today is 2026-10-15, which is within 30 days."
• These cases were most of the errors. 37 of the 57 wrong single answers were past the window.
• The bigger model's clear win was a different kind of case: refunds over $200. The nano setups were right 64–72% of the time there, mostly by refunding a single item priced over $200 (the $249 bag delivered 4 days ago, a $229 backpack) instead of sending it to a person. The bigger model got all 18 right. It also got the lost $329 tent right (send it to a person) 3 of 3 times; asked once, nano refunded it every time.

Switch to the bigger model and look at P05, the cold-open bag: it said "escalate" twice. It knew $249 is over $200, but not that 44 days is past 30.

Inside the debates

A debate can only fix an answer if someone in it has the right one. Here's how the 120 debates started, from the two agents' first answers:

How it started	Debates	Ended right
Both gave the same right answer	88	88
Both gave the same wrong answer	24	1
They disagreed, one was right	7	6
They disagreed, both were wrong	1	0

• When they disagreed, the right one usually won: 6 of 7. Across the 240 revisions, a revision fixed a wrong answer 8 times and broke a right one once.
• But that rarely happened: 7 of 120 debates.
• 23 of the 25 wrong debates started with both agents giving the same wrong answer. Reading each other's answer didn't fix it: they're the same model, with the same blind spot. In the one exception, both refunded a lost $329 tent and the judge sent it to a person.

So a debate or a vote helps almost only when the answers disagree. To get disagreement, the checkers need something different: another model, other evidence, a different role. Two copies of the same model mostly share their mistakes.

Hand it the number

What was the mistake? Counting the days between two dates. That's easy for code. So code works out the count and writes it into the case, one line:

c["text"] = c["text"].replace(f"Delivered on {c['delivered_date']}.",
                              f"Delivered on {c['delivered_date']} ({c['days_since_delivery']} days ago).")

The same single nano call, nothing else changed:

Case text	Past 30 days (7 cases × 6)	All 40 cases (× 6)
Dates only	5/42 = 12%	183/240 = 76%
With "N days ago"	42/42 = 100%	217/240 = 90%

p < 0.001, at the same $0.14–0.15 per 1,000 decisions. One line of code did more than every extra-agent setup tried. Pick "Ask once + 'N days ago'" in the widget above to watch the right side of the line turn green.

What's left is the next blind spot: 16 of the 23 remaining errors are refunds over $200, plus the lost $329 tent (6) and one parcel that wasn't delivered. That's the rule the bigger model handled. The same reasoning applies: if code can work out "this refund is over $200", work it out and hand it over.

A reviewer agent is a classifier

Script 05 checks a different kind of work with a second agent: a reviewer that approves or rejects a code change. The lab has 34 patches for Lab 09's coding tasks, all of which pass the visible tests: 15 correct, 15 wrong in a plausible way, and 4 that cheat the tests (special-casing the test's input, or a fake equality from Lab 09's run). Each was reviewed 3 times by each model.

Reviewer	Wrong patches stopped	Cheating patches stopped	Correct patches rejected (false alarms)	Cost per 1,000 reviews
gpt-5.4-nano	35/45 = 78%	12/12	18/45 = 40%	$0.19
gpt-5.4-mini	45/45 = 100%	12/12	5/45 = 11%	$0.61

• Cheating was the easy case. Both models stopped every patch that gamed the tests.
• A partial fix looks like a fix. Nano approved the median that uses fixed indexes and the LRU cache fix that misses one case, 3 times out of 3 each.
• Nano also cried wolf: it rejected 40% of the correct fixes.
• Count note: two of nano's replies on wrong patches had no verdict. The lab counts those as stopped, so 35 of 45; it explicitly approved 10.
• The reviewer's model mattered more than having a reviewer.

The lab also tried a test-writer agent: only 28 of its 45 generated test files passed on the correct code. The ones that did caught 31 of 32 wrong or cheating patches. A generated test is a claim too: run it against a known-good version before you use it as a gate.

In the loop. Lab 09's coder alone, then with the nano reviewer able to send the work back twice (15 fixable tasks and one impossible task with contradictory tests, 3 runs each):

Setup	Fixable runs: accepted and right	Correct fixes blocked	Impossible task: cheating code accepted	Cost per 1,000 tasks
Coder alone	45/45	0	2/3	$0.99
Coder + reviewer	42/45	3	0/3	$1.89 (1.9×)

The coder alone already fixed everything it could, so the reviewer couldn't add fixes. What it did: it stopped the cheating code every time it saw it (in one of the three runs, the coder submitted nothing to review), and it blocked 3 correct fixes, two of them over edge cases the tests don't check (whitespace-only input, a width of 0 or less). A second agent pays off where "the tests pass" isn't the truth.

Caveats

• 40 made-up cases, chosen to sit on the policy's edges. That's good for finding blind spots, but the percentages aren't what a real refund queue would see.
• The setups aren't independent samples. The votes and debates reuse the same 6 nano answers per case, and the p-values treat rows as independent, so take the borderline ones (the bigger model's lead) as "maybe".
• Every setup was one model family with no thinking. A debate between different models, or with thinking turned on, wasn't tried; the lab lists it as worth measuring. Date arithmetic belongs in code either way.
• The video's and the lab's numbers agree. The lab prints the computed-days p-value as "p = 0.000"; it's below 0.001. The lab's README says the reviewer stopped the cheating code "every time"; in one of the three runs there was nothing to review, so "it was never accepted" is more exact.

The checklist

1. Look at where the errors are before adding agents. Split accuracy by kind of case; an average hides a blind spot.
2. If code can work something out (a day count, a total, a threshold), work it out and hand it to the model.
3. Compare checks at the same budget: a debate against a vote of 5 and one bigger-model call, with intervals, and report the cost.
4. Expect a vote or a debate to help only when the answers disagree. If every checker is the same model, they share its mistakes; get diversity from another model, other evidence or another role.
5. Measure a reviewer as a classifier on labelled examples: what it catches, and what it wrongly blocks.
6. Run a generated test on known-good code before trusting it as a gate.

Practice: the policy, the vote and the fix

Write the policy in code, the vote, the one-line fix (count the days and write them into the case), the debate's starting state, and a reviewer's two rates. The tests use the lab's recorded answers: the policy on all 40 cases, the past-window votes (1 of 14), 5 of 42 → 42 of 42 with the day count, the 120 debates (23 of 25 wrong ones started agreed and wrong), and the 05 reviewers' 35/45 and 18/45.

Quiz

Check your understanding

Try each question before opening its answer.

What are the three ways to check an answer with extra calls, and what does each cost here?

A vote (ask 3 or 5 times, keep the most common answer: $0.44 and $0.74 per 1,000 decisions), a debate (two agents answer, read each other's answer and try again, a judge picks: 5 calls, $0.76), and one call to a bigger model ($0.50). Asking the small model once cost $0.15.

Which did best, and how sure can you be?

The bigger model, gpt-5.4-mini: 85%, against 76–79% for the votes and the debate and 76% for asking once. With 40 cases its lead is borderline (p = 0.05 against asking once, 0.24 against the debate), and it cost about a third less than the debate.

Why couldn't voting or debate fix the past-window cases?

All the answers came from the same model with the same blind spot: it miscounted the days between two dates. 23 of the 25 wrong debates started with both agents giving the same wrong answer, and a vote of wrong answers is still wrong (1 of 14 votes of 3 was right past the window).

When does a debate help?

When the two agents disagree and one of them is right: the right one won 6 of 7 such debates. That happened in only 7 of 120 here, because two copies of the same model mostly agree, including on their mistakes.

What was the one-line fix, and what did it change?

Code wrote the day count into the case: "Delivered on 2026-09-01 (44 days ago)." The same single nano call went from 5 of 42 to 42 of 42 right on the past-window cases, and from 76% to 90% overall.

Which two numbers describe a reviewer agent?

Its catch rate (wrong work stopped; a miss ships a bug) and its false-alarm rate (correct work rejected; each costs another round, or correct work that never ships). The nano reviewer: 78% caught, 40% false alarms. The mini reviewer: 100% caught, 11% false alarms.