AI Writes the Code. Who Reviews It?

An agent can write a pull request in five minutes. Someone still has to review it, and that's now the slow part.

The popular fixes all sound reasonable: add an AI reviewer, auto-approve the low-risk PRs, let a second AI check the first one. I wanted to know which of them actually work, so I built a set of labs and measured.

The short answer:

• Run an AI reviewer and your automated checks on every PR.
• Have one person read every PR, and look at the AI's comments only after forming their own view.
• Skip the extra layers.

That saved about a quarter of review time, and no more bugs got through. The fancier pipeline I had planned did worse.

How I tested it

To grade a reviewer, you need to know where the bugs are. So I made them myself.

I took one TypeScript app and wrote 77 realistic changes to it. Then I made a buggy copy of each change, planting a bug that a hidden test proves is there. Some changes got more than one buggy copy. That gave me 250 pull requests: 173 with a bug and 77 without. Every one of them passes CI. 38 of the 173 bugs are serious ones, the kind that loses data or opens a security hole: deleting a sync also deletes where it sends data, or every webhook leaks part of its signing secret.

flowchart TB
  C["77 realistic changes"] --> B["173 PRs with a planted bug"]
  C --> K["77 clean twins"]
  B --> P["250 PRs, all with green CI"]
  K --> P
  P --> R["Reviewer: AI or person"]
  R --> M["Comments"]
  H["Answer key: a hidden test per bug"] --> S["Score"]
  M --> S
  class P accent
  class R warn

Now any reviewer, human or AI, can be scored on two things:

• How many bugs it found.
• How much noise it made: comments that weren't about a real bug.

You need both. A reviewer that flags every line finds every bug and is useless. That's why the clean PRs are there.

One thing to know up front: the "people" in these numbers are AI too. Real reviewers can't read the same 250 PRs several times without remembering them, so every "person" is an AI reviewer told to review like one: a careful reader (gpt-5.4-mini, set to think hard and read line by line) and, in one later test, a rushed one. Review minutes aren't timed either; they come from an assumed reading speed. More on what that means in What this doesn't prove.

One lesson came before any reviewing started: green CI doesn't mean the code is right. When I broke the code those PRs added in small ways on purpose, the app's own tests missed 18% of the breaks. About 3 in 4 of those were bugs a user would hit.

All the AI in these labs is small and cheap (gpt-5.4-nano and mini). The paid labs cost under $14 each, and four cost nothing.

What I found

An AI reviewer finds most bugs, but not the same ones every time

I tried three AI reviewers: a single prompt, an agent that can read the whole repo, and the open-source PR-Agent. Depending on the reviewer, they found 62% to 79% of the bugs. Run the same reviewer twice and the result changes.

The agent wasn't better overall than the simple prompt. But it caught a kind of bug the prompt missed in all three runs: where the API saved a field as default and a worker read it as defaultValue. The worker code wasn't in the PR, so only a reviewer that could look around the repo had a chance.

Having an AI double-check the AI can quietly lose bugs

AI reviewers write some comments that are simply wrong. So a natural idea is to add a filter: before a comment reaches the author, a second AI step checks it and deletes the ones it doesn't believe. I tried two filters:

• Argue: the model tries to talk each comment out of being a problem.
• Prove: the model has to write a test that shows the bug is real, plus a fix. The test is actually run: the comment survives only if the test fails on the PR and passes once the fix is applied.

In my first test, both filters looked free. They deleted some wrong comments (the share of correct comments rose from 71% to 74% and 76%), and not one bug the reviewer had found was lost.

Why did they look so good? Because of how I ran that test. To see how consistent the AI reviewer was, I had run it three times on every PR and kept all the comments.

Imagine a PR with one bug:

• In my test, the reviewer ran three times, and each run commented on the bug. So the bug had three comments pointing at it. The filter wrongly deleted one. Two were left, so the author still heard about the bug. Caught.
• In real use, you run the reviewer once. The bug gets one comment. If the filter wrongly deletes it, there's nothing left. The author never hears about the bug, and it ships.

So the filter wasn't free. My first test just couldn't see the cost, because the extra copies hid it.

It's like a spam filter that sometimes deletes a real email. If the email arrives three times, you never notice. If it arrives once, it's gone.

One more catch with the Prove filter: a test can only show what the code does, not whether that's wrong. The model happily "proved" 12 of 16 comments that were really just style preferences.

Linters and extra tests catch less than you'd hope

I turned on everything: stricter types, linting, Semgrep, secret scanning, and property, contract and browser tests. Together they caught 11% of the bugs, mostly broken conventions and code that can't run.

Turned on all at once, they would also have blocked 69% of the PRs that had no bug. Trimmed to the 11 rules that caught something, that drops to 12%. Keep them, but they don't replace reading the code.

Smaller PRs get better reviews

I had an agent split each large PR into a stack of small ones, each passing CI on its own. On those large PRs, a mid-priced AI reviewer went from finding 63% of the bugs to 79%. That's close to the careful, expensive reviewer, at under half its cost.

One warning from the same lab: AI-written PR summaries can mislead. On about 1 in 4 buggy PRs, the summary described the buggy line as fine. A summary can talk a reader out of looking.

Many bugs are decisions, not typos

Almost half the planted bugs (47%) weren't typos but decisions, the kind someone could have written down before coding. For example: a list that quietly stops at 50 items when nobody asked for a limit, a partly successful run counted as a failure, or keeping the older of two duplicate rows instead of the newer one.

So I tried reviewing the plan before the code. It caught 77% of the wrong plans, but it also sent back 64% of the right ones, which is a lot of friction. And it didn't make the code review any easier afterward. An AI reading the code caught even more of the same bugs (84%) than it did reading the plan.

Auto-approving "low-risk" PRs lets serious bugs through

I trained a risk score to pick which PRs could merge without a person reading them. It's a simple model (a logistic regression) that looks at eight things about each PR: its size, how much of it is tests, which packages and sensitive paths it touches, how much other code depends on it, how much of it no test protects, what the automated checks found, and whether a quick AI review flagged a bug. With it, about a third of PRs could skip review and the team still stayed under 2 bugs slipping through per 100 PRs.

The catch is the serious bugs. Even when only 1 in 10 PRs skipped review, one serious bug still got through. So if your rule is "no serious bug merges unread," no setting of the score meets it.

Why can't it do better? Most of what a risk score looks at is the outside of a PR: which files it changes, how many lines, how much other code depends on it. That tells you how risky a change could be, not whether it actually has a bug.

My test shows this clearly. Every buggy PR has a twin: the same change, touching the same files, the same size, just without the bug. Looking only at the outside, the two are identical, so the score can't tell which one to stop. The only thing that could tell them apart was a quick AI review, because it reads the code itself.

Speed dashboards look best when review is worst

Most teams track review with the numbers their Git host gives them for free, like time to merge. Those numbers tell you how fast review is. I wanted to know: would they notice if people stopped actually reviewing?

Real projects couldn't answer that. I looked at six months of PRs (5,570) from three open-source projects that use AI reviewers: formbricks, infisical and trigger.dev. Thanks to all three for working in the open. The only sign of a bad review there is a revert: a later PR that undoes it within a week (marked "Revert" in its title or linking the PR it undoes). Reverts are rare: 7 in 4,213 merged PRs. That's far too few to tell good review from bad.

So I simulated it. Picture a team reviewing PRs for six months. Halfway through, half the reviewers stop reading and approve everything in two minutes. I ran that 200 times, plus 200 teams where nothing changes, to see what normal ups and downs look like.

Here's what happened after the reviewers stopped reading:

• PRs merged faster, with fewer rounds of back-and-forth. On a dashboard, that looks like the team got better.
• Reverts didn't move. Even three months later, they showed nothing unusual.
• Bugs slipping through almost doubled. But that's not a number any dashboard shows.
• More PRs were approved without a single comment. This was the one number that went the wrong way, and it showed up within two weeks. It's the one to watch.

What to do: don't judge review by speed alone. Also watch how many approvals come with no comment. And now and then, plant a known bug and see whether review catches it.

Putting it all together

Finally, I combined the pieces into seven complete review setups and ran each one on the same 250 PRs. The numbers below assume 1 in 10 PRs has a bug.

Setup	What's in it	Review minutes per PR	Bugs that slip through, per 100 PRs	Serious bugs that slip through (of the 38 planted)
People review everything	One person per PR (two on risky changes)	18.3	1.04	2
The pipeline I planned	Checks, AI review, Prove filter, risk routing	14.4	1.23	3.7
AI only, no people	Checks + AI review, no person	1.3	1.79	6
AI review + one person	Checks + AI review + one person per PR	13.6	0.75	2

"Risky changes" are migrations, schema and validation. "Risk routing" means the risk score auto-approves some PRs, gives some a quick look and the riskiest a full read. Here are all seven setups. Click any point for details, and enter your own team's PR count:

For an example team merging 60 PRs a week, the winner means about 13.6 hours of review a week instead of 18.3. Treat those as estimates: the minutes come from an assumed reading speed (500 lines an hour, at least 5 minutes a PR), not from timing anyone.

I've left out a dollar figure on purpose. In the labs, one AI review cost about half a cent, but that's a small model reading small PRs once. Real PRs are bigger, get reviewed again on every push, and review tools charge per seat, so your bill will be much higher. The comparison that matters is against reviewer time: a few saved hours a week cost far more than the AI review does.

Why the simple setup won

Adding or removing one piece at a time shows what each piece is worth:

• Adding the AI reviewer helps. Fewer bugs slip through, for about one extra minute per PR.
• A second human reviewer wasn't worth it. The "people review everything" setup had a second reviewer on migration, schema and validation changes. It cost about 6 minutes per PR and caught very little extra. Dropping it is where most of the time saving comes from.
• Routing by risk let more bugs through than it saved time. It replaces careful reads with quick looks, and a quick look adds almost nothing on top of the AI's comments.
• The AI double-check lost bugs, for the reason above: with one review run, there's no second copy to fall back on.

The winner is boring, and I count that as a plus: there's no router to maintain and no second AI to tune.

One condition: read the code before the AI's comments

All of the above assumes the person catches what they'd catch alone, whatever the AI said. So I tested that too. The same simulated reviewers read the same PRs three ways: with no AI comments, with the AI review's comments above the diff, and with a deliberately wrong but confident AI comment pointing away from the bug.

The bugs that matter are the 35 the AI missed, because only the person can catch those:

• The careful reviewer found 16 of them after reading the AI's comments, against 21 and 18 in two runs without. That's close to how much two identical runs differ on their own. But of the 14 bugs it found in both runs without the comments, it missed 6 once it saw them. A third identical run only misses about 1 in 12 bugs found twice.
• The rushed reviewer leaned much harder: 5 found, against 10 and 15, and it wrote half as many bug comments. It treated the AI's review as the review.
• The deliberately wrong comment barely made things worse. What cost recall was the AI having spoken at all, not where it pointed.

Put back into the comparison, this changes the answer. If the person reads the AI's comments first, "AI review + one person" lets through as many bugs as people reviewing everything (1.04 per 100 PRs), and more of the serious ones: 5 of 38 instead of 2. The advantage only holds if the two catches stay independent: read the diff, write your comments, then open the AI's.

These are model reviewers, so this shows how a model leans, not how much a person does. But it's cheap to protect against, and the test showed that even a 5% chance of a person missing what the AI didn't flag is enough to tip the result.

What this doesn't prove

Some limits, so you can judge the results for yourself:

• The "people" are simulated. The human reviewer in these numbers is a careful AI reviewer that never gets tired. Real people get tired, and how much they lean on the AI's comments was measured in models, not in people.
• New PRs, same codebase. The labs were tuned on these 250 PRs, so afterwards I wrote 99 new ones, nothing tuned on them, and reran the AI reviewers, the checks and the risk score on them. The answer held: AI review + one person let through 0.87 bugs per 100 PRs against 1.16 for people reviewing everything, and the AI found 84% of the new bugs, against 80% before. They're still the same app, though, so another team's code could move the numbers.
• One codebase, planted bugs. Real bugs, in other languages, may behave differently.
• Serious bugs are rare. With only 38 of them, the data can't tell 2 slipping through from 3.
• Review isn't only about catching bugs. Sharing knowledge, discussing design and building trust don't show up in any of these numbers. They may be worth a second reviewer even when the bug count says otherwise.

What I'd do

If AI is writing your code:

1. Put an AI reviewer, your automated checks and one person on every PR, and have the person read the code before the AI's comments.
2. Keep PRs small. It helps human and AI reviewers alike.
3. Review the plan if you want decisions settled early, but it won't replace code review, and it adds friction.
4. Be suspicious of "AI checks the AI." Measure what it throws away.
5. Judge review by bugs caught, not time to merge.

Most of all, test your own setup. Plant a few bugs and see who catches them. Several of my "obvious" improvements made things worse, and no dashboard would have told me.

I walk through each lab, with the code and full results, in the explainer videos.