AI code review does not get cheaper as the agent gets better

By the Cerevisor team

In the last six days, four coding-agent vendors shipped something that asks a person to decide less often. GitHub’s July update for Visual Studio Code, published July 30, added assisted tool approvals, where the model judges how risky a tool call is so it can stop interrupting for the safe ones. Claude Code’s August 4 release fixed two cases where commands had been running with no prompt at all. Cursor put a review inbox on the iPad on July 29. On August 3, GitHub made agent runs start from a single comment on a pull request.

Every one of those is a sensible change. Read together they describe a direction: the number of times a person is asked to look is falling, while the volume of machine-written code arriving for review is rising.

The promise of AI-powered code review, and the part it leaves out

The quiet assumption underneath is that review is a temporary tax. The agent makes mistakes, humans catch them, and as models improve the catching gets cheaper until it nearly disappears. Review cost, on that view, is a function of the agent’s error rate.

Reasonable model. Wrong shape. The cost of reviewing agent output does not track the agent’s error rate. It tracks how carefully one tired person is still looking, and that curve moves in a direction nobody budgets for.

Why fewer approval prompts feel like progress

It feels right because it matches the first six months exactly. Early on the agent fails visibly, the catches are frequent and satisfying, and review is obviously worth the time. Later the agent is right most of the time, the prompts feel like a toll booth on an empty road, and clicking approve forty times a day stops being judgment and starts being data entry.

So vendors do the humane thing and remove prompts. GitHub’s AI code review has supported custom skills and connected tooling since July 29, so more of the first pass is machine-generated, with attribution showing where each comment came from. Good engineering. Also a system that gets smoother exactly where a person used to be interrupted.

What 11,429 reviews of agent-written code show about approval rates

In June a group of researchers measured the thing everyone assumes. They followed the same reviewers over time rather than comparing different people, which is what makes it useful.

The rise held after controlling for calendar time, so it tracks a reviewer’s accumulated experience rather than the year improving. It was specific to agent pull requests, since the same reviewers’ approval rates for human pull requests fell over the same period, and it was not easier work: median pull request size stayed flat.

Then the part that should stop a reader. Review latency rose three and a half times while inline comment volume dropped 22%. More time waiting in the queue, less time inspecting. The authors read that as reflexive habituation under growing workload rather than reviewers learning where the agent can be trusted. Being precise: this is a workshop paper built on observational data, not a controlled trial. It shows a pattern, not a proven cause.

The 1993 monitoring result AI code review tools keep rediscovering

The pattern is not new, and the older evidence is stronger. Human factors research has a term for it: automation complacency, the measured tendency to monitor a machine less closely the longer it keeps being right.

In 1993, Parasuraman, Molloy and Singh ran 40 people through a flight simulation while an automated routine watched for engine faults. When that routine’s reliability varied across the session, people caught 82% of its failures. When it stayed steady, they caught 33%. With nothing else to do, detection ran near 97%.

The variable was steadiness, not the reliability level. Predictable performance dulled the watcher. Erratic performance kept people looking.

A 2010 study by Reichenbach, Onnasch and Manzey put a number on the antidote. Among 88 participants, those who had never seen the aid fail made a commission error, meaning they accepted an answer that the evidence in front of them already contradicted, 20.4% of the time. Those shown one failure early in the session made that error 4.5% of the time, and checked 97.4% of the readings needed to verify the aid instead of 92.0%. One experience of the machine being wrong beat any amount of training.

What keeps a reviewer sharp is not the agent's error rate. It is whether the record of the last failure is still on the screen.

What we changed in Cerevisor after the Operator journal went blank

We hit this inside our own product, which is how I know the fix is not a slogan.

Cerevisor has a feature called Operator: it wakes on a timer, by default every 30 minutes while the app is open, and makes exactly one bounded move on its own. Ceilings of $0.50 per check and $5.00 a day, nothing it drafts is applied without a person accepting it, and it turns itself off after three consecutive checks that failed or escalated.

Then a review found the hole. Operator keeps a plain-language journal of its checks, and it is capped, oldest evicted first. A self-disarmed Operator kept writing cheap skipped entries, and enough of them piled up to push out both the record of the shutdown and the three real failures that caused it. What remained was a journal saying only that autonomy was off. The record of the failure aged out faster than the memory of it.

The fix changed what gets evicted rather than how much is kept. Entries that cost money, plus the two entries recording a shutdown or a free failure, are now evicted last, so routine noise cannot bury the one entry a person needs. Two related decisions followed. Failure notifications fire once per episode instead of once per check, because an alert arriving every 30 minutes stops being read by week two. And the Operator panel was inverted so the review queue leads and the activity log collapses to one line, since the log was the first thing tired eyes learned to skip.

An honest footnote: our own Operator guide still described a spending allowance the code removed on July 31. Documentation that has been correct for a while is exactly what people stop re-reading.

Three checks the best AI code review tools cannot run for you

None of this needs a new tool. It needs three questions and about twenty minutes.

Count opens against accepts. Take the last twenty agent-authored items merged in one service area and find how many were actually opened before approval. The gap between reviewed and approved is the number worth knowing.

Ask when a human last caught an agent failure that the tests did not. If nobody can name one from the past month, that is not evidence of a clean record. On the 1993 result, it is the condition under which detection falls to a third.

Then check that the last failure survives. Whatever surface holds the agent’s history, a run log, a journal, a pull request feed, confirm that a month of ordinary activity cannot push the most recent real failure off the screen. Cheapest fix on this list, and the one nobody thinks to make.

The reassuring part is that the research points at design, not discipline. The people running agent fleets are already trying hard, which is exactly why habituation reaches them. For somewhere concrete to start, open the run history in Cerevisor v1.9.0 or whatever the team uses, look at the last twenty entries, and find the most recent real failure. If it has scrolled away, that is this week’s fix, and a smaller job than it sounds.