Human in the loop AI agents: review has a speed limit, and it is human

By the founder of Cerevisor. Checked on 2 November 2026 against the source code of Cerevisor 4.0.

Last Wednesday GitHub published its yearly Octoverse report, and one line in it stopped me. A pull request is a proposed change to a codebase that someone has to look at before it is accepted. GitHub’s count of how many of those now begin with an agent has climbed steeply since March.

GitHub frames this as good news for review. The same post says review is where developers shape what comes back. I agree with that. What the report cannot show is the speed at which those reviews happen, and that speed is the part I think about most when I build a workflow with a person in it.

What code review speed research actually found

The best evidence I know on this is old, careful and a little unglamorous. In 2009 Chris Kemerer and Mark Paulk analysed two data sets of programs written under the personal software process, a disciplined method in which each developer logs their own work, including how long they spend checking it. That logging is what made the study possible: they knew how fast each person reviewed and how many defects slipped through.

Their result, in the paper’s own words:

LOC means lines of code, so the review rate is simply how many lines a person reads per hour. The sentence I find more important sits just before that one: the authors found review rate was a significant factor in how many defects were removed “even after accounting for developer ability.” Put plainly, speed mattered on its own, not just as a stand-in for skill.

The honest limits matter here. These were individual developers checking their own programs, years before any AI agent wrote a line. The study does not tell anyone how to review an agent’s work. It tells us something about the reader, and the reader has not changed.

Where human approval for AI agents gets rushed

Here is the mind side of the problem. When agents produce work faster than one person can read it, the pressure does not show up as a decision to skim. It shows up as a queue. Each item looks reasonable, the agent was right the last several times, and the button that says continue is right there. Nobody chooses to review at triple speed. The pace drifts there because the volume asks it to.

I see the same pull in my own harness. Cerevisor 4.0 added a human step: a card in a workflow where the run stops and waits for a person to look at what the agents prepared, fill something in, and click a button. The run waits however long that takes, which is the right instinct. Nothing times the person out.

But the default shape is broad. A new human step arrives with the prompt “Look at what the agents prepared and tell them how to continue,” and two buttons, Continue and Stop here. Put one of those at the end of a long workflow and the person is handed everything at once, with one click to approve all of it. That is the exact setup the review rate research warns about. The checkpoint exists, and the pace it invites is the fast one.

There is also a quieter failure that I want every builder to know about. In Cerevisor, a connection is the line that carries one step’s work to the next. Each connection leaving a human step can be tied to one of its buttons, and then it is followed only when the person clicks that button. A connection that is not tied to any button is followed whatever is clicked. So if someone adds a button called Revise and forgets to tie the revision path to it, that revision path runs whatever the person clicks, Continue included. The person thinks they made a choice. The run ignores it. The fix is simple once it is named: tie every connection that leaves a human step to exactly one button, and test each button once before trusting it.

Building a human in the loop step for a slow read

The pattern I use now treats the reviewer’s pace as something the workflow has to protect, because the reviewer under pressure will not.

1. Put the checkpoint where the work is small

   Place a human step right after the one agent whose output is worth reading, not one step at the very end. A short piece read slowly beats a long one read fast.

2. Ask a question the person cannot answer by skimming

   Add a required written field such as "What did I check?" Cerevisor refuses an answer while a required field is empty, so the click alone is never enough.

3. Give each real choice its own button

   Continue, Revise and Stop here are three different decisions. A human step can carry up to six buttons and up to twenty fields, which is plenty for a careful decision.

4. Tie every outgoing connection to a button

   Then click each button once on a test run and watch where the work goes. The guide to connecting agents shows how connections carry work between steps.

None of this makes review fast. That is the point. The workflow can do the fast part, and the person keeps the part that only goes well slowly.

Why AI agent review fatigue is a design problem, not a character flaw

Teams evaluating an agent harness tend to ask whether it has human approval at all. It is the wrong place to stop. Most agent tools now offer some kind of approve button. The better question is what each approval hands the person, and how big that pile is.

The Kemerer and Paulk result is a reminder that review quality is mostly about conditions, not virtue. Their finding held after accounting for ability, which means the fix is rarely “hire more careful people.” It is giving careful people smaller, clearer things to look at, and fewer of them at a time. GitHub’s numbers say the volume of agent work arriving for review is rising quickly. The reader on the other end still reads at a human pace, and that pace is not a weakness to engineer away. It is the thing the whole checkpoint depends on.

An approve button proves a person was asked. Only the size of what they were asked to read tells anyone whether they could actually check it.

What I would tell a team over coffee

If I were sitting across from an engineering lead who is about to put a person in the loop of an agent workflow, I would ask one thing: how much work lands in front of that person in a single sitting? If the answer is “the whole run,” I would split it before adding anything else.

The next action is small. Open one workflow, find the place where a person approves the most at once, and break it into a human step after the agent that matters most, with one required question and a button for each real choice. The Cerevisor 4.0 release notes describe how human steps work. Then click every button once and see where the work goes. It is the cheapest way I know to keep a checkpoint honest.