Human-in-the-loop AI has a cost your dashboard doesn't show

Every Harness Now Ships A Human-In-The-Loop Interrupt Button

On September 6, Cerevisor shipped 3.1.0, and the headline feature is a “board”: a shared feed of short notes that the agents in one run leave for each other, and for the person running it, while the work is still going on. One thing anyone can do on that board is ask. An agent can address a direct question to a named colleague, to “anyone,” or to the person supervising the run. This is not a Cerevisor-only idea. GitHub’s Copilot agent mode has its own version of an agent stopping mid-task to ask a focused clarifying question, and Cursor’s plan mode does the same before it starts writing. The pattern is now common enough to call a convention: a harness, the software that runs and supervises AI agents doing a job, is expected to know when to stop and ask, not just when to keep going.

What’s less common is anyone publishing what that costs the person on the other end of the question. The same week Cerevisor shipped the board, Digital Applied published a piece arguing that when two AI agents contribute to the same release, a clean merge only proves there was no textual conflict, not that a human checked whether the two contributions actually agree with each other (Digital Applied, September 7, 2026). Read together, the two stories point the same way: agents are getting better at flagging things for a person to look at, and nobody yet writes down what a growing stream of things-to-look-at does to the person doing the looking.

Agents Get Human-In-The-Loop Rules. The Operator Gets Human-On-The-Loop Silence.

There are two different ways to design a system around a person’s attention. In a human-in-the-loop design, the system pauses and genuinely waits for someone before it continues. In a human-on-the-loop design, the system keeps moving, and the person can check in and step in whenever they choose, but nothing on the system’s side is waiting on them. Cerevisor’s board uses both, and it uses them on two different people.

Here is the asymmetry, verified directly in Cerevisor’s own board code before publishing this piece, not inferred from the release notes. When one agent asks a named peer a question, that is treated as human-in-the-loop-style, except the “human” is another agent: the wait is bounded to ninety seconds, and if the peer is still busy the asker’s turn pauses for at most that long, then moves on with an honest “nothing yet” rather than sitting frozen. It’s rationed too: an agent may only ask three such questions in one run. Interrupting a specific busy peer is treated as expensive on purpose, so the code makes it deliberately scarce.

A question addressed to the operator gets the human-on-the-loop treatment instead, minus the part where anyone is watching the loop. Any agent can post a note marked “question” and address it to “user” as many times as it wants in a run. It shows up in the run window as an amber “Question for you” card, and it just waits there: no ninety-second clock, no cap of three, nothing that measures how many of those cards piled up before anyone happened to open the tab. The people who built the feature clearly reasoned about the cost of interrupting a busy machine. Nothing in the code suggests anyone budgeted the cost of interrupting a busy person the same way.

What A 2008 Study Still Gets Right About Interrupted Work

This isn’t a new problem, and it has a real answer, just not one that comes out of software. In 2008, Gloria Mark, Daniela Gudith and Ulrich Klocke ran an experiment at CHI, the main human-computer-interaction research conference, with 48 people split across interrupted and uninterrupted versions of the same office task. The surprising part of their result is that the interrupted group did not take longer overall (the difference in time-to-finish across conditions was statistically real, F(2, 77.98) = 3.36, p < .05, and the uninterrupted baseline group actually took the longest). People compensate for interruptions by working faster. What compensating cost them is the part that never shows up on a stopwatch.

This is the missing half of any feature that pings a person mid-task. If the only instrument is “how long did it take to answer,” the number will look clean and suggest nothing is wrong, because the person on the other end quietly sped up to cover the gap. The cost that study measured is workload, stress, frustration and effort, none of which appear in a run log. A harness that markets itself as human-in-the-loop AI but never counts how many unanswered questions are sitting in the operator’s queue right now has no way to notice the thing that’s actually accumulating.

An interruption that takes no extra time on the clock is not an interruption that cost nothing.

The Three Gaps In The Interrupt Budget

Three questions worth putting to any harness vendor, including us, before the next rollout:

1. Does anything in the product count how many questions are addressed to a single person in a day, the way it counts questions between agents? In Cerevisor today, the honest answer is no: the caps that exist protect an agent’s turn, not a person’s afternoon.
2. If the system is willing to make an agent wait, and willing to make an agent give up after three tries, on what basis did it decide the operator’s attention doesn’t need the same protection?
3. What would the board look like if a growing pile of unanswered questions triggered anything at all, the way a stuck agent already does?

None of these are rhetorical. They are the same questions a security team would ask about an unbounded queue, applied to a queue made of a person’s attention instead of a server’s.

What Human-In-The-Loop AI Should Actually Track Next

To be specific about what’s actually true today rather than what should be true: Cerevisor’s board is three days old at the time of writing, and this asymmetry is not something we are hiding. It’s a genuinely open design question, and naming it here is the fastest way to make sure it gets fixed rather than quietly normalized. For anyone already running Cerevisor, the concrete next step is to open the Board tab on the busiest workflow after a full day of runs and count the amber “Question for you” cards, answered and unanswered, from that one day. That count, not the time each one took to answer, is the real interrupt budget nobody is tracking yet. Read more about how the board works, including who can ask what, in the board guide, and see what else shipped in 3.1.0.

By the Cerevisor team