Failed, denied, or stopped: the AI observability gap in agent run logs

By the founder of Cerevisor

On 31 August, Anthropic shipped Claude Code 2.1.252. Four fixes in the release. Three of them had nothing to do with an agent doing the wrong thing: oversized failure output from a background task breaking the conversation, shell commands on some Macs failing over a moved scratch directory, and remote sessions appearing to stall for minutes after the tool had already finished.

In each case the work was either fine, or broken for a reason that had nothing to do with the code. What was broken was the account of it. The signal that reached the person watching.

That is a harness problem, and I think it is the most under-priced one in the category. A harness here means the software that actually runs an AI coding agent: it decides what the agent may touch, when to stop it, and, above all, what to tell the human afterwards.

Why a red row in the run log stopped meaning anything

The comfortable belief goes like this: a run log is a record, records are neutral, and a team that cares about quality will read theirs. That is the model most agent dashboards are built to satisfy.

It falls apart on contact with the actual categories. Watch a wave of agents for an afternoon and count how many distinct things arrive wearing the same red badge. An agent crashed. An agent reached for a folder it was not allowed to touch and was correctly refused. Somebody hit stop because they had seen enough. A run hit its spending cap. A shell command exited non-zero because the test it ran genuinely failed, which is the system working.

Four of those five are the harness doing its job. One is a defect. All five look identical from three metres away.

Trust is high and daily friction is high at the same time

Temporal published its State of Development 2026 report on 25 August, surveying 554 engineers and engineering leaders across the US and UK, fielded in April and May. Coverage of it on 29 August pulled out two findings that sit oddly together.

The same coverage records that 85.5% of respondents trust agent outputs at least somewhat. A large majority trusts the output. Two in five hit problems every single day.

That is not scepticism, and it is not naivety either. It is the signature of a channel people have stopped reading closely. They keep a general sense that things are mostly fine, and stop parsing individual rows.

Four of those five red rows are the harness doing its job. One is a defect. All five look identical from three metres away.

What false-alarm research says about a run log nobody reads

The predictive value of an alert is the share of times it fires and turns out to be real. Fires ten times, right ten times: high. Fires ten times, right twice: low, no matter how reliably it catches the two.

Getty, Swets, Pickett and Gonthier published a laboratory study in the Journal of Experimental Psychology: Applied in 1995 testing what happens to operators as that value moves. The finding, replicated many times since under the older folk name of the cry-wolf effect, is that human response degrades as a function of predictive value and not of sensitivity. An alarm that catches every real event still loses the operator if it also fires on plenty of non-events. Nobody decides to ignore it. Response time simply lengthens until it is no longer useful.

So a reviewer is not a fixed resource who reads everything. They are a finite one, whose care is spent by every row that turns out not to have needed it. Hiring a second reviewer buys more capacity. Cleaning the channel buys back capacity already being wasted, and costs a great deal less.

The seven words Cerevisor makes a tool call answer with

We shipped Cerevisor 2.4.0 today, and this is the part I would defend hardest, though it will get less attention than the four-agent ceiling being gone.

Every tool call an agent makes now ends in exactly one of seven dispositions: success, failed, denied, timed out, cancelled, unavailable, or partial. That vocabulary is shared, deliberately, between the part of the app that decides what happened, the record written to disk, and the interface that draws the row. All three name the same thing, so a log line cannot drift from the fact behind it.

Denied means the harness refused: the agent reached outside the folder it was pointed at, and the boundary held. That renders as an amber notice reading “Not allowed”, naming the path, rather than a red error. Partial means real work landed but not all of it, a directory walk that hit a subtree it could not read, and it is deliberately not treated as an error.

Underneath sits a split that took us a while to get right. A tool call has two independent ways of not working. The call itself can fail: the thing that executes tools threw, or the tool did not exist. Separately, the call can execute perfectly and the work can come back bad: a non-zero exit code, an HTTP error, a search provider with no key configured. Only the first kind counts as a call failure.

That matters, because Cerevisor also has a circuit breaker for agents that get stuck repeating themselves. Once two identical failures are on record, the next attempt is short-circuited before the tool runs at all, so no further money goes into a retry we already know the shape of. Two, not three: an audit found the third expensive retry was where a hot loop did most of its damage.

Here is the named failure that forced all of it. Before this design, a path refusal came back as an ordinary error, and it counted toward the breaker. So an agent being correctly and repeatedly kept out of a folder it should never have touched would get shut down for being stuck, on the strength of the safety system working exactly as intended. The fix was not a better message. It was giving refusal its own word. A refusal now carries no error, is excluded from the breaker’s count, and tells the agent which path was refused so it can adapt. The full change list is in the Cerevisor 2.4.0 release notes.

Four checks to run on the next AI observability tools demo

Whatever harness is on your shortlist, these are answerable in a single session, and they sort the category quickly.

1. Put a refusal next to a crash

   Ask to see a run where the agent was refused something, side by side with a run where the agent genuinely crashed. Do the two rows look different without clicking into either? If not, the failure number in the summary is a mixture, and it will drift further as the guardrails get better at their job.

2. Repeat a refusal and watch the stuck-agent protection

   Point an agent at a folder it is not allowed to touch, three or four times. If refusals feed the same counter as crashes, the tool will eventually shut down an agent for being correctly contained, which is the exact failure that forced our own redesign.

3. Stop a run by hand, and let another one hit its budget cap

   Read what each one says afterwards. Anything that resolves to plain failure is discarding the only two outcomes a reviewer can act on immediately, because both have an obvious owner and neither needs an investigation.

4. Count the red rows from one real afternoon

   Take a day of your own agent runs and sort every red row into defect or not-a-defect. The share that are not defects is the number that decides how much attention the log is still worth. It is usually the first time anyone on the team has seen it written down.

None of this is urgent the way a security finding is urgent. It gets quietly expensive over a year, in the form of a senior engineer who has stopped reading the log and has not mentioned it to anyone, including themselves. The reassuring part is that the fix is a vocabulary decision. It costs nothing to run, and any harness team can make it in an afternoon once someone names the problem out loud.