AI agent orchestration frameworks report success. Almost none of them define it.

By the founder of Cerevisor

Earlier this year I opened our own run history and found an entry that read, in effect, three agents completed, zero failed. All three had failed. The software had been asked a question it was never designed to answer, and it answered confidently.

What the green badge on an AI agent orchestration platform actually certifies

An orchestration harness is the layer that runs several AI agents on one job, decides their order, passes each the output of the last, and reports what happened at the end. The belief worth taking apart is that a run reporting success produced work someone can use.

In most of these tools the success value means something narrower: the loop reached the last step without being aborted. Not that the output is correct, not that a reviewing agent approved it, not even that every agent inside the run did its job. One flag, doing the work of five different answers.

Where the success flag came from, and why it was honest when it was written

The design was inherited, and it used to be true. Build systems have reported this way for decades, and there it holds, because a compiler fails loudly. Agents broke that equivalence in the least convenient direction. An agent that cannot do the job usually does not crash. It writes a fluent summary describing work it did not do, exits cleanly, and hands the next agent a confident paragraph built on nothing.

A compiler that fails tells you. An agent that fails writes you a paragraph explaining how well it went.

One in ten incidents now involves AI, and the runs behind them looked fine

The operations company StackGen went through a large archive of public status-page records. Channel Dive covered the analysis on August 12:

The same coverage notes that agents "wiped data, deleted databases and destroyed live systems in at least nine documented cases since last year". Nine is small against 180,000 records. What matters is the category: the service keeps running and keeps answering, and the answers are wrong. Nothing alerts, because from the system's point of view nothing failed.

The tooling has noticed too. Claude Code shipped versions 2.1.232 through 2.1.236 between August 14 and August 18, and those notes read as one theme. A human denial routed through a background sub-agent could be "dropped" without a trace, and now is not. Oversized internal messages now refuse upfront "instead of silently dropping them". One verb across different bugs: what used to disappear quietly now reports.

The wave where every agent failed and the log recorded a clean sweep

Back to my three completed agents. A wave is a group of agents that run at the same time, with the next group waiting for all of them. The cause was event ordering.

When an agent errored, our failure path emitted an error event, then a completion event carrying the error text so downstream agents received something rather than silence. Both were correct alone. The accounting watching them took the latest event it saw, and completion arrived second. So it won. A wave in which every agent failed was recorded, in the source comment's own words, as completed with none failed. The fix is one line of policy expressed in code: a failure marker is terminal. Once an agent is marked failed, a later completion cannot un-fail it.

That tally is not just a number in a history view. It feeds the situation summary handed to the next group of agents, so a wave told the previous stage succeeded builds on output nobody validated. It also feeds the record the product learns from, which means a miscount teaches the system something untrue.

Five outcomes an AI agent orchestration framework should be able to tell apart

The reframe is not "trust nothing". It is that one flag was always too few. A finished run in Cerevisor resolves to one of five results, checked in this order, because a run can qualify for several at once.

1. Stopped

   A person or a gate halted the run. A cancelled run should never be graded on what it produced.

2. Not accepted

   An agent whose declared job was to review or test wrote an explicit rejection. A written no outranks everything below it.

3. Blocked

   An agent could not proceed and said so, which usually means a required check never ran. Absence of a verdict is not a pass.

4. Completed with failures

   Some agents errored and the rest of the work stands. Useful, and not the same as clean.

5. Completed

   Nothing above applied. The only one that means what people assume the green badge means.

The old true-or-false value still sits underneath, and it kept its narrow meaning on purpose: the run reached the end without an abort. A real fact, just not the one anyone wanted.

The second result needed the most care. A reviewing agent discusses failure constantly. It will report that tests failed, describe what it would reject, and use the word "fail" a dozen times in a paragraph that ends in approval. Scanning that prose for the word is worse than useless.

So the reader in Cerevisor refuses to scan prose at all. It looks only inside a region the agent explicitly labelled a verdict: a heading whose text ends in that word, or a bolded verdict line. It reads at most the first 400 characters after the label, and stops earlier if a new heading begins. Inside that window the signal appearing first wins, so a verdict opening with the decision and later mentioning a flaky test still lands correctly. A standalone uppercase GO is the only spelling of "go" that counts as approval, because ordinary writing says "go ahead" constantly. Only agents whose declared role is to test or review get a vote, which is the separation any sane engineering org already runs on humans. The agent roles reference in the Cerevisor docs shows which roles carry that authority.

Three questions worth asking your AI orchestration tools this week

None of this needs a migration or a budget line. It needs about twenty minutes. First: open the most recent run that reported success and find out what the tool meant by the word. If the documentation only says the run finished, that is the answer, and it is better to know before the next board slide is built on it.

Second: check what happens when a reviewing agent says no. In many setups the rejection is written to a file, the run ends green, and nothing reads the verdict. A written no that no system reads is not a control.

Third, the one that found our bug: take a run where something definitely went wrong and count the failures in the summary against the failures in the detailed log. If those numbers disagree, the summary is lying.

Here is the encouraging part. None of this needs a better model. Five results instead of one, a failure marker that cannot be overwritten, a verdict window narrow enough to trust: those are ordinary software decisions, made by people, in code that can be read. Honesty about what a run achieved is something a tool can be built to do, and the ones that do it are not harder to use. They are just harder to fool.