Where multi agent orchestration spends its time with the models switched off

By the founder of Cerevisor

Last measured: 31 August 2026.

On 20 August I captured a performance baseline of the Cerevisor harness. On 31 August I captured it again on the same laptop, and the biggest cell in the matrix came back noticeably faster than it had eleven days earlier. My first instinct was to write that down as an improvement. My second instinct, the correct one, was that two runs on a laptop eleven days apart at different commits do not establish anything about a code change, and publishing it as a win would have been the exact species of benchmark theatre this piece exists to avoid.

So here is the honest version instead, with the method in front of the numbers.

A word on vocabulary before anything else, because the argument leans on three ideas. An orchestration harness, sometimes called a control plane, is the software that decides which agent runs when, hands one agent’s output to the next, and writes down what happened. A stubbed model provider is a stand-in that answers every model call instantly with one fixed sentence, so nothing crosses the network and nothing is billed. And wall clock time is time as a stopwatch measures it, not processor time, which means it includes waiting.

The question a multi agent orchestration benchmark should actually answer

Most published agent benchmarks measure whether the agents got the task right. That is a fine question and a well served one. It is not the question an engineering leader asks in the week they are choosing how to run agents in production.

That question is narrower and much less glamorous: of the time a multi agent run takes, how much is the model thinking, and how much is the machinery around it? Because those two numbers have completely different remedies. If the machinery is expensive, better prompts will not help. If the model is expensive, a faster coordination layer buys nothing.

The trouble is that model latency is enormous, variable, and drowns everything else. A single real model turn can take several seconds and swing wildly between runs, so any measurement that includes it measures the provider that day. The only way to see the coordination layer at all is to take the model out.

How this multi agent orchestration benchmark was measured

Everything below comes from one command against the Cerevisor codebase. No network calls, no API keys, no spend, no telemetry sent anywhere.

node scripts/perf-baseline.mjs --out .tmp/perf-baseline-20260831.json

What runs. Nine cells: three run shapes crossed with three workflow sizes.

The three sizes are one agent in one stage, six agents arranged as three stages of two, and ten agents arranged as five stages of two. A stage here is a group of agents that run at the same time; the next group only begins once the previous group has finished. Each agent in a stage is wired to the matching agent in the stage before it, so the run genuinely serialises across stages rather than collapsing into one big parallel batch.

The three run shapes are:

1. Plain run

   One workflow, executed once, start to finish. This is the floor cost of coordination: session setup, preflight checks, opening the audit log, and writing checkpoints to disk.

2. Pipeline

   Two chained copies of the same workflow, each written to disk as a separate file and loaded by path, with a summary of the first passed into the second. Two chained steps rather than one, because a single step pipeline would only measure the wrapper.

3. Refinement loop

   Exactly two rounds. The last agent is scripted to fail its first attempt. A planner proposes one edit, the edit is applied and acknowledged, and round two converges. Agents that passed in round one are preserved rather than re-run, which is the real product behaviour and not a shortcut taken for the benchmark.

Sample size. Seven timed samples per cell, plus one warm-up run per cell that is discarded before timing starts. Each cell reports the median, which is the middle run, and the p95, which is the sample only one in twenty is slower than. The individual run times are kept in the output file, not averaged away, so anyone reading it can see the spread rather than trusting a single figure.

Hardware and versions. An 11th generation Intel Core i7-11800H at 2.30 GHz, 16 logical cores, 31.65 GB of RAM, Windows build 10.0.26200, Node v22.16.0. The code measured is the Cerevisor development main line at commit 8ecbac0, captured on 31 August 2026. That is a development tree, not a released build, and I am not presenting these numbers as a claim about what any shipped installer does on anyone else’s machine.

What is deliberately excluded. Every model reply is served by a stub that returns instantly, so nothing here reflects provider latency, token cost, or model quality. Application startup time is recorded as null rather than estimated, because it cannot be observed from outside the desktop shell. The measurement runs in a single process with no parallel test workers, because performance figures are only meaningful when nothing else on the box is competing for the same cores.

Nine cells, three run shapes, one laptop

Median wall clock milliseconds, seven samples per cell, warm-up discarded, captured 31 August 2026.

The p95 figures, in the same order: 307.07, 381.75 and 594.85 for the plain run; 1610.30, 1789.99 and 1995.80 for the pipeline; 1866.06, 1951.07 and 1939.07 for the refinement loop.

One more number that sits outside the matrix. Saving a checkpoint, which is how a run’s state survives a crash, measured a p95 of 118.96 milliseconds across 200 consecutive writes against a ten agent workflow snapshot. That operation reads the whole session file, updates one entry, and writes the file back, so it is dominated by disk and JSON rather than by anything clever.

Sit with that for a second. Ten agents, five sequential stages, full session setup, audit log, checkpoints to disk, and the whole thing costs under half a second of harness time. A single real model turn from any frontier provider costs several times that on its own.

The coordination layer was never the expensive part of a multi agent run. It just happens to be the part that is visible, so it is the part that gets blamed.

Why the refinement loop barely notices whether it runs one agent or ten

This is the result I did not expect, and it is the most useful thing in the table.

Look along the top row. The plain run climbs steadily with size: 136.16, then 340.16, then 466.45. That is what anyone would predict. More agents, more stages, more time.

Now look along the bottom row. The refinement loop goes 1685.77, then 1760.12, then 1915.37. It barely moves. Running that loop over ten agents costs only a little more than running it over one.

That tells me something concrete about where the cost lives. The refinement loop’s expense is in the round machinery itself: running the workflow, collecting what failed, asking a planner for an edit, applying the edit, acknowledging it, and running again. Fanning that machinery out across more agents adds comparatively little. The refinement loop is the mode people reach for when a run half worked and they want the harness to fix it rather than starting over, and the practical implication is that its cost is roughly a fixed toll per round, not a per agent charge.

Where exactly inside a round that fixed cost sits, this matrix does not say. It reports total wall clock per cell and nothing finer. I have not attributed it, and I am not going to guess at a breakdown I did not measure.

The parts that failed, including the perf run that was really the whole test suite

A benchmark that only publishes its wins is marketing. Here are four things that went wrong or came back unusable.

The first version of this harness silently ran the entire correctness suite. The test runner auto-discovers a default project list even when it is explicitly pointed at a different configuration file, so a command that looked like a performance capture was in fact running every unit and integration test in the repository. It took twenty five minutes and produced a “baseline” that measured nothing. The fix was a dedicated single-project list passed explicitly on the command line, and a comment in the file recording why it exists, because the failure is completely invisible from the outside. Anyone building a similar measurement on this test runner will hit it.

Application startup is unmeasurable here and is published as null. Startup involves the desktop shell creating a window, which cannot be observed from inside a test worker. There is a rough stand-in in the output, the cost of loading the main code graph, but it includes build-time transformation and is only comparable against other runs of the same harness on the same machine. It is not a startup number and I do not present it as one.

The single agent plain run is the noisiest cell in the matrix and I would not defend it to two decimal places. Its median was 136.16 milliseconds, its p95 was 307.07, and the individual samples ranged from 67.09 to 307.07. The 20 August capture had a single sample in that same cell at 562.72 milliseconds. At that scale the measurement is mostly reporting what the operating system felt like doing that second.

The two captures disagree, and I am not resolving the disagreement. On 20 August the ten agent plain run had a median of 650.55 milliseconds. On 31 August the same cell measured 466.45. Same laptop, same sample count, different commits, unknown background load. That gap is large enough to be interesting and unattributed enough to be meaningless. Checkpoint writes moved the other way, from a p95 of 108.91 to 118.96, which on a Windows machine is as likely to be antivirus scanning a temporary directory as anything in the code. Neither movement is a finding. Two data points on one machine are two data points on one machine.

What the research published this week says about where the cost actually sits

I do not want a single laptop to carry this argument alone, so I looked at what landed elsewhere in the last week.

On 25 August a group publishing a controlled ablation on a production agentic science platform reported something that lines up uncomfortably well with the table above. They varied the coordination topology, the harness type, the model, and the prompt, on one verifiable task, and found the model choice swamped everything else. Their phrasing was blunt: federation imposed a negligible penalty on performance, while the choice of language model dominated prediction quality far more than topology or prompting. Their reported spread was roughly 92 to 94 percent accuracy for the strongest model against roughly 40 to 50 percent for a small one, on the same coordination structure.

The same week, a paper on routing between models across the steps of a multi agent workflow opened by naming the cost source directly: these workflows incur substantial operating costs due to repeated model invocations and long-horizon context accumulation. Repeated invocations. Accumulated context. Not coordination.

And then the counterweight, which is the one I would put in front of a board. On 26 August a team published a failure study of a real agentic software delivery platform, and it is the most honest artifact I read all week.

Their finding is that the standard reliability tools borrowed from web infrastructure, retry and timeout and error-rate tripping, rest on assumptions that agent delegation breaks. Among the incidents they catalogue: twelve in which the safety enforcement layer blocked correct work, the most expensive costing 107 agent turns and zero accepted writes. They are careful to say their study motivates a controlled evaluation rather than constituting one, which is a rigour I wish more agent research showed.

Put the three together and the picture is consistent. Coordination is cheap. Models are expensive. And the thing that actually destroys a run is not latency at all, it is a control decision taken on bad evidence.

What a multi agent orchestration framework comparison like this cannot tell you

Six limits, stated plainly, because a benchmark without them is a sales sheet.

It is synthetic. Every model reply is one fixed sentence returned instantly. Real replies are long, variable, and arrive in a stream, and streaming has its own costs this does not touch.

It is one machine. One laptop, one operating system, one storage device. Checkpoint timing in particular is dominated by disk behaviour and will move on different hardware.

It is one harness. These numbers describe Cerevisor’s coordination layer and say nothing about anyone else’s. The method transfers; the figures do not.

It measures overhead, not quality. Nothing here indicates whether the agents produced good work, because the stub always returns the same sentence.

The run shapes are not comparable to each other. The pipeline cell runs two chained copies of the workflow, so it is doing roughly twice the agent turns of the plain run at the same size plus a handoff. Compare a cell against the same cell in a later capture, never against its neighbour.

And it is a development tree, not a released build. The commit is recorded so the capture can be repeated exactly, but a shipped installer is built and configured differently.

Re-running this yourself, and the two numbers worth writing down

If a coordination layer is currently under evaluation on an engineering team, the useful move is not to trust my table. It is to build the equivalent for whatever is being evaluated, because the method is what transfers.

Three things make it worth doing.

First, stub the model. Whatever tool is under evaluation, find a way to make model calls return instantly with fixed text. Everything measured after that belongs to the harness, and it is usually the first time anyone on the team sees that number at all.

Second, vary one dimension at a time and keep every sample. Sizes on one axis, run shapes on the other, seven samples per cell, warm-up discarded, individual timings retained. The spread matters more than the median, and the cells that refuse to stabilise are telling you where the system is fragile.

Third, record the shape of each curve, not just its height. A curve that climbs with agent count and a curve that stays flat have different fixes, and knowing which one is in front of you saves a week of optimising the wrong thing.

Two numbers are worth putting on a page and revisiting each quarter: coordination cost at the largest workflow size a team actually runs, and the date it was last measured. The second number is the one that keeps the first honest.

The method here, the harness that produces it, and the run history that makes a slow run diagnosable after the fact are documented in running and monitoring a workflow. The next capture of this matrix is scheduled for the next release boundary, and it will be published whether or not the numbers flatter anyone.