A Multi-GPU Box Won't Make Your Agents Run In Parallel By Itself

By the Cerevisor team

Two posts from the vLLM engineering team landed two days apart this week, and together they are the clearest public sizing guide I have seen yet for a specific, narrow question: what does it actually cost in GPUs to serve real AI-agent traffic, not a chat demo. vLLM is one of the most widely used programs for running an open-weight language model on your own hardware and serving it to multiple requests at once. The team published three real configurations: 12 of Nvidia’s newest GB300 chips serving 256 concurrent agent conversations against a model called DeepSeek V4 Pro, at up to 83,000 tokens produced per GPU per second. That throughput number is not a marketing benchmark. It is the shape of what a team actually has to buy if it wants many AI agents hitting one self-hosted model at once.

And it exposes something worth checking before anyone signs off on that hardware order: what your own agent-orchestration software assumes happens when several agents hit one local model at the same time. I write agent-orchestration software for a living, so I went and checked what Cerevisor assumes. The answer is not what most teams reading those GPU numbers would guess.

What Makes Agent Traffic Different From A Chat Demo

The vLLM team’s September 8 post, built on public agentic-coding traffic traces, describes the shape of a real agent session in a way that is worth sitting with. A typical session runs a median of 43 back-and-forth turns, not one question and one answer. Each turn carries a long input, a median of 142,000 tokens (a token is roughly three-quarters of a word, so that is well over 100,000 words of accumulated conversation and tool output) against a short output of just 444 tokens. And more than 96 percent of the time, the model can reuse work it already did on an earlier turn instead of recomputing it from scratch, something the industry calls prefix caching.

That last point matters because it is the opposite of how most people picture GPU load. An agent session is not many short independent questions. It is one long, slowly growing conversation that gets replayed and extended on every turn, which is exactly the shape a coding agent, a research agent, or a workflow orchestrator produces.

These are three different models on three different hardware setups, not one scaling curve, so I am not going to stretch them into a formula. What they show plainly is that serving real agent concurrency is a deliberate, tested configuration, not a side effect of owning more GPUs.

The Memory Cliff That Shows Up Once Conversations Pile Up

A companion post two days later, on September 10, names the point where a single GPU’s own memory stops being enough. Every ongoing conversation with a language model keeps a working memory of everything said so far, called a KV cache, sitting in GPU memory so the model does not have to recompute earlier turns from scratch. That cache grows with every turn, and GPU memory is finite.

The vLLM team put a number on the cliff: up to roughly 64 simultaneous conversations, a GPU’s own fast memory holds everything comfortably. Between 64 and 128, that memory fills up and throughput drops sharply if nothing is done about it. Past 128, even the server’s regular system memory fills up too. Their fix, tiered offloading, moves older cached conversations down to slower storage instead of throwing them away, so a conversation that has gone quiet for a moment can be reloaded rather than recomputed from the start. Done well, that more than doubles how many conversations one machine can carry.

There is a multi-GPU angle here too: once cached conversations live in shared storage rather than only on one card’s memory, that cache can be shared across machines, letting a new GPU pick up where another left off instead of starting cold.

The GPU count on a spec sheet says nothing about how many agent conversations a box can actually carry once real concurrency shows up.

What Cerevisor’s Own Harness Assumes While None Of This Runs

Here is the part I can only answer by reading our own code, and it is worth doing before trusting any orchestration tool with a serious local deployment.

Cerevisor is a desktop control plane that runs multiple AI agents against whichever model you point it at, local or hosted. Our own documentation for connecting a local or OpenAI-compatible model is blunt about what the harness does and does not do: it sends a standard request and waits. It does not try to optimize around a local model’s own speed, and on a single machine, running several agents in what the interface calls a parallel wave still behaves one-at-a-time in practice, because the model itself is the bottleneck, not how the harness schedules work. The documentation’s own recommendation for a local model is to switch to Sequential execution mode in settings, meaning agents run strictly one after another, precisely because a Parallel setting buys nothing when the thing underneath it cannot actually serve more than one request well at once.

That default made sense back when most local setups were one modest GPU running one model for one agent at a time. It stops making sense the moment a team builds something closer to what the vLLM posts describe: a real multi-GPU box, tuned for exactly the kind of long, cache-heavy, many-turn sessions an agent produces. At that point the serving side genuinely can carry dozens or hundreds of concurrent sessions. The harness sitting on top of it, left on its old Sequential default, will still run agents one after another and never ask the new hardware to do more.

There is a second assumption worth naming, and it is a subtler one. Every agent in Cerevisor runs under a single stall timer: if an agent produces literally nothing, not one byte of output, for ten minutes, it is assumed frozen and killed. That timer applies the same way whether the agent is talking to a hosted API or a local model, and there is no separate, longer grace period for a local endpoint. The only thing that resets that timer early is the first sign of life from the model, the first character of a streamed response. A model that is still loading, still warming up a large cache, or still initializing the kind of tiered offloading the September 10 post describes, and therefore producing zero output for a while, gets no special treatment. It is treated exactly like a call that might genuinely be stuck.

The Named Failure: New Hardware, Same Old Setting

Picture a team that reads the vLLM posts, builds something close to the smaller of the three published setups, two capable GPUs serving one model with real concurrency and tiered offloading turned on, and expects their agent harness to immediately put a dozen agents on it at once. If they never touch the Sequential default, they will not get the parallel throughput they just paid for. Agents will still queue one behind another, and the new hardware will look underused even though it is working exactly as configured.

Flip the setting to Parallel without checking anything else, and the second assumption becomes the risk. A fresh multi-GPU deployment under real load, still warming caches, still settling into tiered offloading, is more likely to have a slow, quiet stretch than a single small model that has been running for weeks. If that stretch runs past ten minutes with zero bytes streamed, agents get killed as stalled, and the failure looks exactly like a hardware or model problem. It is neither. It is a generic safety timer that was never told the deployment underneath it had changed shape.

Match The Setting To The Hardware You Actually Bought

None of this is a reason to distrust either side. The GPU sizing numbers vLLM published are honest, tested, and more useful than almost anything I have seen on agentic serving specifically. Cerevisor’s Sequential default and its one stall timer are reasonable, conservative choices for the common case, a single machine, one model, a handful of agents.

The mismatch only bites teams who change one side and assume the other adjusted with it. If a serving upgrade like the ones described above is on the roadmap, the concrete next step is small: before flipping execution mode from Sequential to Parallel, confirm the serving stack underneath genuinely supports the concurrency being asked for, the way vLLM’s published configurations do rather than guessing. And before running the first real load test against new hardware, check the stall timer in the Workflow settings reference and make sure it is long enough to survive a slow, cold start rather than killing the first honest test run. Cerevisor’s guide to local and OpenAI-compatible providers is the right place to start; it says plainly what the harness does and does not do, which is exactly the kind of thing worth reading before, not after, the hardware invoice arrives.