The concurrency number that actually decides local-model GPU sizing

Infra leads size a GPU the way they’d size an API rate limit: pick a concurrent-agent count, multiply by a tokens-per-second guess, buy the card that clears the bar. That treats concurrency as a permission problem, something a provider grants or throttles. A local model does not work that way. Nobody is rate limiting anyone. The limit is the box itself, and the mechanism deciding how many agents can share it looks like a fixed budget, not a rate limit. I went looking at how Cerevisor’s own harness handles this, since it runs both kinds of providers side by side and had to solve exactly this to avoid a runaway local server or an underused GPU.

Two different limits wearing the same word

A cloud API credential (Anthropic, OpenAI, or any hosted endpoint) shares a queue nobody outside the provider controls. Cerevisor’s harness treats this as “remote-api” work: a credential starts with 16 requests in flight, and that cap moves on its own. A 429 (the standard “going too fast” error) cuts it by 30 percent; a repeat within 60 seconds doubles that to 50 percent, since a repeat means real pressure, not a blip. A 500-series error, meaning the provider is broken rather than busy, trims the cap by one instead. Ten clean requests in a row, with at least 30 seconds since the last cut, earns one unit back. This is a small version of AIMD (additive increase, multiplicative decrease), the decades-old internet congestion-control idea: back off hard when something breaks, recover slowly once calm returns. The cap never drops below one.

A local model reached over an OpenAI-compatible connection, meaning Ollama, vLLM, LM Studio, or any self-hosted server speaking the same request format as a cloud API, is different. The harness checks the address a request targets: a loopback or local-network host (localhost, a 127.x/10.x/172.16-31.x/192.168.x address, or a name ending in .local) gets reclassified as local inference, regardless of speaking the identical protocol. Once marked local, none of the AIMD logic above applies. There is no rate limit to adapt to, because what holds the machine back is the GPU’s own memory and compute, not someone else’s throttle.

What actually caps a local model’s concurrency

Local work draws from a separate, fixed pool: 48 units total in Cerevisor’s harness, with different work costing different amounts depending on how much of the machine it consumes. A cloud API call barely touches the local machine while it waits on the network, so it costs almost nothing here. A full local model doing inference is the most expensive kind of work in the pool, since it pins GPU memory and compute directly: one concurrent local-model request costs 12 of the 48 units.

The code comment next to that number does the arithmetic rather than leaving it to guesswork: 48 divided by 12 is 4, chosen to match what a simpler, older version of the same code gave a local Ollama server. Nothing here reacts to an error; it is a fixed budget reacting to what the work costs, set once. That default is not a claim about how many agents a specific GPU can serve well: a 24GB card running a 7B model in 4-bit has very different headroom than the same card running a 34B model at higher precision, and a check made from a URL has no way to know that. What the fixed budget buys is predictability. A different ceiling for specific hardware is a deliberate configuration decision, not something recalculated on the fly.

Why the retry path matters as much as the ceiling

Slot count is half the sizing question. The other half is how much GPU work each request actually burns, and that moved this window. Ollama shipped a fix on August 21 for a hang affecting agent clients that cancel a long prefill (the pass where the model reads the whole prompt before replying), plus a bug where a cancelled or retried prefill lost its progress and restarted from scratch. As the release notes put it, the fix “prevented requests matching 46k of 47k tokens from reprocessing entirely.”

That matters for an agent harness, because agents retry constantly: a watchdog times out a stalled call, a malformed tool result gets rerun, a stream drops mid-response. Before the fix, a retry on an affected model could mean redoing almost the whole prefill, competing with the other three slots on the same card. A serving bug that quietly doubles a retry’s real cost is invisible to a fixed budget; it just shows up as the model feeling slower than the math predicts.

What this means for the hardware purchase

The economics of the GPU itself have not moved. GetDeploying puts the current median on-demand H200 rate at $4.40 per GPU-hour across 40 providers this week, “about 23% above where it was a year ago,” even with a flat trailing 90-day trend. The card is not cheaper on a one-year view, only steadier on a one-quarter view.

A local model is never rate limited by anyone but its own operator, which means nobody comes to warn you when it has been oversubscribed.

Six concurrent agents against a card sized for four will not produce a 429. It produces something quieter: latency creeping up for everyone on the card, looking like “the model isn’t fast” rather than “the GPU is oversubscribed.”

The useful move before sizing a card is measuring the actual per-request GPU cost of the specific model and quantization level on the specific hardware, rather than borrowing someone else’s benchmark, since a serving-layer fix like the one above can silently change that cost. Our guide to connecting local and OpenAI-compatible models covers how the loopback detection works and how to point Cerevisor’s harness at a self-hosted server. The concrete next step: measure what a serving stack costs per retried request before committing to a concurrency target, then size the card to that number instead of a rule of thumb.

By the Cerevisor team