--- title: "How to Size vLLM Quantization for Multi-Agent Workloads" slug: vllm-quantization-multi-agent-memory-sizing date: 2026-09-28 excerpt: "Shrinking model weights with 4-bit quantization solves cold-load graphics memory, but concurrent AI agents immediately crash into context memory limits. Here is how to size vLLM quantization, cache buffers, and execution modes for local multi-agent teams." featured_image: "https://bbtxujdxvidaghmhxkqs.supabase.co/storage/v1/object/public/generated-images/blog-1790592939587-vllm-quantization-multi-agent-memory-sizing.webp" featured_image_alt: "A technical diagram illustrating 4-bit quantized model weights alongside expanding conversation cache buffers across concurrent AI agents." author: Kinan Hamwi author_url: https://www.linkedin.com/in/kinanhamwi/ canonical_url: https://cerevisor.com/blog/vllm-quantization-multi-agent-memory-sizing updated_at: 2026-09-28T10:55:42.153869+00:00 --- # How to Size vLLM Quantization for Multi-Agent Workloads By the Cerevisor team TLDR Shrinking open-weight models down to 4-bit numbers lets a 70B parameter model fit onto a single 48 GB graphics card. But when four AI agents run simultaneously, their conversation history and tool outputs create an expanding memory footprint that weight compression never touches. Here is the operational playbook for sizing vLLM quantization, enabling 8-bit cache storage, and setting your local orchestration harness so multi-agent workflows complete without crashing your server. I watched an [engineering team](/blog/harness-coding-agents-engineering-headcount-board-question) celebrate on Monday because their new quantized 70B model loaded cleanly into memory on a single workstation. Model quantization is the process of converting neural network weight numbers from 16-bit precision to smaller 8-bit or 4-bit numbers to drastically reduce the graphics memory needed to load a model. By four in the afternoon, the team was staring at an out-of-memory error and wondering why their local machine froze during a routine four-agent code review. The model had loaded with room to spare. The failure happened the moment four agents began calling tools and reading files in parallel. That gap between static weights and dynamic conversation history is where most self-hosted agent deployments quietly break. --- ## Why 4-Bit Weights Do Not Solve Multi-Agent Memory The math that tricks most teams looks simple on paper. A 70B parameter model in standard 16-bit weights needs roughly 140 GB of graphics memory. Compressing those weights with 4-bit Activation-aware Weight Quantization (AWQ) brings that number down to roughly 39 GB. On a workstation equipped with a 48 GB professional card, that leaves 9 GB of free graphics memory. Serving a single human chatting in a web browser, 9 GB feels like plenty of room. The problem begins because AI agents are not single human chats. An agentic session accumulates long system prompts, tool schemas, file contents, and step-by-step reasoning traces across dozens of turns. To avoid recalculating every prior token on every single turn, the serving engine stores those intermediate states in a Key-Value cache. The KV cache is the memory buffer a language model uses to store previously calculated attention states for all preceding words in a conversation so it does not have to recompute them from scratch for every new token. Weight quantization shrinks the static checkpoint on disk and in memory, but it does not shrink the KV cache by a single byte. For a 70B model using grouped-query attention, storing a 32,000-token context in 16-bit precision consumes roughly 2.5 GB of graphics memory per stream. Run four agents in a concurrent wave, and their active caches demand 10 GB of memory. Add 1.5 GB for runtime buffers and CUDA overhead, and the total memory requirement hits 50.5 GB. On a 48 GB card, the engine runs out of physical memory and either crashes or thrashes into host memory. 50% reduction in active KV-cache graphics memory achieved by switching from 16-bit cache to FP8 --- ## A Five-Step Playbook for Sizing vLLM and Cache Footprints Solving this memory bottleneck requires treating conversation memory with the same discipline applied to model weights. In official documentation published on September 18, 2026, the [vLLM](/blog/how-to-run-open-weight-llm-vllm-production) team detailed how quantizing active conversation states resolves this wall. > "Quantizing the KV cache to FP8 halves the memory footprint compared to 16-bit formats, allowing double the concurrent context capacity without measurable loss in perplexity." vLLM Documentation, September 2026 When setting up [local inference](/blog/build-vs-buy-gpus-memory-spike) for [multi-agent workflows](/blog/ai-agent-orchestration-platform), follow this sequence before letting agents run concurrently: - **Calculate the dynamic context ceiling before selecting weights** Determine the maximum number of concurrent agents and their expected context depth. Multiply the number of parallel streams by the cache size per token to calculate the necessary memory buffer before picking a weight quantization tier. - **Quantize both weights and attention caches** Do not stop at 4-bit AWQ weights. Launch vLLM with the flag setting the cache data type to 8-bit floating point. This immediately halves cache memory from 2.5 GB to 1.25 GB per 32,000-token stream. - **Activate automatic prefix caching** Multi-agent pipelines in harnesses like Cerevisor share extensive context, including system instructions, codebase rules, and tool descriptions. Turning on prefix caching ensures those identical prompt segments share a single set of memory blocks across all agents instead of duplicating them. - **Cap the maximum model length explicitly** Many modern open models advertise a 128,000-token window. Leaving the engine context length unconstrained forces the scheduler to reserve excessive memory structures. Cap the maximum model length to 32,768 tokens if the workflow tools stay within that limit. - **Align the orchestration harness execution mode** Configure the orchestration harness to respect local hardware limits. If physical graphics memory cannot support peak agent concurrency, switch execution to sequential mode so agents run one after another instead of contending for memory. --- ## Where Static Allocation in Ollama Collides With Dynamic PagedAttention Choosing the right local inference server directly determines whether concurrent agents run smoothly or stall. Ollama and vLLM take fundamentally different architectural approaches to managing memory. Ollama is built on llama.cpp and excels at developer setup and single-user tasks. But when configured for concurrency, it pre-allocates static memory buffers for each parallel slot. If configured for four parallel streams, Ollama reserves memory upfront for each slot. On a single GPU running a quantized 70B model, this static reservation leaves no margin. If memory limits are exceeded, Ollama either fails the request or offloads model layers into system RAM, slowing token generation down to a crawl. In contrast, vLLM manages memory through PagedAttention. PagedAttention is an inference memory management system that divides the memory holding past conversation tokens into small, non-contiguous virtual blocks, preventing wasted graphics memory. Instead of pre-allocating large static blocks per user, vLLM assigns 16-token memory pages on demand as tokens are generated. When connected to an orchestration harness like Cerevisor, this difference becomes critical. In Cerevisor, any local server connects through an OpenAI-compatible interface, as outlined in the [OpenAI-compatible and local models guide](/docs/guides/providers/openai-compatible-and-local). Cerevisor normalizes the endpoint URL and sends standard completion requests. Inside the harness runner, Cerevisor maintains a strict ten-minute inactivity watchdog. If an un-tuned inference server runs out of cache memory and spends ten minutes swapping memory blocks without emitting a single token, the harness assumes the agent has stalled and terminates the execution. Setting up vLLM quantization and cache compression properly prevents those silent freezes. --- ## The Memory Math of Active Context Across Concurrent Agents To see the operational difference in practice, consider the exact memory distribution for a 70B model running four concurrent agents with a 32,000-token window on a 48 GB card. 70B Model Graphics Memory on a 48 GB Card (4 Concurrent Agents at 32K Context) ConfigurationWeight VRAM[KV Cache VRAM](/blog/kv-cache-quantization-long-context)Runtime OverheadTotal Footprint Standard 16-Bit Cache39.0 GB10.0 GB1.5 GB**50.5 GB (OOM)** vLLM 8-Bit Cache39.0 GB5.0 GB1.5 GB**45.5 GB (Fits)** 8-Bit Cache + Prefix Reuse39.0 GB2.8 GB1.5 GB**43.3 GB (Safe Margin)** In standard 16-bit precision, four parallel streams push the total graphics memory to 50.5 GB, causing immediate failure. With 8-bit cache quantization enabled, the cache footprint drops to 5.0 GB, fitting the entire workload into 45.5 GB. When prefix caching is activated and the four agents share a 10,000-token prefix containing the project specification and tool schemas, the cache memory drops further to 2.8 GB, leaving a comfortable 4.7 GB buffer. --- ## What to Reconfigure on Monday Morning For teams self-hosting models for agentic tasks, weight quantization alone does not complete the hardware sizing equation. Compressing weights is merely table stakes to get the checkpoint loaded. The real operational boundary is the memory footprint of active conversation history. Audit serving flags before the next sprint. When serving with vLLM, add the 8-bit cache setting and enable prefix caching today. When running Ollama on a single workstation for multi-agent experiments, check workflow execution settings in Cerevisor and set the execution mode to sequential. Taking thirty minutes to align serving memory with multi-agent concurrency turns a fragile inference setup into a dependable, production-ready local cluster. #### Sources - [Quantization and KV Cache Compression in vLLM v0.29.0](https://docs.vllm.ai/en/latest/features/quantization/fp8.html) - vLLM Documentation, 2026-09-18 - [vllm-metal: Fast Agentic Serving on Apple Silicon](https://vllm.ai/blog/2026-09-22-vllm-metal) - vLLM Blog, 2026-09-22 - [Qwen3.8-2.4T PD Disaggregated Serving](https://vllm.ai/blog/2026-09-21-qwen-pd-serving) - vLLM Blog, 2026-09-21 - [Ollama Multi-Model Concurrency Architecture](https://github.com/ollama/ollama/blob/main/docs/faq.md) - GitHub (ollama/ollama), 2026-09-16