How to Run an Open-Weight LLM in Production With vLLM

This week two updated self-hosting guides walked through the same exercise, and both landed on a number that surprises people the first time they do the math: the model weights are the easy part of the bill. A Lushbinary guide updated June 25 sized GLM-5.2, the MIT-licensed 744-billion-parameter model that shipped open weights this month, onto private GPUs and put the practical floor at an eight-card H200 node. Not because the weights are huge, though they are. Because once the KV cache and the runtime overhead come in, a node that looks roomy on paper gets tight fast.

That is the whole story of self-hosting in one sentence. The download is free. The operations are not.

So this is a playbook for the part the model card skips: how to take an open-weight model and actually serve it in production with vLLM without it turning into a second on-call rotation. I will use GLM-5.2 as the worked example because it is fresh and well-documented, but the method is the same whether the target is a 744B coding model or a 27B support bot.

What vLLM actually is, and why the memory math below is not optional

Worth being precise about this before the sizing math, because that precision is what makes the rest of this playbook trustworthy instead of guesswork.

vLLM started in 2023 as a research project out of Ion Stoica’s Sky Computing Lab at UC Berkeley, built around one idea worth understanding on its own terms. Every request to a language model carries a running memory of everything generated so far in that conversation, the KV cache mentioned throughout this piece. The naive way to hold that memory is one large contiguous slab of VRAM reserved per request, sized for the worst case. That wastes enormous amounts of memory on padding, because most conversations never reach their worst case. PagedAttention, the algorithm vLLM is built around, borrows a trick from how operating systems manage RAM: split the cache into small fixed-size blocks, hand them out on demand, and let requests that share an identical prefix share the same blocks instead of copying them. Memory gets reclaimed the instant a request finishes instead of sitting reserved and idle. A GPU that used to hold a handful of concurrent requests can suddenly hold dozens on the same hardware. That single idea is the entire reason a cache line item exists in the sizing math two sections from here, and it is the reason vLLM became the default serving choice for teams that were previously watching GPUs sit half-idle under naive memory allocation.

The project has outgrown “one lab’s research code” for a while now. vLLM’s own governance writeup describes it as maintained by a consortium of groups including UC Berkeley, Anyscale, AWS, CentML, Databricks, IBM, Neural Magic, Roblox, and Snowflake, among others. It moved through Linux Foundation AI incubation starting mid-2024 and became a PyTorch Foundation-hosted project in May 2025, specifically so no single company controls the roadmap. That is worth knowing before betting a production stack on it. A single-vendor open-source tool can get abandoned or repriced the moment its backer’s incentives shift. A foundation-governed project with a dozen infrastructure companies running their own production traffic through it is a steadier bet, though steadier still means pinning versions and testing a model on them, which is exactly what step two below is for.

The number that ends most “we’ll just self-host it” plans

Start with the sizing, because the sizing decides everything downstream and it is where the optimism dies.

A model’s weight footprint is simple arithmetic: parameters times bytes per parameter. At FP8, GLM-5.2’s roughly 744 billion parameters need about 744GB. An eight-card H200 node gives 1,128GB of aggregate VRAM. Looks comfortable. It is not, and the Lushbinary guide is explicit about why.

Read that again. The 384GB that looked like spare capacity is not spare. It is the working space for the KV cache, the per-request memory that grows with context length and concurrency, plus a runtime overhead that the same guide puts at 10 to 20 percent. The KV cache alone can run into many tens of gigabytes per concurrent request even with grouped-query attention and an FP8 cache. Go to full BF16 precision and the weights roughly double to 1,488GB, which points at sixteen GPUs instead of eight.

So the binding constraint is rarely the weights. It is the weights plus everyone talking to the model at once, plus the context window they were promised. That is the trap. A team sizes the box for the weights, ships it, and watches it fall over at moderate concurrency because nobody budgeted the cache.

The five steps from open weights to a serving endpoint

Here is the sequence I would run, in order, before a single GPU gets provisioned. None of it is exotic. The value is in doing it in this order, because each step kills a plan that the next step would have wasted money on.

1. Budget VRAM as weights plus cache plus overhead

   Compute weights from parameters times bytes per parameter at the chosen precision. Then add KV cache (tens of GB per concurrent request at long context) and a 10 to 20 percent runtime overhead. For GLM-5.2 at FP8 that math lands on an eight-card H200 node as the practical floor, with real headroom only after the cache is accounted for.

2. Pin a known-good vLLM version before trusting the FP8 path

   Serving-engine version is not a detail. The ofox.ai guide is specific: vLLM v0.23.0 is the minimum that serves the FP8 GLM-5.2 path on a general-availability release. Patch releases after that add throughput, but the floor is the floor. Pin the version in the container and test the exact model-and-quant pair on it before promising anyone an endpoint.

3. Pick a quant on purpose, not by default

   Quantization is a lever, not a freebie. AWQ INT4 cuts GLM-5.2 from about 744GB to about 372GB, which moves it from eight H200s to four, at a stated 1 to 3 percent quality regression on coding benchmarks. That is a real tradeoff to make deliberately: halve the box, accept a small measured quality cost, then verify it on the one task that matters rather than trusting the benchmark.

4. Set the cache and concurrency flags, then load-test them

   The serving config is where sizing meets reality. An FP8 KV cache roughly halves per-token cache memory versus BF16, which is often what makes the required concurrency fit at all. Set the memory-utilization ceiling with headroom for allocation spikes, cap the context length to what the workload truly uses, and then push real traffic at it. vLLM ships its own tool for this, a command called vllm bench serve that fires synthetic requests at a running server and reports the throughput and latency numbers this step actually needs, so there is no reason to hand-roll a load-test script that quietly measures the wrong thing. The cache saturating under bursty load is a far more common failure than the weights not fitting.

5. Run the break-even before you buy anything

   This is the step that should happen first emotionally and last procedurally, because it can cancel the whole project. Price the real prompt volume against a hosted baseline. If the math says hosted wins, the cleanest deployment is the one nobody runs.

That last step deserves its own section, because it is where the most money is saved or wasted.

Why the GPU bill almost never breaks even at the volume you have

The reason most self-hosting plans should stop at step five is that the economics are unforgiving below a surprisingly high volume line. The ofox.ai guide, updated June 23, puts hard numbers on it.

An eight-card H200 cloud node runs roughly 30 to 50 dollars an hour blended. A hosted plan for the same model can be on the order of 30 dollars a month. The crossover, where standing up a private serving stack beats just paying for the hosted endpoint, sits around 3,000 prompts a day, which the guide frames as a twenty-developer team running coding agents constantly. Below about 100 prompts a day with no compliance constraint, its advice is blunt: do not self-host.

And that number is only the GPU side. The part that quietly wrecks the comparison is the operations cost on top. A Digital Applied decision guide from late May put the labor multiplier plainly: DevOps salaries, model update cycles, and infrastructure overhead typically add a three-to-five-times multiplier on top of the GPU rental alone. The GPU is the cheap part of running the GPU.

I have watched this play out the expensive way. A team moves off the API to “save money,” provisions a node, runs it at single-digit utilization because their traffic is bursty, and somehow spends more than before while also now owning a pager. The fix was never a better model. It was batching the traffic, capping the context window to what the workload actually used, and admitting that a chunk of their volume was fine staying hosted. Local won where it should and the API kept the rest.

The GPU is the cheap part of running the GPU. The forgotten line item is the engineer who reboots it at 2 a.m.

That mirage shows up in tooling, too, not just spreadsheets. Cerevisor, a local-first control plane for AI agents, treats a self-hosted vLLM endpoint as just another provider choice, and its own documentation says what happens to the number plainly: a vLLM target reports token counts but no cost, so the interface prints $0.00 for the run and labels the entry Local. That figure is honest about the missing per-token bill and silent about the eight-card node underneath it. Nothing in the method above changes because of that. The VRAM-plus-cache-plus-overhead math from the top of this page is still the real number. Free is a property of the dashboard, not the electricity bill, in more places than this one.

What the serving guides agree on, and where to be careful

Step back and the picture across this week’s in-window sources and the recent operations corpus is consistent, which is reassuring because it means the method is stable even as models churn.

The sizing logic (weights plus cache plus overhead) is the same in the June 25 Lushbinary guide and the June 17 Spheron deployment writeup, which lands on the same eight-card H200 floor for GLM-5.2 and adds the concrete serving flags: tensor parallelism across the eight cards and an FP8 KV-cache dtype for the long-context path. The break-even direction (self-host wins only at sustained high volume) is the same in the ofox.ai prompts-per-day framing and the older tokens-per-month analyses, even though they use different units. When independent guides built for different audiences agree on the shape of the answer, that is the part to trust.

Where to be careful: the specific in-window numbers come from a small set of serving guides all keyed to one release, GLM-5.2. The version floor, the eight-card sizing, the prompts-per-day break-even are well-attributed but they describe a single model’s deployment as of late June, not a universal constant. The failure-mode detail (memory-utilization ceilings, the gap between PCIe and NVLink bandwidth that can wreck tail latency in tensor-parallel setups, the 30-to-90-second cold start on serverless) is solid operations knowledge, but it predates this week, so treat it as durable background rather than fresh news. None of that changes the method. It just means the numbers get verified against the actual model and the actual hardware instead of copied.

Where vLLM sits against SGLang, and the honest answer for which one to run

vLLM is not the only serving engine solving the KV-cache problem, and a comprehensive answer to “how do I run this in production” has to say so. The other name that comes up constantly is SGLang, and the detail most people miss is that it shares vLLM’s own family tree: both trace back to Ion Stoica’s Sky Computing Lab at UC Berkeley, just a couple of research cycles apart. Where they diverge is the memory trick. SGLang uses RadixAttention, which organizes the KV cache as a tree instead of vLLM’s fixed-size blocks, so any two requests sharing a prefix, not just an identical fixed system prompt, automatically reuse the same cached memory. vLLM has its own answer for the fixed-prefix case, automatic prefix caching, but it is narrower by design: strong when the shared text is constant and stable, weaker on the branching, ever-extending histories that agent workloads produce.

That difference maps onto a real decision, not a benchmark trophy. Bulk, short-turn traffic where throughput is the scarce resource, the GLM-5.2 example running through this piece, is the case PagedAttention was built for, and it carries the deeper production track record and the widest hardware support of the two. Traffic shaped like an agent harness working through one long, shared, constantly-growing conversation is closer to what RadixAttention was built for. Trust neither camp’s benchmark page for a specific percentage without running it against real traffic first.

The two projects have also taken different bets on who controls them, which matters as much as the architecture for a production decision. vLLM went the neutral route, the PyTorch Foundation-hosted structure the earlier section on what vLLM actually is already covers. SGLang’s core team took the opposite path: in 2026 it spun out as a venture-backed company, RadixArk, raising a hundred million dollars in seed funding at roughly a 400 million dollar valuation from Accel, Spark Capital, and strategic investors that include NVIDIA’s and AMD’s venture arms, with adoption already reported at xAI and the coding tool Cursor. Neither structure is wrong. A foundation is slower to move and answers to a committee. A funded startup moves faster and answers to investors who want a return. Know which one is backing a production bet before pinning the version and moving to the next decision.

What’s new in the vLLM ecosystem this year, and whether it changes the playbook

Four things have moved in the vLLM world since this piece’s original GLM-5.2 example, worth a scan even though none of them change the sizing method above.

The most consequential for anyone running more than one model is vLLM Semantic Router, an officially adopted vLLM sub-project that sits in front of a fleet of models as a classifier, not a generator. It reads each incoming prompt and decides which backend should answer it: a cheap, fast model for a simple factual question, a heavyweight reasoning model for a multi-step financial projection, rather than sending every request to whichever model is biggest. For a team that has already sized one production box, per this page, and is now looking at a second or third model for a different job, this is the piece that turns three separate deployments into one routed fleet.

The most consequential for teams eyeing OpenAI’s move into open weights is fast, well-documented vLLM support for gpt-oss, the two models OpenAI released under an open license in 2025 as its first open-weight release since GPT-2. The smaller of the two, gpt-oss-20b, runs on roughly 16GB of VRAM, a fraction of the eight-card node this page spent most of its word count sizing for GLM-5.2. That is not a knock on GLM-5.2’s numbers. The two models are not competing for the same job. It is a reminder that the five-step method above (size the cache, pin the version, choose the quant on purpose, load-test, run the break-even) applies just as much to a model that fits on one GPU as to one that needs eight.

The most consequential for multimodal roadmaps is vLLM-Omni, an official vLLM extension purpose-built for any-to-any models that jointly handle text, image, audio, and video rather than text alone. It is a genuinely different serving problem: those models chain multiple specialized components (autoregressive language layers next to diffusion image or video generators) instead of one uniform transformer, and vLLM-Omni’s architecture disaggregates the serving stack so each stage scales independently. Worth tracking if the roadmap includes a voice or video model. Not worth a rewrite of a text-only deployment today.

And for anyone who wants to understand PagedAttention by reading code instead of a blog post, nano-vLLM is the shortcut: a from-scratch reimplementation of vLLM’s core ideas in roughly 1,200 lines of Python, built by a DeepSeek engineer as a teaching exercise, that gets within range of vLLM’s own throughput on small models despite the size difference. It will not replace vLLM in production. It is the fastest way I know to actually understand what PagedAttention is doing, in an afternoon instead of a week.

What I’d tell you over coffee

If a CTO asked me how to run an open-weight model in production right now, I would not start with vLLM flags. I would start with the prompt volume, because most of the time the honest answer is that the hosted endpoint is fine and the sovereign, control, and compliance reasons to bring it in-house have to carry the decision on their own merits, not on a cost story that does not survive contact with the utilization numbers.

When self-hosting is genuinely the right call, the work is calmer than the breathless version makes it sound and harder than the casual version admits. Size the box for weights plus cache plus overhead, not weights. Pin the serving-engine version. Choose the quant deliberately and test it on the one task that actually matters. Load-test the cache, not just the weights. And run the break-even before signing anything, because the cheapest deployment is still the one a team correctly decided not to run.

That is the whole trick. It is figure-out-able. It just rewards the people who do the sizing before they do the buying.