vLLM vs llama.cpp: do you actually need vLLM to self-host?

I keep meeting teams who treat the serving engine like a rite of passage. They start on llama.cpp or Ollama, it works fine, and then someone in a planning meeting says the quiet part out loud: “sure, but that’s not really production. We need vLLM.” The unspoken belief underneath that sentence is that llama.cpp is the toy and vLLM is the grown-up, and that moving a self-hosted model to production means moving off the first and onto the second.

Here is the line that quietly breaks the hierarchy. Ollama shipped v0.31.2 on July 8, and buried in the release notes, right next to the flash-attention fix for older NVIDIA cards, is this: it “Updated the MLX and llama.cpp engines.” llama.cpp is not the thing you graduate away from. It is the engine running underneath the easy button that half the market already uses in production.

Why vLLM’s throughput numbers make the choice look obvious

The myth earns its believers honestly, because the benchmarks that feed it are real and lopsided. Once concurrent users climb past five or six, vLLM pulls ahead sharply. A concurrent stress test written up by D-Central in June clocked vLLM at roughly 793 aggregate tokens per second against about 41 for a single-stream path under the same simultaneous load. That is not a rounding difference. That is a different category of machine.

The reason is architectural, not marketing. vLLM’s PagedAttention and continuous batching are built to keep an expensive GPU saturated while dozens of requests overlap. If your mental model is “more throughput is strictly better,” vLLM wins every slide in the deck, and it deserves to.

What the benchmarks show once you count concurrent users

Now the part that never makes the slide. For a single request, the engine everyone calls a toy is often the faster one. Markaicode’s July benchmark on an A100 80GB ran QwQ-32B through both: llama.cpp finished in 1 minute 42.7 seconds, vLLM in 2 minutes 7.4 seconds. Continuous batching is overhead you agree to pay so the machine can serve many requests at once. With exactly one request in flight, there is nothing to batch, and that overhead is just tax.

Then there is the format trap, which is where a lot of “we moved to vLLM and it got slower” stories actually come from.

If a team is committed to GGUF weights, llama.cpp is the home field where GGUF is native and tuned. Dropping those same weights into vLLM throws away most of what vLLM is good at and keeps all of its operational weight. The honest other side holds too: run the right native format at real concurrency on a GPU and vLLM wins by multiples, not percentages. Both facts are true at the same time, which is exactly why the blanket “vLLM is production, llama.cpp is not” framing falls apart.

For a single request, the engine everyone calls a toy finished faster than the one everyone calls production.

llama.cpp vs vLLM is a concurrency line, not a seniority ladder

Here is the mental model that survives contact with the benchmarks. The serving engine is a match to the shape of the traffic, not a badge of how serious the team is. One user at a time, CPU or Apple Silicon or mixed hardware, or a portable single binary that just runs on whatever box is available: that is llama.cpp’s territory, and Red Hat’s own engineering writeup lands in the same place. Sustained multi-user load on a GPU, an OpenAI-compatible endpoint, Kubernetes-native scaling: that is where vLLM stops being overhead and starts being the point.

The fresh release this week is a clean test case. Tencent put out Hy3 on July 6, an open 295B mixture-of-experts model with 21B active parameters, a 256K context window, and Apache 2.0 weights. That shape, large memory footprint but low active compute per token, is precisely where the engine question bites. Both engines can plausibly serve it. The deciding factor is not which one is “real.” It is how many people hit the endpoint at the same moment.

What to run this week for your actual load

Count the real number first. Not the number of registered users, the number of requests genuinely in flight at once during a busy minute. For a lot of internal tools, batch jobs, and single-founder prototypes, that number is one, and the whole vLLM migration is solving a problem the traffic does not have.

The cost backdrop reinforces the point. GetDeploying’s tracker had the average H100 at $3.41 per hour on July 9, with on-demand rates up about 14 percent over the past year. The GPU a team rents to justify vLLM’s throughput keeps getting more expensive, so paying for batching that mostly sits idle is a worse deal every quarter. Most healthy setups I see run both without drama: llama.cpp or Ollama for development, low-concurrency internal use, and edge boxes, then vLLM the moment a specific workload proves it lives above the concurrency line. That is not indecision. That is matching the tool to the traffic, which is the only version of this question that has ever had a real answer.