Ollama or vLLM: which serving engine should run your model

Here is the number that surprises most teams the first time they benchmark it for themselves: at one concurrent user, Ollama answers faster than vLLM. Time to first token, roughly 45 milliseconds against 82. The engine everyone files under “toy” beats the engine everyone files under “grown-up,” as long as only one person is talking to it at a time.

That single fact is where a lot of serving-engine decisions quietly go wrong. The story we tell about Ollama and vLLM is a story about seniority, and a GPU does not care about seniority. It cares about how many requests arrive at the same moment.

The myth that you graduate from Ollama to vLLM

There is a belief, and I have repeated a softer version of it myself, that Ollama is where a project starts and vLLM is where it goes once things get serious. Prototype on Ollama, then earn your way up to vLLM when you have real traffic. It gets phrased as maturity. As if the simpler engine is training wheels and the harder engine is what adults ship.

The framing is comfortable because it turns a technical decision into a status decision, and status decisions are easy. Pick the thing that sounds more production-grade and nobody in the architecture review will question it.

The problem is that the ladder does not exist. Ollama and vLLM solve two different shapes of problem, and neither one is a junior version of the other. One is tuned for a single stream and a fast start. The other is tuned for many streams sharing a GPU. Deciding between them by seniority is like choosing a vehicle by how impressive it looks in the parking lot instead of by what you need to haul.

Why the throughput charts make vLLM look like the only adult in the room

The reason the myth sticks is that the charts people share are real. They are just measured at one end of the range. Almost every comparison that goes viral is run at high concurrency, where vLLM is built to win, and the gap looks enormous.

As Particula Tech put it in a March benchmark writeup: “vLLM achieved 8,033 tokens per second compared to Ollama’s 484, a 16.6x throughput advantage.” That is not a marketing number, it is a fair result for the workload it describes: a batch of concurrent requests on a modern GPU, exactly the case continuous batching was designed for.

The architecture explains the whole thing. vLLM uses continuous batching and PagedAttention, so it packs many active sequences into shared GPU work and pages memory in and out as requests come and go. Ollama, which sits on top of llama.cpp, processes requests through a simpler queue and allocates memory statically per model. At one request that simplicity is an advantage. At a hundred requests it becomes the bottleneck. Same GPU, completely different behavior, entirely because of how each engine schedules work.

Where Ollama wins and where vLLM pulls ahead

The honest version of the vLLM vs Ollama comparison is not a ranking, it is a crossover. The two lines start close, touch, and then separate as load climbs.

Read the table top to bottom and the decision writes itself. If the model serves one person at a time, an internal tool a handful of colleagues poke at, a batch job that runs one prompt after another, Ollama is not the compromise. It is the correct answer, and it will have the team serving a model in about five minutes with model management built in.

Cross into sustained concurrency and the picture flips hard. By eight to ten simultaneous requests vLLM is already two to three times ahead on total throughput, and at fifty users Ollama flattens out near 155 tokens per second while vLLM holds close to 920 with far better tail latency. A peer-reviewed evaluation in Applied Sciences this spring found the same pattern across scenarios: the engines are close until concurrency rises, and then continuous batching decides the outcome. When independent benchmarks and an academic study agree on the direction, that is about as settled as this field gets.

SGLang and llama.cpp: the two engines the vs-posts skip

Framing the whole question as Ollama against vLLM also hides the other two names that belong in the room.

SGLang is not a step above vLLM, it is a sibling for a specific workload. On standard traffic it runs modestly ahead, roughly 16,200 tokens per second against vLLM’s 12,500 on an H100 in one 2026 comparison, thanks to prefix reuse. That edge nearly vanishes on unique-prompt batch jobs and can balloon on prefix-heavy work like retrieval pipelines and multi-step agents that share a long system prompt. SGLang earns its place when a team is already at vLLM scale and the workload reuses prefixes. Below that, it is complexity without a payoff.

llama.cpp is the layer underneath Ollama, which means the “vLLM vs llama.cpp” debate is mostly a hardware question wearing a software costume. Datacenter GPU with steady concurrency points to vLLM. CPU, Apple Silicon, a mixed or edge box points to the llama.cpp family, and Ollama is the friendly way to run it. That is a sizing decision about the metal on hand, not a verdict on which project is more serious.

The real trigger to switch is a concurrency number

Here is the reframe worth carrying into the next planning meeting. The signal to move off Ollama is not that the product got important. It is one of three measurable conditions: sustained concurrency above roughly four or five simultaneous requests, a rig with more than one GPU to fill, or a hard tail-latency promise you have to keep. Hit one of those and vLLM’s batching stops being overkill and starts being the reason the service stays responsive. Miss all three and the simpler engine is the right engineering call, not a placeholder.

The serving engine is not a seniority ladder. It is a match to how many people hit the model at the same instant.

Notice that every trigger is a number, not a feeling. A counter on concurrent requests is easy to add, and a tail-latency chart is easy to read. That turns a debate that usually runs on vibes into a decision an infra lead can defend in one sentence.

Pick by the load, and expect to run both engines

The tidy takeaway is that most teams do not choose one engine, they end up running two: Ollama for local development and low-concurrency internal tools, vLLM for the multi-user production path. That is not indecision, it is matching the tool to the load in each place. The average “we’ll just standardize on vLLM everywhere” plan has a slide for throughput and no slide for the developer who now waits on a heavier server to test a one-line change.

Nothing in the market this week forces the decision either way, which is a small gift. No open-weight model or serving engine shipped a release inside the last few days, and the GPU rental side barely moved: GetDeploying’s tracker put the average H100 around 3.37 dollars an hour across 47 providers on July 5, up about 4 percent on the year. When neither the models nor the hardware are lurching, the serving choice stops being a reaction and becomes what it should have been all along, a calm read of how many requests actually arrive at once.

So measure the concurrency you really have, not the concurrency the pitch deck imagines. Then let that number pick the engine, and let the other engine keep doing the job it is better at.