Llama cpp vs Ollama: the decision hiding under a 3 percent speed gap

This week handed me the cleanest side-by-side I have seen in months. On July 16, Ollama shipped exactly one release, v0.32.1: improved Gemma 4 tool calling, a fix for an MLX model cache leak, better sign-in prompts for its agent. In the same 48 hours, llama.cpp shipped thirteen build-tagged releases, b10043 through b10056: CUDA virtual devices, server slot save and restore, tensor-parallel fixes, Vulkan synchronization, Hexagon cache rework. Same two days. Same underlying engine, because Ollama runs llama.cpp inside it. Two completely different ideas of what shipping software means.

That contrast is the actual answer to the llama cpp vs ollama question, and it is a better answer than the one most teams reach for, which is a throughput chart.

The 3 to 10 percent that starts the argument, and why it settles nothing

Every ollama vs llama cpp thread eventually produces a benchmark, so let me put the current one on the table. A July 6 benchmark from Markaicode ran both on an RTX 4090 and found llama.cpp at roughly 104 tokens per second against Ollama’s 95 to 110 on the same model and quant, with time-to-first-token under 200 milliseconds for both.

The same writeup puts the structural gap at llama.cpp “coming in 3-10% faster than Ollama on NVIDIA GPUs,” and names the reason: Ollama wraps the same llama.cpp inference engine in a Go server with a REST API and model registry. The overhead is the wrapper, not the math. At steady state the two are executing identical kernels on identical weights.

Here is the thing about a 3 to 10 percent gap: it is real, and it almost never decides anything. If the margin between self-hosting working and not working is 7 percent of throughput, the deployment was undersized before the engine choice came up. Compare that with the decisions the benchmark says nothing about, which is where the two projects genuinely diverge. This is also why the endless llama cpp vs vllm comparisons feel more decisive; there the gap at concurrency is measured in multiples, not percent.

Ollama MLX, llama-cpp MLX, and a 4096-token default that bites both ways

The MLX lane on Apple Silicon is a good place to watch the wrapper tradeoff up close, because this is where the two projects stopped sharing an engine. Ollama’s newer Apple Silicon path runs on MLX, and Tuesday’s v0.32.1 fixed, in the release’s own words, “a recurrent MLX model cache leak that could increase memory use across requests.” That is the wrapper deal in one changelog line. The convenience layer added a caching layer, the caching layer grew a leak, and the fix arrived on Ollama’s schedule. A team running llama-cpp MLX-adjacent setups (llama.cpp has its own Metal path) or raw llama-server never had this particular bug, and never had the convenience that caused it.

Defaults are the same story with the signs flipped. Both projects ship a 4096-token context window out of the box, which is the software equivalent of a demo car with a quarter tank. Fine for a first drive, quietly ruinous for an agentic workload that stuffs 20k tokens of tool definitions and history into every request and then wonders why the model forgot the system prompt. In llama.cpp the fix is a flag sitting in plain sight. In Ollama it is an environment variable a lot of teams discover only after the truncation has been silently eating context for a month.

There is a third variation of this pattern, and it is the one I would weight heaviest: model support timelines. When a new architecture lands, engine support and wrapper support are separate events. This stream watched Ollama ship its Qwen 3.5 parser five months after that model’s release, after months of tool calls being routed through the wrong template. Kimi K3 arrived this Wednesday, a 2.8 trillion parameter open MoE activating 16 of 896 experts, per llm-stats. Nobody is running that on a workstation. But the next runnable release will repeat the sequence: llama.cpp support first, on engine time, wrapper polish later, on product time.

Docker images, install paths, and the update cadence being signed up for

The llama-cpp install story used to be the strongest argument for Ollama, and it has quietly weakened. llama-server ships prebuilt binaries, a maintained llama-cpp docker image, and since December, something that removes most of the remaining gap: router mode, which the ggml team explicitly described as answering “a popular request to bring Ollama-style model management to llama.cpp.” Auto-discovery of downloaded GGUF files, on-demand loading, LRU eviction when the model count passes a limit, and each model in its own process so one crash does not take down the rest.

So the convenience gap narrowed from the llama.cpp side. What did not narrow is the cadence gap, and this is the commitment I think teams actually sign when they choose.

Thirteen releases in two days is not a changelog, it is a weather report. llama.cpp does not do semantic versions; it ships a numbered build for nearly every merged change, and staying current means owning a version policy: pin a build, test against it, move deliberately. Ollama compresses that stream into a curated release every week or so and tests the bundle for its users. That is genuine value. The cost is visible in what those curated notes now contain: an interactive coding agent, a renamed ChatGPT integration, sign-in flows, auto-installation of coding tools. Ollama is becoming an AI application platform that contains an inference server. Anyone adopting it for the server is also adopting the platform’s roadmap, its deprecation nudges, and its opinions.

Two projects trading places while everyone benchmarks them

Step back from the week and the pattern is that the two projects are converging on each other’s old territory from opposite directions. llama.cpp absorbed the model management that made Ollama famous. Ollama’s release notes are increasingly about agents, sign-ins, and app integrations, with the engine work happening in the llama.cpp and MLX runtimes underneath. The comparison everyone still frames as a fork in the road is turning into a stack: an engine that wants to be everywhere, and a product that wants to own the desktop experience above it.

The benchmark measures the 3 percent that is the same. The release feed shows the 100 percent that is different.

For an infra lead making this call for a team, my honest read is that the decision reduces to one question: who should be deciding the operational details, the team or the project? Serving internal traffic with an SLA, wanting explicit control of quantization, batching, context, and GPU placement, and having someone on staff who reads release notes: that is llama-server, pinned to a build number, wearing its thirteen-releases-a-week cadence as a feature. Standardizing laptops for a dev team, running one shared box for internal experiments, valuing that a new hire is productive in ten minutes: that is Ollama, accepting that defaults and model-support timing arrive on the product’s schedule.

And the backdrop keeps this decision cheap to get wrong and cheap to fix. H100 rental averaged $3.78 an hour across 48 providers on GetDeploying’s July 17 snapshot, GPU pricing is drifting rather than spiking, and both tools read the same GGUF files. This is not a database migration. Swapping the serving layer is a weekend, not a quarter.

If I were deciding this over coffee: run both against the same 30 to 50 examples from the real workload for one week, at the real context length, not the default. Not to measure tokens per second. To find out which project’s opinions the team disagrees with less. The engine is shared, the weights are shared, the speed is within noise of shared. What is being chosen is whose release notes get read every Monday. Choose the ones the team will actually read.