KV cache quantization: how far to compress before long context breaks

An infra lead I know spent a morning last week convinced a rented H100 was broken. The model, a 70B, loaded with room to spare. Chat worked. Then a customer pasted a 90,000-token contract into the support flow and the whole thing fell over with a CUDA out-of-memory error. The weights were never the problem. The context was. The box that loads a model with plenty of headroom is the same box that dies the moment someone hands it a real document.

Here is the odd thing about the week that happened in. Nothing shipped that would have helped. The open-weight release tracker at llm-stats read “no open source releases this week.” The freshest serving-engine update, Ollama v0.31.1 on July 1, was a speed change for Apple Silicon, not a memory one. And the rental market barely moved: GetDeploying’s July 3 snapshot has the H100 averaging $3.47 an hour across 46 providers, up only about 4% on the year. When no new model lands and the rental line is flat, the lever an engineering leader actually reaches for is inside the box already paid for. For long context, the setting that moves the most is KV cache quantization.

Why long context eats the VRAM the weights already fit

The KV cache is the model’s running memory of the conversation. Every token it has seen stores a key vector and a value vector in every layer, so the cache grows linearly with context length and never shrinks until the request ends. Digital Applied laid the arithmetic out cleanly in late June: the cache is roughly 2 times layers times KV-heads times head-dim times tokens times bytes-per-element. For a Llama 3.1 70B that works out to about 41.94 GB of cache at 128K tokens in BF16, and about 327 GB at a million tokens, all of it sitting on top of the roughly 40 GB of weights.

That is why the sizing that looked comfortable on the model card is a trap. Weights are a fixed cost you pay once at load. The cache is a variable cost that scales with how long your users talk, and long-context traffic is exactly what people self-host a long-context model to serve.

The fix is not a bigger card. It is storing each cell of that cache in fewer bits. FP8 or Q8 halves it. INT4 or Q4 quarters it. The flags are refreshingly boring: llama.cpp takes --cache-type-k and --cache-type-v, vLLM takes --kv-cache-dtype fp8, and Ollama reads a single OLLAMA_KV_CACHE_TYPE environment variable. One line, and the cache table shrinks.

ScaleMindLabs framed the same move in headroom terms back in May, and the headroom is what makes it land. An 8B model on an 80 GB card, reserving 45 GB for cache, holds roughly 258,000 safe tokens at FP16, about 516,000 at FP8, and about 971,000 at INT4. Same silicon, close to four times the live context, decided by one setting.

Q8 is nearly free; Q4 is a loan the model repays on long context

Start with the good news, because most teams leave it on the table. Halving the cache with Q8 or FP8 costs almost nothing you can measure. ModelPiper’s tests put the q8_0 hit at a rounding error: “Benchmark perplexity increases by 0.002 to 0.05 depending on the model and context length.” Digital Applied reached the same conclusion from the other direction, noting that an FP8 cache “halves every cell in the table above and an INT4 KV quarters it, often with negligible quality cost on long-context retrieval.” And FP8 is not only cheap on quality, it can be faster: the vLLM team’s April writeup on FP8 attention reported the inter-token-latency slope dropping to 54% of BF16 and output throughput on Llama-3.1-8B rising 14.9% at concurrency 8. A near-free quality trade that also serves more users is about as close to a free lunch as this stack offers.

Then there is Q4, where the average benchmark still looks fine, which is precisely the trap. Two things are worth knowing before flipping that switch.

The first is that the damage is asymmetric. The value cache carries more of the output signal than the key cache, so quantizing both to Q4 by reflex is the wrong default; if pushed for memory, compress the keys harder and keep the values gentler. The second is that the damage concentrates on the workloads people self-host for. A May test at dasroot on long-horizon agentic tasks found that moving from full precision to a 4-bit cache cost about 6 to 7% accuracy, and 2-bit cost 15 to 20%, with the errors compounding across steps. A short-context benchmark would have shown none of it.

None of these numbers are this week’s news, and I want to be straight about that. The freshest careful writeup I found on KV cache quality was late June, and the behavior underneath has been stable for months across a stack of independent tests that all agree on the shape. That is actually the calming part. This is not a moving target. It is a known trade with a known failure mode.

Treat the KV cache like weights: Q8 by default, Q4 on trial

The discipline here is the same one that governs weight quantization, one level down. Q8 or FP8 is the safe default. Take it everywhere, today, and move on. Q4 on the cache is not a default; it is a decision earned with evidence.

The evidence is cheap to gather. Assemble 30 to 50 real examples of the longest, most retrieval-heavy task in the product, run them at the exact cache precision headed for production, and compare the outputs against the Q8 baseline. If Q4 holds, bank the memory and the extra concurrency. If it wobbles on retrieval or multi-step reasoning, keep the value cache at Q8 and only drop the keys. That single test is the difference between a memory win and a silent regression that surfaces as a support ticket three weeks later.

Q8 on the KV cache is close to free. Q4 is a loan the model repays on its longest, most important requests.

This matters more each quarter, and not because of any one release. Context windows keep growing while the memory to hold them keeps getting more expensive, so the cache, not the weights, is increasingly what decides how many users can share a GPU. The team that tunes the cache well fits more conversation on the same rented card than the team that only shopped for a bigger model.

What I’d set before the next long-context ticket

If I were sitting across the table from that infra lead this morning, I would say three things. Turn on a Q8 or FP8 KV cache today, because it is the closest thing to free in this whole stack and habit is the only reason it is ever left off. Leave Q4 in the drawer until a real eval on a real long task earns it, and when it does, quantize the keys before the values. And read the flat rental line for what it is telling everyone right now: the cheapest capacity available this quarter is not a new GPU and not a new model, it is the context already being paid to store and, on a lot of boxes, quietly wasted.

That is the whole trade. Halve the cache for nothing, and be honest about the one place the second halving bites.