vLLM speculative decoding: the free speedup with a batch-size catch

An infra lead I was comparing notes with last week flipped on speculative decoding in vLLM, watched a summarization job run almost three times faster in a quick test, and shipped it to production with a small celebration. Two days later the p99 latency on their chat endpoint was worse than before they touched anything. Same feature, same model, opposite result. Nothing was broken. Speculative decoding was doing exactly what it does, which is help on one shape of traffic and quietly tax another.

That gap is worth understanding this week specifically, because there is not much else to chase. The open-weight release feed has gone quiet again: llm-stats showed nothing new shipped over the last few days as of July 9, with GLM-5.2 from mid-June still the freshest frontier open weight. Meanwhile renting the GPU keeps getting more expensive, not less. GetDeploying’s tracker had the average H100 at 3.43 dollars an hour on July 10, up about 14 percent over the past year. When no new model is going to lower the bill, the lever actually in reach is the config on the server already running. Speculative decoding is the most tempting one, and the easiest to get wrong.

What “draft and verify” actually does per token

Normal decoding is a slow, patient loop. The model runs one full forward pass and produces one token, then does it again for the next token, and again. Most of that time is not spent computing. It is spent hauling the model’s weights from memory into the compute units, over and over. On a 70-billion-parameter model that memory traffic, not the math, is the bottleneck at low batch sizes.

Speculative decoding attacks that waste. A small, cheap drafter proposes several tokens ahead. The big model then verifies all of them in a single forward pass and keeps the ones it agrees with. When the drafter guesses well, the team gets three or four tokens out of roughly one expensive verification step instead of one. The drafter can be a compact model trained for the job, as with EAGLE, a set of extra prediction heads bolted onto the main model, as with Medusa, or nothing more than a lookup over recent text, as with n-gram or prompt-lookup speculation.

Here is the part that makes it genuinely appealing, and not just another knob. The verifier still has the final say. It only accepts tokens the full model would have produced anyway, so the output is identical to running without speculation. This is a rare optimization that changes the speed without changing the answer. That is also exactly why it lulls teams into switching it on everywhere.

Speculative decoding does not make the GPU faster. It trades spare compute for fewer memory round trips, and a saturated GPU has no spare compute to trade.

Where the 2.8x came from, and where it turned into a slowdown

The impressive demo numbers are real. A configuration and decision framework that DigitalOcean published in early July put the vLLM team’s own benchmarks plainly.

Read the conditions in that sentence, because they are the whole story. Low query rates. Structured, low-temperature output. Those are the cases where the GPU has compute to spare and the drafter can guess reliably. Change either condition and the math inverts.

The inversion is not subtle. The same writeup notes the same vLLM benchmarks show 1.4 to 1.8 times slowdowns once the GPU is saturated with concurrent requests.

The reason sits one layer down. At batch size 1 to 4, inference is memory-bound, so the verifier has idle compute that speculation puts to work for free. Pack the same GPU with dozens of simultaneous requests and it flips to compute-bound. Now every extra draft token the verifier has to check is competing for compute that paying requests already need. The speculation becomes overhead, and worse, it is overhead billed by the hour. At 3.43 dollars an H100-hour, a feature that adds 40 percent to latency under load is not a curiosity. It is a line item.

Acceptance rate is the number that decides it

If there is one metric to watch, it is acceptance rate: the fraction of drafted tokens the verifier keeps. It is the hinge the whole tradeoff turns on. The DigitalOcean framework draws the line clearly: below roughly 0.5, speculation is adding latency rather than removing it, because the verifier keeps paying to check guesses it then throws away.

The uncomfortable part is that acceptance rate is mostly not something a team controls. It is a property of the workload. The same framework reports acceptance around 0.81 at temperature 0.0 and around 0.38 at temperature 1.0 for the same setup. Deterministic, low-temperature output is predictable, so the drafter is usually right. Turn the temperature up for creative or open-ended generation and the drafter starts guessing wrong, each rejection costs a wasted verification, and the feature quietly becomes a tax.

That is what happened to the infra lead from the start. The summarization test ran at low temperature with one request at a time, so acceptance was high and the GPU was idle. The chat endpoint ran hotter, more concurrently, and more creatively. Nothing was misconfigured. The two endpoints simply had different acceptance rates, and only one of them cleared the bar.

Picking n-gram, EAGLE, or Medusa is a memory decision

Once a workload does qualify, the next choice is which drafter to run, and that decision is really about memory more than accuracy. A method guide from Local AI Master in May lined the options up.

n-gram speculation is the honest starting point because it costs nothing: no training, no extra weights, no VRAM. It just looks for repeated patterns in recent text, which is why it shines on code and structured output and shrugs on free-form chat. A trained drafter like EAGLE accepts far more tokens and delivers the headline 3.5 to 4 times gains, but it is a second model, and its weights have to live somewhere.

That “somewhere” is the catch most sizing spreadsheets miss. The DigitalOcean framework spells out that the draft model’s VRAM comes directly out of the KV-cache budget. In their worked example, a setup on two H100s spends about 86 GB on weights and leaves roughly 74 GB for the cache. Add a drafter and the cache shrinks, which lowers how many requests can run at once, which pushes the GPU toward the saturated regime where speculation stops helping in the first place. There is a real circularity there worth respecting. The same guide suggests keeping the drafter to roughly a 1-to-8 or 1-to-12 size ratio against the target model, from the same family, so the guesses stay cheap and well-aligned.

What I’d check before flipping it on

So, back to the decision. Speculative decoding is not a global switch to leave on, and it is not snake oil either. It is a workload-specific optimization with a clean test for whether it belongs.

Before you turn it on for an endpoint, I would ask three things. What is the real concurrency at peak, honestly measured and not hoped for? If that endpoint routinely runs the GPU hot, speculation will likely cost more than it saves. What does the output look like, deterministic and structured, or creative and high-temperature? And once it is live, what is the measured acceptance rate on real traffic, not a demo? If it sits below 0.5, switch it back off and move on.

The larger point keeps showing up across every one of these local-inference levers. The gains are real, the failure modes are just as real, and which one a team gets is decided by matching the technique to the traffic rather than by the technique itself. A quiet release week is a good time to make that match, because the fastest model on the leaderboard is not going to lower this month’s GPU bill. The server already running, tuned to the shape of its own traffic, might.