The vLLM Data Parallel Question Your Board Should Ask Before the Next GPU Order

I saw a number this week that should make any board pause before signing off on a second GPU: a team ran a 7-billion-parameter model on one RTX 4090 at 3,965 tokens per second, added a second identical card, and watched throughput drop to 1,796 tokens per second. Not flat. Down by more than half. The card wasn’t broken and the model wasn’t the problem. The parallelism mode was wrong, and vLLM quietly used the second GPU in a way that made things worse, not better.

DatabaseMart, an infra vendor that ran the test on its own hardware, put it plainly: “Even if TP=1, vLLM may still use multiple cards for pipeline parallelism.” That single sentence is worth more to a board than any slide about “scaling our AI infrastructure.”

Why a second GPU can make inference slower, not faster

vLLM has three ways to spread a model across GPUs, and they answer different questions. Tensor parallel (TP) splits one model’s weights across cards, which only helps when the model is too big to fit on one GPU in the first place. Data parallel (DP) puts a full copy of the model on each card and routes different requests to different copies, which helps when the model already fits but you need more concurrent throughput. Pipeline parallel (PP) splits the model into stages across cards and is what vLLM falls back to when a tensor-parallel setup isn’t configured cleanly, which is exactly what happened in the benchmark above.

Two independent writeups I read on the same question this week (GIGAGPU and Kentino, neither carries a clean publish date but both agree with the dated benchmark above) describe the mechanism the same way: data parallel gets close to double throughput with no cross-card chatter, because each card just answers its own requests. Tensor parallel “shouts across the bus on every layer,” as Kentino put it, and only earns its keep once the model genuinely can’t fit on one card. If a model already fits on a 24GB or 48GB card, tensor parallel is usually the wrong tool, and data parallel or simply running two independent single-GPU servers behind a load balancer is the cheaper, faster answer.

The question isn't "do we need more GPUs." It's "does the model need to be split, or does the traffic need to be split." Those are opposite answers with the same hardware bill.

What board members actually need to hear about the GPU order

Three questions decide whether a second card is a good purchase, and none of them are “how much VRAM does the model need.”

First: does the model actually exceed the memory of the card already on hand? If yes, tensor parallel is the right tool and a second card buys real capability. If no, the honest move is data parallel or simply doubling up single-GPU instances, and the “why is our new GPU slower” postmortem gets skipped entirely.

Second: what is the sustained concurrency the workload needs? Data parallel earns its cost only when there’s enough simultaneous traffic to keep both copies busy. A GPU sitting at 15% utilization because there was never enough traffic to justify a second copy is a familiar and expensive mistake, and it has nothing to do with which parallel mode was chosen.

Third: what does the market actually charge for the card being requested? The live pricing tracker I check for this stream showed H100 at $3.46 an hour across 48 providers as of today, and B200 at an average of $5.95 an hour with spot instances as low as $2.14, per GetDeploying. Worth a small aside: that same tracker showed H100 prices climbing through most of the last month and now shows a 4% year-over-year dip, on the same page, using the same methodology. Live pricing trackers move around with how they average spot and on-demand listings. Treat the plain dollar figure as the signal and the percentage change as noise you check twice before repeating in a board deck.

The one-line version for the board deck

Before approving a second GPU: confirm in writing whether the model needs to be split (tensor parallel, buy for capability) or the traffic needs to be split (data parallel, buy for throughput). If nobody on the team can answer that in one sentence, the purchase order isn’t ready yet, and that’s a cheaper problem to catch in a meeting than in a benchmark six weeks later.

What to watch next

Ollama shipped version 0.32.0 on July 13, adding native support for Qwen 3.5’s parser and renderer and nudging users off older agent models with deprecation warnings. It’s a small release, but it’s the same pattern this stream keeps flagging: serving-engine support for a model architecture lags the model itself, and the parallelism question sits on the same shelf. Before the next GPU line item goes to the board, get someone on the infra team to say out loud which of the two purchases this actually is. It takes five minutes and it’s the cheapest test you’ll run all quarter.