The vLLM-Ascend question your board should ask before the next GPU order

NVIDIA’s rental meter keeps climbing, and the alternatives finally boot

On July 6 the live GPU rental trackers had NVIDIA’s H200 sitting at $4.24 an hour on-demand across 31 providers, up about 8 percent since last June, and the H100 at $3.88 across 47 providers, up 4 percent. Nothing dramatic. Just the quiet fact that the meter keeps ticking up, not down, while the memory needed to own the box got pricier too. That is the standing pressure behind a question I am hearing more often in board rooms: does the company have to keep signing NVIDIA purchase orders, or is there finally a credible second supplier.

This year, for the first time, two non-NVIDIA answers actually boot and serve real traffic. AMD’s Instinct line on ROCm, and Huawei’s Ascend reached through the vLLM-Ascend backend. Twelve months ago I would have filed both under science project. I cannot anymore, and the reason has almost nothing to do with the chips themselves.

The real unlock is the serving engine: ROCm and vLLM-Ascend

Here is the part the hardware press skips. What made non-NVIDIA inference viable is not a chip, it is the software layer that hides the chip. vLLM now serves on NVIDIA, AMD ROCm, Intel Gaudi, Google TPU, Apple Silicon, and Huawei Ascend from broadly the same serve command. SGLang and LMDeploy cover a similar spread. The model, the API, and the operations runbook stay mostly put while the silicon underneath changes.

On the AMD side that layer got serious fast. The ROCm continuous-integration suite for vLLM went from 37 percent of tests passing last November to 93 percent by January, with a prebuilt Docker image and pip wheels to match. An MI300X carries 192GB of HBM3, so a 70B-class model fits on one card without the multi-GPU sharding an 80GB H100 forces. The honest catch, from a hands-on ROCm writeup this spring, is that it still runs roughly 10 to 25 percent slower than CUDA on equivalent hardware, and it strongly prefers Linux.

On the Ascend side the specs are real and the ecosystem proof is stronger still.

The credibility signal I keep pointing boards to is not a benchmark, it is a calendar. When DeepSeek V4, a 1.6-trillion-parameter model trained on Ascend, launched in late April, Huawei Ascend, Cambricon, and Hygon all shipped day-zero inference support the same afternoon. That same-day readiness used to be an NVIDIA-only privilege. With Barclays estimating roughly 70 percent of 2026 compute demand is inference rather than training, that is precisely the workload where the non-NVIDIA gap is narrowest.

Three questions before you sign for non-NVIDIA silicon

Does it actually lower our cost per token? Sometimes, and rarely for the reason people expect. AMD’s real argument is memory per dollar: one rental tracker lists the MI300X at $1.44 to $5.20 an hour against an H100 range of $2.01 to $11.06, and the 192GB means fewer cards per model. But the 10 to 25 percent throughput gap eats into that, and below the real token break-even the hosted API is still cheaper than any box, NVIDIA or not.

Is the bet reversible if we are wrong? This is the question that should calm the room. Because vLLM, SGLang, and LMDeploy abstract the backend, moving between NVIDIA, AMD, and Ascend is a supplier change, not an application rewrite. The same model can stand up on two backends at once, which keeps the option open instead of locking a year of roadmap to one vendor’s supply.

Do we even need it? Two clean reasons, and a lot of noise around them. If sovereignty or Chinese supply is the binding constraint, Ascend is the serious answer. For teams that want cheaper VRAM at real scale and can run Linux-first operations, AMD is. If neither is pressing, NVIDIA plus the API is a perfectly respectable place to stay another quarter.

The one-minute answer: a supplier choice, not a rewrite

If the conversation gets sixty seconds on the agenda, say this. NVIDIA rental is drifting up, not down, so a second supplier is worth having on the shelf. Two now exist that serve production traffic: AMD on ROCm for cheaper memory, and Huawei Ascend through vLLM-Ascend for sovereignty and supply. Both trail NVIDIA by 10 to 25 percent and both ask for extra operations effort. The reason to explore now rather than later is that the serving layer makes the choice reversible, so a pilot risks engineering time, never the roadmap.

A second inference supplier used to mean a rewrite. Now it means a config change and a Linux box, which is a very different risk to put in front of a board.

The signals that would move this from optional to obvious

Watch three things, none of them urgent. Whether the ROCm-to-CUDA performance gap keeps closing past this year’s 10 to 25 percent. Whether AMD and Ascend show up in the same public rental trackers that price NVIDIA today, because a listed hourly rate is the moment a chip stops being a procurement adventure. And whether sustained token volume ever crosses the line where owning any of this beats renting. Until one of those moves, a quiet two-backend pilot is the whole assignment. No rip-and-replace, no drama. Just a second key cut for a door no company wants a single vendor holding.