The Best Local LLM for Coding Is the One That Fits Your GPU, Not the Leaderboard

By the Cerevisor team. Benchmark figures are dated where they appear and come from the June 2026 writeups listed under Sources. The setup section was last checked against the published Cerevisor docs on 5 October 2026.

Infra leads keep having the same conversation. It starts with “we want to self-host a coding model” and within ten minutes it has drifted into which open-weight model just topped the SWE-bench chart. The trouble is that the model topping the chart and the model a team can actually run on the hardware it already owns are almost never the same thing, and the gap between them is where a quarter goes to die.

Here is the number that makes it concrete. The open-weight coding leader in the June snapshot, GLM-5.2, is a 753-billion-parameter mixture-of-experts model. A guide from Pinggy, updated June 18, clocks it at 79.65 on the LiveBench Coding Average, the highest open score they list. It also needs a multi-GPU rig to run. That is the whole tension in one model: the best score on the board belongs to a thing a single workstation cannot load.

So this page walks through how to pick a local coding model, the named models that go with each decision, and how to wire the winner into a coding agent. The scores come from benchmark writeups dated between February and mid June 2026, and each one carries its date. The durable part, the method for choosing, does not change when a new model drops. The leaderboard reshuffles. The method holds.

Start with the card on hand, not the model on the wishlist

The first move is boring and it is the one teams skip: write down how much VRAM the box actually has, then pick from the models that fit. VRAM is the memory on the graphics card, and it is the limit that decides what can load. Everything else is downstream of that one number.

The rough math is worth memorizing because it sizes any model in a few seconds of mental arithmetic. At FP16 a model needs about 2 bytes per parameter, so a 70B model wants roughly 140GB. At INT4 it drops to about half a byte per parameter, so that same 70B lands near 35GB. Then add 10 to 20 percent on top for the KV cache, activations, and framework overhead. The KV cache is the working memory a model keeps for the conversation so far, and it grows with context length. Pinggy and InsiderLLM (updated May 26) both land on the same tiers from there: 8GB of VRAM covers 7B-to-8B models, 24GB is the practical floor for a 30B-class model, and 40GB-plus is the territory for 70B unless the team quantizes hard.

That single calculation kills most of the bad decisions before they happen. It says, before anything has been downloaded, whether the exciting model is a download or a hardware purchase order.

The 5 open-weight coding models worth shortlisting, by the card they fit

These are ranked by the hardware they run on, not by raw score. Each one is grounded in a specific benchmark from the June writeups, with the date attached.

1. Devstral Small 2 (24B) for agentic coding on a single card

   This is the default pick for a team that wants multi-file edits and debugging loops on one GPU. Pinggy (June 18) puts it at 68% on SWE-bench Verified, running on a single RTX 4090 or a 32GB Mac. It was built with All Hands AI specifically to drive coding agents, and Apache 2.0 means the license will not surprise a legal team. One good GPU plus a desire for an agent: start here.

2. Qwen 3.6 27B for the strongest dense model on 24GB

   The dense 27B is the best "fits one 24GB card" coder with published numbers. InsiderLLM (May 26) and Pinggy (June 18) both report 77.2 on SWE-bench Verified at roughly 17 to 22GB depending on quant and context. InsiderLLM's own line is that this "puts the 27B in the same range as Claude Sonnet." Every parameter activates per token, so reasoning stays consistent. This is the one to reach for when the task is real code, not autocomplete.

3. Qwen3.6-35B-A3B for MoE latency on the same card

   For a team that likes the idea of a larger model but cannot afford the latency of a dense one, the 35B mixture-of-experts variant activates only 3B parameters per token. A DEV Community writeup (June 8) reports 73.4 on SWE-bench Verified and notes it "fits on a single RTX 4090 or M-series Mac with 32GB." The payoff is a big-model knowledge base with small-model speed, which is the entire reason MoE exists.

4. Codestral 22B when the job is autocomplete, not agents

   This is the one teams forget, because it does not win the headline benchmark. The DEV writeup (June 8) puts Codestral 22B at 95.3% fill-in-the-middle pass@1 at about 14GB in Q4_K_M. Fill-in-the-middle is what powers inline autocomplete, and it is a different job from agentic editing. When developers mostly want fast, accurate tab-completion in the editor, the agent-tuned models are the wrong tool and this is the right one.

5. Qwen3-Coder-Next (80B / 3B active) as the worked example of model-to-hardware sizing

   This one is older, from a February DEV guide, but it is the cleanest worked example of fitting a model to a specific box. The 80B MoE activates 3B per token at 256K context under Apache 2.0, scores 44.3 on SWE-bench Pro, and the guide pairs it with an RTX 5090 32GB plus 128GB of DDR5 to hit 30 to 40 tokens per second at full context. Notice the pattern: the answer is not just a GPU, it is a GPU-and-system-RAM split. MoE models lean on that split, and budgeting for it is half the sizing job.

Why most teams reach for the model they cannot run

The mistake is not stupidity. It is that the leaderboard is the most visible artifact in the whole space, so it becomes the default proxy for “good.” And the top of that board is genuinely impressive. The same Pinggy snapshot that crowns GLM-5.2 also lists Kimi K2.6, a 1-trillion-parameter MoE at 78.57 LiveBench Coding, needing two H100 80GB cards or four A100s and 512GB of RAM, and DeepSeek V3.2 at 75.69, also multi-GPU. These are real, excellent models. They are also operational commitments most teams have not budgeted for.

Here is the part that does not fit on a leaderboard: the three-point gap between the 79.65 model nobody can run and the 77.2 model that fits one card is almost never the thing that decides whether a coding workflow works. What decides it is whether the model is loaded, fast, and available when a developer hits tab. A model that scores three points higher and runs on hardware the team does not own scores zero in production.

A model that scores three points higher and runs on hardware the team does not own scores zero in production.

And “best” splits by task in a way a single number hides. Agentic multi-file editing and inline fill-in-the-middle autocomplete are different jobs, and the model that wins one can lose the other. The 95.3% FIM number for Codestral is meaningless for an agent workload, and the agentic scores are meaningless for developers who just want fast autocomplete. The leaderboard collapses both into one ranking. The real workload does not.

For honesty’s sake, here is one of these scores in its original words, describing GLM-5.1.

That is a real, specific, attributable number, and it is exactly the kind of line that pulls a team toward a model it has no way to serve. The number is true. The conclusion most people draw from it is the trap.

How to prove a local coding model is good enough for your codebase

Once a shortlist fits the card, the move is to stop reading benchmarks and start measuring the actual task. Public coding benchmarks show that a model can code in general. They say nothing about whether it can code this codebase, with these conventions, against this test suite.

Build a standing set of 30 to 50 real examples from the team’s own repository: tickets the model should be able to close, functions it should complete, bugs it should find. Run the shortlisted models against that set at the quant actually destined for production, not the full-precision weights the benchmark used. A quant is a compressed version of the model that trades a little accuracy for a much smaller memory footprint. Measure pass rate, but also measure the things that quietly break first, like whether tool calls come back well-formed and whether multi-file edits stay coherent. The gap between a model’s headline SWE-bench score and its score on those 50 examples is the only gap that pays the bills.

Then watch utilization and tokens per second under real traffic. A model that clears the quality bar but serves at single-digit utilization is a model the team is overpaying to run. The point of self-hosting was the unit economics, and the unit economics only show up when the card is busy.

Serving the model: Ollama, vLLM or LM Studio

A shortlisted model still has to be served, which means running it behind an address that other tools can call. Three local servers cover most teams, and they have one useful thing in common: each can expose an OpenAI-style chat endpoint, so a coding agent that speaks that format can talk to any of them without custom code.

• Ollama is the smoothest path on a single workstation. Install it, pull a model, and it serves on port 11434.
• vLLM is the choice when a shared GPU box has to serve several developers or several agents at once, because it is built for throughput rather than a single user.
• LM Studio suits an individual who wants a desktop app with a model picker and a local server switch.

The choice follows the same logic as the model choice. One developer on one card wants the simplest server. A team sharing a rack wants the one that batches requests well. For a deeper comparison of serving engines, see the notes on open-weight model serving in production.

Connecting a local coding model to an agent, with the limits that show up

Picking and serving a model is half the work. The other half is giving it a job inside a workflow, and that is where local models show their real constraints. The Cerevisor documentation for OpenAI-compatible and local providers spells out the setup, and it is worth reading for the specifics rather than taking a summary on trust.

Setup is six moves: add a provider of the OpenAI-compatible type, name it, paste the address, leave the key blank if the server needs none, test the connection, and pick a default model from the list the server reports. Cerevisor confirms the connection by asking the server for its model list, so a model that has been pulled shows up in the picker, and a newly pulled one appears after a refresh. The full steps are in the local models guide, and the broader product picture is on the Cerevisor product page.

Three behaviours from that guide matter for anyone choosing a coding model, and none of them is flattering to local hardware.

First, a single machine takes turns. The docs say that for multi-agent workflows on one machine, agents take turns even when a step is set to run in parallel, because the model is the bottleneck. A team planning four agents against one 24GB card should expect them to queue, and should prefer the sequential execution setting rather than pretending the parallelism is real.

Second, give small local models the forgiving roles. The same guide suggests lighter jobs such as researcher and reviewer for small models, and that matches what the benchmark gap says: a model in the 24B to 27B range is a good fit for focused work with clear tests, and a poorer fit for open-ended orchestration.

Third, local runs report token counts but no price. Cerevisor shows $0.00 for these runs and badges the entry as Local. That is accurate as a cash figure and misleading as a cost figure, because the card, the power and the machine’s time are real. Compare a local run against a hosted one by tokens per second and utilization, not by the zero on the invoice.

The most common failure is also the most mundane: a connection refused error on port 11434 means the server is not running, and the fix is to start it. If the model list comes back empty or a request returns a 404, the address is usually missing a path prefix. These are setup problems, not model problems, and they are worth ruling out before blaming the model.

What to do Monday morning

Write the VRAM number on a sticky note. Cross off every model that does not fit it, including the one at the top of the chart, especially the one at the top of the chart. From what is left, pick by task: Devstral Small 2 or Qwen 3.6 27B for a coding agent on one card, Codestral 22B for autocomplete-heavy editing, the 35B MoE for a bigger brain without the latency tax. Serve it with the simplest server that fits the team. Then run those 50 examples against the quant headed for production before anyone calls it done.

The leaderboard will have a new champion by next month, and it will probably be one that still cannot run on the card on the desk. That is fine. The model that ships features this quarter is the one that fits, clears the task bar, and stays loaded. Nothing humbles a roadmap faster than a 753B model and a single 24GB card, and nothing speeds it up like admitting that the 27B was always enough.