Which AI coding agent do you commit to when the best one keeps changing?

I pulled up a coding-agent leaderboard this week to settle a question a founder had asked me, and the top model on the SWE-bench Verified column, the one scoring 95 percent, had a little note next to it: export-suspended. As in, it cannot be run. The model with the single best score in the category is unavailable to most teams. The number-one result on the scoreboard is a model nobody can deploy.

That is the whole problem in one screenshot.

If a founder or CEO is reading this, they are probably being asked to bless a decision right now: standardize the engineering team on one AI coding agent, put real budget behind it, and tell the board there is a plan. The instinct is to pick the one with the best benchmark this quarter. I want to talk anyone with that instinct out of it, because the last three days are a clean demonstration of why it leads somewhere expensive.

The decision a founder is actually being asked to make

The question on the table sounds like “which AI coding agent is best.” It is really “which one does the whole org commit to, knowing it is a multi-quarter decision and the thing underneath it changes constantly.”

Those are different questions. The first one has an answer that expires in about two weeks. The second one has an answer that can actually hold.

Here is what I mean by “changes constantly.” In the past 72 hours, two of the biggest harness vendors each shipped a stack of releases. Anthropic moved Claude Code from v2.1.186 to v2.1.193 between June 23 and June 26. GitHub shipped Copilot CLI v1.0.64, v1.0.65, and v1.0.66 over the same window. Neither of those was a new model. Every one of those releases changed the controls around the model: what it can touch, what an admin can lock down, how teams observe it, which models a developer is even allowed to pick.

That is the tell. When the vendors themselves spend their release energy on the control surface instead of the model, they are pointing at where the durable part of the product lives. The commitment decision should follow theirs.

Pick the harness, not the model of the month

So here is the approach I would walk a founder through, and it inverts the usual order.

Most teams start with “which model scores highest,” pick the agent wrapped around it, and discover three weeks later that a better model dropped, or the model they chose got suspended, or the price tripled. Then the bakeoff starts over and everyone is annoyed.

Start instead with the surface the team will live inside for a year. The model swaps in and out of that surface. The surface is what security configures, what auditors read, what the CFO’s spend controls plug into, and what developers actually feel every day. Three things make up that surface, and all three moved this week, which is exactly why they are worth anchoring on.

The first is the governable controls. Claude Code’s June 24 release added a setting called sandbox.credentials that blocks sandboxed commands from reading credential files, and it started showing organization-configured model restrictions right in the picker with a “restricted by your organization settings” message. GitHub’s Copilot CLI added managed-settings control of OpenTelemetry export and a toggle to switch MCP servers on and off. These are knobs. Someone in the org now owns each one. The harness worth committing to is the one whose knobs match the controls the business actually needs.

The second is portability. As one enterprise buying guide put it, the decision now “spans model access, data retention, audit logs, SSO, SCIM, usage analytics, cloud execution, local IDE support, and whether the vendor locks your team into one model provider,” and then the line that matters most: “Models will change. Prices will change. Rate limits will change.” (That guide is published by a vendor, so weigh it accordingly, but the point stands on its own.) The harness worth committing to is the one a team could walk away from. If the rules, the skills, and the MCP config only live inside one vendor’s box, that is not a tool, that is a landlord.

The third is consistency across the team. GitHub’s June 22 update let administrators publish a curated set of agents at the organization level so they are “automatically available to everyone in the organization or enterprise” for “consistent behavior and standards across your team.” That is the feature an executive should actually care about, because it is the difference between forty developers each freelancing their own setup and an org that behaves like one org.

If the rules, the skills, and the config only live inside one vendor's box, that is not a tool. That is a landlord.

Why the benchmark trap keeps catching smart teams

Here is the part that catches even careful leaders, and it caught me early too.

Benchmarks feel like the responsible way to decide. They are numbers. They are comparable. They make the decision look rigorous in the board deck. And they are almost entirely the wrong input for a commitment decision, because they measure the part that changes fastest.

Look at what the scoreboard actually said this week. On Terminal-Bench 2.1, the leader was Codex CLI paired with GPT-5.5 at 83.4 percent, with Claude Code paired with Fable 5 right behind at 83.1 percent, and Claude Code with Opus 4.8 at 78.9 percent. On SWE-bench Verified, Fable 5 sat at 95.0 percent, ahead of GPT-5.5 at 88.7 and Opus 4.8 at 88.6. Tight clusters, different leaders per benchmark, and the standout score belongs to a suspended model.

A team that committed last month to whatever harness was paired with the top SWE-bench model would today be standing on a model its developers cannot run, for a reason that had nothing to do with quality and everything to do with an export-control event no one could have predicted. The benchmark did not protect anyone. It pointed at a cliff.

That is the trap. A benchmark answers “which model is best at this task today.” The commitment decision needs to answer “which harness can absorb the next model swap, price change, or availability shock without forcing a re-platform.” Those are not the same question, and only one of them survives contact with a Tuesday.

What betting on the harness looks like in numbers

I know “bet on the harness, not the model” sounds like a slogan, so let me make it concrete, because the math is the part that makes it real for a board.

The model leader changed at least twice across two benchmarks in a single month. The top model on one benchmark became unrunnable mid-month. Two major harnesses shipped a combined ten-plus releases in 72 hours, none of them a new model, all of them changes to the controllable surface. Put those three facts next to each other and the conclusion is not subtle: anything picked based on this month’s model is a decision that gets re-made within weeks, and re-platforming an engineering org is not free. It costs a re-evaluation cycle, retraining, config migration, and a quarter of “which tool are we on again” confusion.

Whereas a harness chosen on the strength of its governable surface and its portability absorbs the model swaps underneath, at almost no cost. A better model drops? Point the existing config at it. A model gets suspended? Fall back to the next one inside the same surface the team already knows. The harness absorbs the churn. That is the entire value of choosing the right layer to commit to.

The cost of getting the layer wrong is a re-platform per model cycle. The cost of getting it right is one good evaluation, run against the surface, refreshed on a trigger rather than a calendar. Over a year, that gap is enormous, and it shows up directly in delivered work and in how much of engineering leadership’s attention gets eaten by tool churn instead of shipping.

What an open source AI coding agent actually gets you

At some point in this conversation someone says “let us just use an open source AI coding agent,” and the room nods, because open source sounds like the answer to lock-in. It is a much better instinct than picking on a benchmark. It is also not the whole answer, and the gap between those two things is where teams get surprised six months later.

Start with what is genuinely open, because the list is longer than most executives assume. OpenCode ships under MIT, and so does OpenHands. Cline, Aider, Kilo Code, Block’s Goose, and OpenAI’s own Codex CLI all ship under Apache 2.0. These are real open source licenses on real, actively used agents, not hobby projects.

Now the part that gets missed in the nodding. What is open source is almost always the harness, not the model. Cline under Apache 2.0 still calls Claude or GPT through the team’s own API key. The orchestration is open, the intelligence is rented. That is a perfectly sound arrangement, and it is precisely the argument this piece has been making, because the open part is the durable layer and the rented part is the one that churns. But nobody should tell a board the AI stack is open source when the only open thing is the wrapper around someone else’s model.

A genuinely open stack does exist. Alibaba’s Qwen Code, an Apache 2.0 fork of Google’s Gemini CLI, is built to run against Qwen3-Coder, which ships as open weights. Open harness, open weights, no vendor sitting in the middle of the request. Even there the vocabulary needs care: open weights means the model can be downloaded and run, not that the training code and data were published. Nearly every “open” model on the market, Llama and DeepSeek and Qwen and Gemma included, is open weight rather than open source in the sense an engineer means the phrase.

Three traps are worth knowing before anyone commits on the strength of the word.

The first is licenses that read as open and are not. Charm’s Crush ships under the Functional Source License, which is source available with restrictions on commercial competitors and converts to plain MIT only two years after each release. Read the license file, never the marketing page.

The second is scope. Codex CLI is Apache 2.0, but its VS Code extension is not, and there is an open request on the repository asking OpenAI to open the extension so it matches the CLI. An agent can be open at the terminal and closed in the IDE where developers actually spend the day.

The third is permanence. Roo Code, one of the more widely adopted open agents, was archived in May 2026 when its maintainers pivoted to a hosted product, and the community continuation surfaced as Kilo Code. The code stayed available, which is exactly what the license promised. The maintenance did not. So when scoring an open source AI coding agent, the fork lineage and the commit cadence belong on the scorecard next to the license, not in the trivia column.

Why AGENTS.md did not settle the portability question

The portability test sounds harder to run than it is, partly because the last year handed the industry something that looks a great deal like a solution to it.

In December 2025 the Linux Foundation launched the Agentic AI Foundation, and three things landed inside it at once: Block donated Goose, Anthropic donated the Model Context Protocol, and OpenAI donated the AGENTS.md convention. The platinum member list reads like a roll call of the entire field, with AWS, Anthropic, Block, Bloomberg, Cloudflare, Google, Microsoft, and OpenAI all in. Neutral governance for the config layer is real progress, and it is the clearest signal yet that the connective tissue between agents is meant to be shared rather than owned by whoever wins.

Then there is what happens on an actual laptop. Codex reads AGENTS.md. Claude Code reads CLAUDE.md. Gemini CLI reads GEMINI.md. Same idea, three filenames, and instructions written for one do not simply travel to another. A developer walkthrough published in May 2026 traced this fragmentation file by file and landed on the unglamorous conclusion: cross-tool portability of agent instruction files is still partial in practice, even with the convention widely adopted.

So the standards logo on the vendor page is not the test. The test is the literal one: take the rules, skills, and MCP configuration out of a live project, point a second agent at them, and see what survives untouched. Whatever survives is portable. Whatever has to be rewritten is the switching cost, and it belongs in the decision as a number rather than as a feeling.

MCP is the genuinely good news inside that story. It is the piece that does move between agents, it now sits under neutral governance rather than one vendor’s roadmap, and an agent that speaks it inherits a whole tool ecosystem instead of a single catalogue. If a finalist does not support MCP, that is a portability answer all by itself, and it arrived without anyone having to run a test.

A five-step way to choose without re-deciding every fortnight

Here is the playbook I would actually run. It is built to produce a decision that holds, not a decision redone every time a model drops.

1. Write the non-negotiable controls before looking at a single agent

   List the governable surface the business actually requires: credential isolation, organization model allowlists, audit logging, OpenTelemetry export, SSO and SCIM, kill switch. Write it as a checklist. This is the scorecard, and it comes from security and finance needs, not from a vendor's feature page.

2. Score the harness against that surface, never the model against a benchmark

   For each candidate agent, check off which of the required controls it actually exposes today. Claude Code's sandbox.credentials and organization model restrictions, Copilot's managed OpenTelemetry export and org-published agents: these are the kind of line items being scored. The model's benchmark score does not appear on this scorecard at all.

3. Run the portability test out loud

   Ask one question of each finalist: if the team had to leave this agent in six months, what comes with it and what stays trapped? If the rules, skills, and MCP configuration are portable, the agent passes. If leaving means rebuilding from scratch, mark it as lock-in and price that risk into the decision.

4. Name one owner for every knob the harness exposes

   Every setting that shipped this week is a default someone has to set, and the default nobody sets is the vendor's. Assign credential controls, model allowlists, spend caps, and observability export to named people before rollout, not after the first incident. An unowned control is an unset control.

5. Set a re-trigger, not a renewal date

   Do not put "re-evaluate the agent" on next year's calendar. Define the event that forces a fresh look: a depended-on model gets suspended, the price changes, a relied-on control gets removed. Re-evaluate on the trigger. Between triggers, let the model churn happen underneath the harness and leave the decision alone.

The one-page sheet I score finalists on

Step two says score the harness against the surface, which is easy to write and easy to abandon the moment a real quarter starts. So here is the actual sheet I use, unedited. It has six lines because a scorecard nobody finishes is worse than no scorecard at all, and every line is answerable in an afternoon by someone with the agent installed.

Two things I learned from running this sheet rather than reading a comparison table. The first is that the portability line is the only one most vendors cannot answer for themselves, which makes it the most informative question on the page. The second is that a finalist scoring well on lines one through four and badly on line five is the classic expensive mistake: excellent governance, inside a box with no door.

This is also the reasoning behind how we built Cerevisor: the run history and the policy config sit on the operator’s own machine, so lines three and five are answerable by looking at files a team already owns instead of by filing a request with a vendor. The documentation covers where that config lives on disk for anyone who wants to build their own version of this sheet.

The whole point of running it this way is that the choice stops getting re-litigated every time the scoreboard reshuffles. The commitment was made at the layer that holds. The model under it can be the best one this week, a different best one next week, or a suspended one the week after, and none of that forces anyone back into the room.

The founder who asked me the original question wanted to know which AI coding agent is best. The honest answer is that “best” expired sometime between when they asked and when I replied. The better question, the one that actually has a durable answer, is which harness a team would still be glad it chose after the model under it has changed three times. Pick for that, and the leaderboard becomes something to read for curiosity instead of something to chase.