Which AI coding agent do you commit to when the best one keeps changing?

I pulled up a coding-agent leaderboard this week to settle a question a founder had asked me, and the top model on the SWE-bench Verified column, the one scoring 95 percent, had a little note next to it: export-suspended. As in, it cannot be run. The model with the single best score in the category is unavailable to most teams. The number-one result on the scoreboard is a model nobody can deploy.

That is the whole problem in one screenshot.

If a founder or CEO is reading this, they are probably being asked to bless a decision right now: standardize the engineering team on one AI coding agent, put real budget behind it, and tell the board there is a plan. The instinct is to pick the one with the best benchmark this quarter. I want to talk anyone with that instinct out of it, because the last three days are a clean demonstration of why it leads somewhere expensive.

The decision a founder is actually being asked to make

The question on the table sounds like “which AI coding agent is best.” It is really “which one does the whole org commit to, knowing it is a multi-quarter decision and the thing underneath it changes constantly.”

Those are different questions. The first one has an answer that expires in about two weeks. The second one has an answer that can actually hold.

Here is what I mean by “changes constantly.” In the past 72 hours, two of the biggest harness vendors each shipped a stack of releases. Anthropic moved Claude Code from v2.1.186 to v2.1.193 between June 23 and June 26. GitHub shipped Copilot CLI v1.0.64, v1.0.65, and v1.0.66 over the same window. Neither of those was a new model. Every one of those releases changed the controls around the model: what it can touch, what an admin can lock down, how teams observe it, which models a developer is even allowed to pick.

That is the tell. When the vendors themselves spend their release energy on the control surface instead of the model, they are pointing at where the durable part of the product lives. The commitment decision should follow theirs.

Pick the harness, not the model of the month

So here is the approach I would walk a founder through, and it inverts the usual order.

Most teams start with “which model scores highest,” pick the agent wrapped around it, and discover three weeks later that a better model dropped, or the model they chose got suspended, or the price tripled. Then the bakeoff starts over and everyone is annoyed.

Start instead with the surface the team will live inside for a year. The model swaps in and out of that surface. The surface is what security configures, what auditors read, what the CFO’s spend controls plug into, and what developers actually feel every day. Three things make up that surface, and all three moved this week, which is exactly why they are worth anchoring on.

The first is the governable controls. Claude Code’s June 24 release added a setting called sandbox.credentials that blocks sandboxed commands from reading credential files, and it started showing organization-configured model restrictions right in the picker with a “restricted by your organization settings” message. GitHub’s Copilot CLI added managed-settings control of OpenTelemetry export and a toggle to switch MCP servers on and off. These are knobs. Someone in the org now owns each one. The harness worth committing to is the one whose knobs match the controls the business actually needs.

The second is portability. As one enterprise buying guide put it, the decision now “spans model access, data retention, audit logs, SSO, SCIM, usage analytics, cloud execution, local IDE support, and whether the vendor locks your team into one model provider,” and then the line that matters most: “Models will change. Prices will change. Rate limits will change.” (That guide is published by a vendor, so weigh it accordingly, but the point stands on its own.) The harness worth committing to is the one a team could walk away from. If the rules, the skills, and the MCP config only live inside one vendor’s box, that is not a tool, that is a landlord.

The third is consistency across the team. GitHub’s June 22 update let administrators publish a curated set of agents at the organization level so they are “automatically available to everyone in the organization or enterprise” for “consistent behavior and standards across your team.” That is the feature an executive should actually care about, because it is the difference between forty developers each freelancing their own setup and an org that behaves like one org.

If the rules, the skills, and the config only live inside one vendor's box, that is not a tool. That is a landlord.

Why the benchmark trap keeps catching smart teams

Here is the part that catches even careful leaders, and it caught me early too.

Benchmarks feel like the responsible way to decide. They are numbers. They are comparable. They make the decision look rigorous in the board deck. And they are almost entirely the wrong input for a commitment decision, because they measure the part that changes fastest.

Look at what the scoreboard actually said this week. On Terminal-Bench 2.1, the leader was Codex CLI paired with GPT-5.5 at 83.4 percent, with Claude Code paired with Fable 5 right behind at 83.1 percent, and Claude Code with Opus 4.8 at 78.9 percent. On SWE-bench Verified, Fable 5 sat at 95.0 percent, ahead of GPT-5.5 at 88.7 and Opus 4.8 at 88.6. Tight clusters, different leaders per benchmark, and the standout score belongs to a suspended model.

A team that committed last month to whatever harness was paired with the top SWE-bench model would today be standing on a model its developers cannot run, for a reason that had nothing to do with quality and everything to do with an export-control event no one could have predicted. The benchmark did not protect anyone. It pointed at a cliff.

That is the trap. A benchmark answers “which model is best at this task today.” The commitment decision needs to answer “which harness can absorb the next model swap, price change, or availability shock without forcing a re-platform.” Those are not the same question, and only one of them survives contact with a Tuesday.

What betting on the harness looks like in numbers

I know “bet on the harness, not the model” sounds like a slogan, so let me make it concrete, because the math is the part that makes it real for a board.

The model leader changed at least twice across two benchmarks in a single month. The top model on one benchmark became unrunnable mid-month. Two major harnesses shipped a combined ten-plus releases in 72 hours, none of them a new model, all of them changes to the controllable surface. Put those three facts next to each other and the conclusion is not subtle: anything picked based on this month’s model is a decision that gets re-made within weeks, and re-platforming an engineering org is not free. It costs a re-evaluation cycle, retraining, config migration, and a quarter of “which tool are we on again” confusion.

Whereas a harness chosen on the strength of its governable surface and its portability absorbs the model swaps underneath, at almost no cost. A better model drops? Point the existing config at it. A model gets suspended? Fall back to the next one inside the same surface the team already knows. The harness absorbs the churn. That is the entire value of choosing the right layer to commit to.

The cost of getting the layer wrong is a re-platform per model cycle. The cost of getting it right is one good evaluation, run against the surface, refreshed on a trigger rather than a calendar. Over a year, that gap is enormous, and it shows up directly in delivered work and in how much of engineering leadership’s attention gets eaten by tool churn instead of shipping.

A five-step way to choose without re-deciding every fortnight

Here is the playbook I would actually run. It is built to produce a decision that holds, not a decision redone every time a model drops.

1. Write the non-negotiable controls before looking at a single agent

   List the governable surface the business actually requires: credential isolation, organization model allowlists, audit logging, OpenTelemetry export, SSO and SCIM, kill switch. Write it as a checklist. This is the scorecard, and it comes from security and finance needs, not from a vendor's feature page.

2. Score the harness against that surface, never the model against a benchmark

   For each candidate agent, check off which of the required controls it actually exposes today. Claude Code's sandbox.credentials and organization model restrictions, Copilot's managed OpenTelemetry export and org-published agents: these are the kind of line items being scored. The model's benchmark score does not appear on this scorecard at all.

3. Run the portability test out loud

   Ask one question of each finalist: if the team had to leave this agent in six months, what comes with it and what stays trapped? If the rules, skills, and MCP configuration are portable, the agent passes. If leaving means rebuilding from scratch, mark it as lock-in and price that risk into the decision.

4. Name one owner for every knob the harness exposes

   Every setting that shipped this week is a default someone has to set, and the default nobody sets is the vendor's. Assign credential controls, model allowlists, spend caps, and observability export to named people before rollout, not after the first incident. An unowned control is an unset control.

5. Set a re-trigger, not a renewal date

   Do not put "re-evaluate the agent" on next year's calendar. Define the event that forces a fresh look: a depended-on model gets suspended, the price changes, a relied-on control gets removed. Re-evaluate on the trigger. Between triggers, let the model churn happen underneath the harness and leave the decision alone.

The whole point of running it this way is that the choice stops getting re-litigated every time the scoreboard reshuffles. The commitment was made at the layer that holds. The model under it can be the best one this week, a different best one next week, or a suspended one the week after, and none of that forces anyone back into the room.

The founder who asked me the original question wanted to know which AI coding agent is best. The honest answer is that “best” expired sometime between when they asked and when I replied. The better question, the one that actually has a durable answer, is which harness a team would still be glad it chose after the model under it has changed three times. Pick for that, and the leaderboard becomes something to read for curiosity instead of something to chase.