How to vet an AI agent vendor without a procurement team

Google renamed its entire AI platform last week, and almost nobody outside the keynote noticed what it meant for buyers. At I/O 2026 on May 19, Vertex AI became the Gemini Enterprise Agent Platform. Sundar Pichai’s keynote was a parade of scale numbers, and one of them stuck with me: “Over the past 12 months, over 375 Google Cloud customers each processed more than one trillion tokens.” That is a lot of production traffic. It is also a quiet announcement that the model is no longer the product. The agent platform is.

Every major vendor made the same move this spring. For a founder, that changes the evaluation completely.

The decision on your desk

A Series A founder raising right now usually has three things at once: an AI agent vendor decision sitting on the desk, a board that wants AI on the roadmap, and no procurement team to run the evaluation. That combination is exactly where bad contracts get signed.

The market is not short on options. It is short on honest ones. Analysts keep pointing out that most products marketed as agents are repackaged chatbots, RPA scripts, and linear workflows. The polite name for that is agent washing, and it is the single biggest reason a vendor evaluation goes sideways. The good news: a real evaluation does not need a procurement department. It needs a week and the right order of operations.

You are not really buying an agent. You are joining an ecosystem, and ecosystems are designed to be hard to leave.

How to vet a vendor in a week

1. Write the one workflow down first

   Before booking a single demo, write one sentence: the workflow being handed to an agent, and the number that proves it worked. Support ticket triage, measured by first-response time. Invoice matching, measured by exceptions caught per hundred. One workflow, one number. Everything in the evaluation gets judged against that sentence.

2. Run the three-part autonomy test

   Ask three questions of any product called an agent: does it make decisions without a human scripting every branch, does it chain multiple steps toward a goal, and does it recover when a step fails. MarkTechPost's May 19 platform analysis set the bar plainly, noting that genuine agentic AI requires autonomous decision-making, multi-step reasoning, and dynamic error handling. If a person hand-codes every path, that is automation wearing an agent label.

3. Make them prove it on real data, not the demo

   Every demo works. That is what a demo is for. The honest test is the vendor running their agent on a slice of actual data, including the ugly edge cases that only exist in production. Watch what happens when the input is malformed, the API times out, or the customer asks something off-script. A vendor who will not touch real data before a signature is telling you something.

4. Ask for the portability signal

   Lock-in is decided the day the contract is signed, not the day someone wants to leave. Ask one concrete question: does the agent speak A2A, the agent-to-agent protocol now under the Linux Foundation and in production at more than 150 organizations. "We support portability" is marketing. "We support A2A and can export agent state" is a fact that can be checked.

5. Price the platform, not the license

   Managed agents, the one-API-call kind Google and other vendors now ship, are billed by session or token, not by seat. A license number is not a cost. Ask for the modeled cost of the one workflow at ten times current volume. If the vendor cannot produce that figure, the evaluation is not finished.

6. Choose the ecosystem on purpose

   The agent rides on a platform: Google, Microsoft, Salesforce, AWS. After I/O 2026, that choice is explicit, and it is the real commitment. Pick the ecosystem already closest to the company's data and identity stack. Picking it by accident is the most expensive way to pick it.

Why most evaluations pick the wrong thing

The instinct, with a board watching and a deadline close, is to evaluate breadth. Line up five vendors, compare feature grids, pick the one that does the most. It feels rigorous. It is the wrong test.

The teams that get burned are rarely the ones who picked a weak vendor. They are the ones who picked a vendor for ten possible workflows and never proved one. The agent looked capable in all of them and dependable in none.

There is a calmer way to read this. A narrow evaluation is not the cautious choice that slows a team down. It is the fast one. One workflow proven in three weeks beats ten workflows half-working in three months, and a board can see the difference between those two without a slide.

What good looks like

A few numbers to calibrate against. That same May 19 platform analysis reported Microsoft Copilot Studio at 160,000 organizations running more than 400,000 custom agents, and Salesforce Agentforce at 800 million dollars in annual recurring revenue across 29,000 deals. The category is real and large. That was never the question.

The question is the gap between bought and working. Sinch published research this month finding that 74 percent of enterprises had already rolled back or shut down a live AI customer communications agent after deployment. Most of those were not weak vendors. They were unproven workflows that met production and lost.

So “good” is concrete. A finished evaluation produces five facts: one workflow, one metric, the vendor’s agent tested on real data, a portability answer on record, and a cost model at ten times volume. Five facts beat fifty slides.

Ship it

Here is the Monday version. Do not open a vendor demo first. Open a document and write the single workflow and the single number that defines success. That one sentence does more evaluation work than any feature grid.

Then invite two vendors, not five. Give each the same real-data slice and the same five questions: decisions, multi-step reasoning, error recovery, A2A support, and cost at ten times volume. The vendor who answers cleanly and works on real data is the one to take to contract.

And keep the term short. A twelve-month agreement with a renewal option beats a thirty-six-month commitment to a market that renames itself every quarter. The goal is not to predict the winner. It is to stay free enough to change a mind.