If you run a support agent, you have probably been handed a number: a resolution rate, reported by the vendor who built the agent. That number tells you almost nothing about whether the agent is doing its job well, because the vendor who reports it is also the vendor who gets paid when it looks good. (If you're earlier in the process, our buyer's guide to AI agents for customer service covers vendor evaluation; this guide covers what to measure once the agent is live.)
A real evaluation program does not start from that number. It starts from the execution trace: the full sequence of tool calls, retrievals, and decisions the agent made between a customer's first message and wherever the conversation ended. Grade that against metrics you define, and you catch policy violations and failed handoffs before they reach customers, not after a quarterly report tells you everything was fine.
To get there, a PM owning the agent roadmap needs three things before trusting any automation number: a metric set that covers more than resolution, a test set built from real tickets, and a regression check that runs before every release. The rest of this guide covers all three.
What Agent Evaluation Means for a Support Team
Most of what gets called "AI evaluation" online is actually model benchmarking: scoring a language model's raw outputs against a fixed question set, independent of any particular workflow. That is useful for picking a model. It tells you almost nothing about whether your support agent handles a billing dispute correctly, because a benchmark never sees your tools, your policies, or your knowledge base.
Agent evaluation is a different exercise. It has to score the trajectory, meaning every tool call, retrieval, and intermediate decision on the way to a final answer, not just the final answer itself. NVIDIA's own evaluation research makes this point directly: a final-output check can mark a conversation as resolved without ever showing which step inside it actually broke. A support agent can reach a technically correct answer through a sequence that should not have worked, pulling the wrong account or skipping a required verification step, and a benchmark would never flag it.
A team that only tracks a benchmark score, or a vendor's aggregate resolution percentage, is blind to the failures happening inside its own workflow. Closing that gap is what the rest of this guide is for.
The Metrics That Actually Predict Support Outcomes
We break the problem into six metric families: outcome, trajectory, reasoning, efficiency, safety, and consistency. Resolution rate is one outcome metric, not the whole picture, and it is the only one of the six a vendor has an incentive to inflate. It carries the same risk as ticket deflection rate: easy to report, easy to game, and quiet about what it hides.
Define each family in your own terms before you trust a number against it.
The efficiency row is where teams get sloppy fastest. Voiceflow's own platform latency figure, around 50 milliseconds, measures only what the platform itself adds on top of a model call. End-to-end latency on a voice agent, closer to 500 milliseconds, includes the model's own response time too. Those are two different numbers answering two different questions, and a "latency" metric that doesn't say which one it means isn't a metric. It's a guess dressed up as data.
Pull resolution rate out of this table and look at the other five. None of them show up in a vendor's dashboard by default. If your evaluation program only tracks what the vendor already reports, you aren't running a program. You're reading theirs.
Build Your Test Set From Tickets You Already Have
A test set built entirely from synthetic conversations will pass every time, because synthetic conversations tend to look like whatever the person who wrote them expected the agent to handle well. Real tickets don't cooperate that way.
Pull your sample from transcripts you already have logged. A usable test set has to represent the actual task distribution, sampled by intent and difficulty, with a harness that resets state and logs the full trace between runs. Your own transcript history is that task distribution. Use it.
Stratify the sample across:
- Common intents, at the volume they actually occur (don't overweight rare edge cases just because they're interesting)
- Known hard cases: tickets that already needed a human escalation, a correction, or a complaint
- At least one example per policy that carries legal or compliance weight
A usable test case needs three things written down before it goes in the set: the input (the real customer message or conversation start), the expected trajectory (which tool calls and retrievals a correct run should make, not just the right final answer), and a pass criterion a reviewer can apply without guessing. A test set missing any of the three is a collection of examples, not a test set.
Skip the edge cases your real customers never hit, and you've built a set that cannot catch the failures your real customers will hit.
Run the Loop: Judges, Human Review, and Regression Checks
Running this as a continuous loop, not a one-time audit, is what turns an evaluation program into a release gate instead of a postmortem. Most platforms, Voiceflow's own evaluations tooling included, give you a place to store the test set, run the judge, and track results over time. The discipline below is what you do with it.
Where an LLM Judge Earns Its Keep
Human review doesn't scale to every conversation, and it shouldn't have to. An LLM judge, calibrated against a smaller set of human-reviewed cases, can score a conversation against your rubric at a volume no review team could sustain.
Anthropic's own research on agent evals is blunt about where this breaks: a judge calibrated once and left alone drifts, because the model behind it changes and the agent's behavior changes, and the judge never notices either. Keep a standing sample, somewhere between 5 and 10 percent of what the judge grades, under ongoing human review. When the judge and the humans start disagreeing more than they used to, that's the signal to recalibrate, not a note for the next quarterly review.
Catching Regressions Before They Ship
A judge and a human reviewer both answer "is this conversation good." Neither answers "did the last change make things worse," and that's a different question with a different method.
Run your full test set against every change to the agent's instructions, tools, or underlying model, before it ships, and diff the trajectory, not just the outcome. An agent can reach the same final answer through a new path that happens to work on your test cases and fails on the ones you haven't written yet. Tracking the trajectory, not just whether the final message looked right, is what catches that before a customer does.
A judge without a human-reviewed baseline drifts quietly. A release without a regression check ships that drift straight into production, and the first anyone hears about it is a complaint, not a test failure.
Evaluate the Handoff and the Guardrails, Not Just the Chat
A transcript that ends in a resolved ticket can still have failed in a way resolution rate will never show: a bad handoff, a skipped verification step, a policy the agent should have enforced and didn't.
Handoff is worth grading as its own metric family, not folded into resolution. A good handoff passes the full conversation context and any relevant variables, an order ID, a routing reason, an AI-written summary, to the human agent who picks it up, so the customer never repeats themselves. A failed handoff drops that context silently, and the experience fails at exactly the moment a human is supposed to save it. Score handoffs on whether the receiving system actually got the context it needed, not on whether the handoff step fired at all.
Policy adherence deserves the same treatment. Our own comparison work grading policy checks across models found that models vary meaningfully in how consistently they enforce a written policy under pressure, which is exactly why policy adherence belongs in your metric set as its own line, graded against your actual policy document, not assumed because the conversation read smoothly.
A compliance reviewer isn't going to ask what your resolution rate was. They're going to ask whether the agent followed the policy on a specific ticket, and whether you can show your work.
Run the Pilot, Then Expand With Evidence
None of the above needs to run against your whole agent on day one. Pick one queue or one intent, run the full evaluation loop (metrics, test set, judge plus human review, regression checks) against it as a pilot, and set your expansion thresholds before the pilot starts, not after you see how it goes.
Our own guide to moving a CX pilot into production covers the operational side of this sequencing in more depth. The evaluation discipline above is what makes the expand decision defensible instead of a judgment call made under deadline pressure.
Before you move past pilot, clear this list:
- Resolution is defined in writing by your team, not inherited from a vendor's dashboard
- Every metric family in the table above has at least one measurement attached to it
- The test set is sampled from real transcripts, stratified by intent and difficulty, not synthetic-only
- An LLM judge is running against a human-reviewed baseline, with a recalibration trigger defined
- A regression check runs against the full test set before every release, graded on trajectory
- Handoff and policy adherence are graded as their own metrics, not folded into resolution
Bring that list, and the numbers behind it, into your next roadmap review instead of a vendor's resolution percentage. Build it as part of how your customer experience org operates, not as a one-off audit before a renewal. It's the only version of "the agent is working" you can actually defend when someone asks how you know.
