AI agent evaluation starts with a question that a model leaderboard cannot answer: does this complete system perform this business task reliably, under the conditions in which we intend to use it?
The model is one component. The result also depends on prompts, retrieval, tools, permissions, state, retries, time limits and the people who review its work.
Changing any of those can change the outcome without changing the model at all.
In brief: use public benchmarks to inform selection, then evaluate the deployed agent as a versioned system. Test business outcomes, prohibited side effects, appropriate escalation and the cost of human review. Repeat the checks when the surrounding system changes.
This article extends the Agentic AI reading path with a specific question: what evidence should justify an agent release? It is not another framework comparison or a replacement for the control plane that constrains agent actions.
Recent research clarifies what a score means
A preprint submitted on 30 September examines evaluation reliability across 22 agent benchmarks. Michael Hardy and colleagues distinguish ranking a fixed model-and-scaffold system from ranking the underlying model.
The distinction matters. A scaffold is the surrounding software that manages state, exposes tools and turns model outputs into actions. The study finds that the reliability of a ranking depends on the claim being made, and that limited coverage of scaffolds can leave uncertainty that simply adding more similar tasks will not resolve.
This is a research preprint, not a settled or independently verified universal rule. Its practical lesson is still useful: before relying on a score, ask what was held constant, what varied and which conclusion the evaluation can actually support.
A vendor's benchmark can be informative about its tested configuration. It does not establish how the same model will behave inside your retrieval system, with your tools and your approval rules.
Likewise, an internal test of your current agent can establish something about that agent. It should not be presented as proof that its underlying model is always better than an alternative.
Separate procurement evidence from release evidence
Model selection and production acceptance are different decisions.
For procurement, public evaluations can help identify candidates with relevant capabilities, context limits, latency and cost. They can also reveal questions worth testing.
For release, the evidence needs to describe the actual system: model version, prompts, tool definitions, retrieval configuration, policies, execution limits and the evaluation environment.
If a result cannot be associated with that configuration, it is difficult to reproduce or interpret.
This is particularly important when comparing providers. A comparison that changes the model, tool descriptions and retry policy together may identify a stronger complete configuration. It does not isolate which change produced the improvement.
Sometimes that is acceptable. The business may care about the best deployable system, not a scientific ranking of models. But the claim should match the experiment.
I would put that distinction on the evaluation report itself. “This configuration passed these acceptance cases” is a more useful release statement than “we are using the top model”.
Begin with an outcome contract
Consider an illustrative contact-centre agent that prepares a service-credit recommendation.
An answer that sounds reasonable is not enough. Success requires the right customer, the applicable agreement, the correct incident classification, a permitted calculation and an explicit approval route where necessary.
The evaluation should define both the desired final state and unacceptable side effects.
For example:
- the recommendation uses the current agreement and cites the applicable rule;
- no credit is issued without the required authority;
- restricted information is not exposed in the response;
- conflicting evidence produces an appropriate escalation;
- the customer record is not changed as a side effect of a read-only task.
These are acceptance conditions, not merely text-quality preferences.
In customer-experience technology, the same principle applies to handoffs, routing and knowledge retrieval. A ticket being closed is not necessarily the same as a problem being resolved. The metric must preserve the business distinction.
Test the path as well as the final answer
An agent can arrive at the correct answer by taking an unacceptable route.
It might read a record outside its permitted scope, use an unsupported policy, retry a transaction twice or rely on a tool result it should have rejected. A final-answer grader can miss those failures.
Use execution traces to evaluate the important steps. Check which sources were retrieved, which tools were called, whether approvals preceded consequential actions and how failures were handled.
Do not require a transcript of private model reasoning. Observable evidence, tool interactions, policy decisions and state changes are the more useful operational record.
Automated checks work well where the expected state is machine-verifiable. Human review remains useful for ambiguous business judgement, misleading presentation and cases where the evaluation rule itself is unclear.
A model used as a grader should also be treated as a fallible component. Test its judgements against human-reviewed examples, and do not let a fluent explanation override a failed permission check.
Build a portfolio of cases, not a demonstration script
Start with common tasks, but include the conditions that make real work difficult.
The test set should cover missing data, stale information, conflicting records, unavailable tools, duplicate requests, revoked permissions and attempts to introduce instructions through retrieved content.
Include tasks the agent should refuse or escalate. Otherwise, a completion-rate metric can reward unsafe persistence.
Separate cases by risk and task family. An overall average can hide weak performance on the small group of cases with serious financial or customer consequences.
Repeat selected cases to understand variation rather than assuming one successful run proves reliability. Keep a held-out set that is not used for prompt tuning, and turn production incidents into regression cases.
That last step matters because the environment changes. New product policies, tools and customer behaviours can invalidate an evaluation that was useful at launch. The wider trusted-autonomy lifecycle explains why release is a beginning, not a permanent approval.
Make evaluation affordable without making it superficial
For coding agents, evaluation also has to fit the team's delivery capacity. The guide to the AI coding-agent review bottleneck explains how to measure accepted changes, review effort and work waiting for approval.
Complete agent evaluations can be expensive because a task may involve many model calls, tool interactions and retries.
A September preprint by Yining She and Lei Lin studies recurring evaluation of a production analytics agent using 574 historical benchmark runs. It compares partial-evaluation approaches and reports that the authors deployed difficulty-stratified fixed subsets for operational simplicity, despite another method offering stronger score fidelity in their analysis.
This is one deployment study, not evidence that a particular subset size will work for your agent. The useful idea is to validate a cheaper recurring test against a fuller benchmark, rather than choosing a convenient sample and assuming it represents the whole system.
A practical release process can combine:
- fast checks for permissions, schemas and known regressions;
- a representative recurring set spanning task types and difficulty;
- a broader evaluation for significant model, tool or policy changes;
- sampled production review and ongoing incident analysis.
Measure the gap between the smaller set and the full evaluation. Revisit the sample when the workflow changes. Never remove a critical safety case simply because it slows the test.
Include the human team in the result
A correct recommendation can still create extra work if people cannot verify it or understand its limits.
MIT Technology Review's March argument for context-specific human-AI evaluation challenges tests that separate AI from the teams and workflows in which it is used. It argues for assessing effects over time and beyond isolated task correctness.
That is an important operational complement to technical benchmarks. Measure review time, correction effort, handoff quality, repeat customer contact and whether employees can detect significant errors.
HBR's September article on trust in AI agents also emphasises clarity about capabilities and limitations. An evaluation report should help employees calibrate reliance, not encourage them to trust every output because an aggregate score is high.
For platform builders, the Omningage customer-engagement context is a useful related reference point: an agent's contribution belongs inside a service workflow, not outside it.
Decide release thresholds before seeing the score
Write down the acceptance conditions before evaluating the new version.
Specify minimum outcome quality, prohibited failures, escalation expectations and cost limits. Define who can accept a known limitation and which tasks remain outside the agent's scope.
Track cost per correct, permitted outcome, including human review and rework. Cheaper model calls do not necessarily produce cheaper service.
If the system improves ordinary task completion but regresses on unauthorised actions, do not let an overall score conceal that trade-off. Reduce its scope or keep the previous version while the problem is fixed.
This is a technology-delivery decision, not just a model-selection exercise.
A release checklist for operators
- State whether the evaluation supports a model comparison, a system comparison or a production-release decision.
- Version the complete tested configuration, including tools, retrieval, policies and execution limits.
- Check final outcomes, important execution steps, prohibited side effects and correct escalation.
- Validate representative test subsets against broader evaluations, and retain high-risk regression cases.
- Include review effort, customer impact and cost per correct outcome in the release decision.
If your next release lacks this evidence, a focused technical review should begin with the outcome contract and test cases. Choosing another model can wait until you know what the current system is failing to do.
The question is not whether the model is impressive. It is whether the complete agent deserves the authority you are about to give it.
Sources and evidence notes
- Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard, Michael Hardy, Ruhana Azam, Anka Reuel, Mykel Kochenderfer and Sanmi Koyejo, submitted 30 September 2026. Research preprint; findings relate to the benchmarks and statistical framework studied.
- Efficient Benchmarking in Production: A Study of an Evolving LLM Agent, Yining She and Lei Lin, submitted 18 September 2026. Research preprint reporting one production analytics-agent evaluation setting.
- AI benchmarks are broken. Here's what we need instead., MIT Technology Review, 31 March 2026. A contextual-evaluation argument, not a universal quantitative estimate of deployment performance.
- To Adopt AI at Scale, Employees Need to Trust Agents, Thomas McKinlay, Stefano Puntoni and Serkan Saka, Harvard Business Review, 9 September 2026. Summarised and linked, not reproduced.