How to Build an AI Agent Evaluation Test Set
An AI agent test set should answer a business question: will the agent perform the intended tasks, under realistic conditions, without creating unacceptable consequences? A collection of convenient prompts cannot answer that question. The test set must represent the intended business tasks, expose material failure modes and preserve enough evidence to explain every result.
Define the decision before collecting cases
The required evidence depends on the decision. A development team may need rapid feedback on a new routing rule. A release committee needs evidence that the complete agent meets defined quality and safety thresholds. An operations team needs tests that explain a production incident and prevent its return.
The team should state the decision, the agent version and the operating scope before selecting cases. This boundary identifies the users, requests, languages, channels, tools and consequences that the test set must represent. Without it, the collection can grow while still omitting the cases that matter to the decision.
Describe the operating population
An agent will encounter a population of tasks rather than a list of prompts. That population includes common requests, unusual but legitimate requests, incomplete information, changing records, unavailable tools and users who express the same intention in different ways.
Teams should examine process records, product requirements, support tickets, incident reports and interviews with employees who perform or supervise the task. Production traces can reveal actual requests and failure patterns, provided that the organisation has the authority to use them and removes data that the evaluation does not require.
However, observed history is not a complete specification. It may exclude a new product, a rare safety event or a user group that received poor service and stopped trying. Product owners and domain specialists must therefore add plausible cases that the historical record does not contain.
Convert business failures into test cases
A useful case begins with a failure that the business needs to detect. The agent may select the wrong account, overlook a restriction, disclose protected information, repeat an irreversible action or claim completion before the underlying system changes. Each failure creates a different consequence and requires different evidence.
The case should identify the initial state, information available to the agent, permitted tools, relevant policy and expected outcome. It should also state what the agent must verify, what it may infer and what it must refuse or escalate. Two qualified reviewers should be able to reach the same verdict from that specification.
A reference solution then demonstrates that the task is possible and that the graders recognise an intended route. If a capable agent repeatedly fails a case, the team must first inspect the task, environment and grader. A broken test can create false confidence in the evaluation process while producing misleading scores.
Test both action and restraint
One-sided test sets reward one-sided behaviour. A suite that tests only when an agent should search may favour an agent that searches every time. A suite that contains only prohibited transactions may favour an agent that refuses legitimate customer requests.
For every important decision boundary, the set should contain cases on both sides. Include situations in which the agent should act, wait, ask for evidence, seek approval and decline. Add close counterexamples that differ by one material fact, such as an amount just below or above an authority limit. These pairs show whether the agent applies the rule or merely recognises familiar wording.
Routine cases also remain necessary. An agent that handles rare adversarial requests but fails ordinary work is not ready for use. The test set should therefore measure useful completion as well as risk control.
Preserve the environment, evidence and timing
A prompt and final answer do not capture an agentic task. The agent may read documents, call tools, update records and respond to events over several steps. Its success depends on what information and authority existed at each point.
Each case should preserve the relevant documents, database state, permissions, tool responses and event order. Trials should begin from an isolated, reproducible state so that one run does not contaminate another. The evaluator should record the final business state as well as the transcript because a confident confirmation does not prove that the requested action occurred.
Timing can change the correct decision. A pending payment may later settle, an approval may expire or a customer may provide missing evidence. The case must reveal each event at the intended time. Otherwise, the test may reward the agent for using information that would not have been available in production.
Separate development, release and monitoring sets
Developers need visible cases to diagnose behaviour and improve the agent. If the same cases also determine release, repeated tuning can produce a system that performs well on the known suite without improving on unseen work.
A held-out release set protects the decision from this leakage. Access should be restricted, and the team should preserve stable anchor cases so that results remain comparable across versions. NIST uses sequestered evaluation environments for the same general reason: separation reduces the risk that knowledge of the test changes the measured performance.
Production monitoring serves another purpose. It finds changes in users, data, tools and operating conditions after release. Verified incidents should enter a regression set, while a reviewed sample of ordinary production cases can test whether the wider distribution has shifted. The operational dataset used by the live agent remains distinct from the evaluation collection used to test it.
Design coverage around consequences
Counting cases does not establish coverage. Fifty variations of a simple request may provide less evidence than ten cases that exercise different tools, authority levels and failure consequences.
The team should build a coverage map across task type, user segment, language, tool path, data condition, permission level and consequence. It should then allocate more cases and review attention to combinations that occur often or could cause serious harm. Rare critical events may require deliberate synthetic cases because production history contains too few examples.
Synthetic variation should change one identified condition while preserving the rest of the case. This discipline allows the team to attribute a changed result to that condition. Uncontrolled paraphrasing can create many prompts without increasing the evidence that the suite provides.
A risk-focused suite cannot by itself estimate the failure rate in production. Deliberately over-sampling rare, serious cases is useful for testing capability, but it changes the case mix. If management needs a prevalence estimate, the team must analyse a representative operational sample separately and report the two results for their different purposes.
Plan repeated trials and segmented analysis
Agent behaviour varies across runs. One success does not establish dependable performance, and one failure does not show the failure rate. The evaluation plan should specify which cases require repeated trials and whether the decision values occasional success or consistent success.
Aggregate scores can conceal concentrated weaknesses. Results should therefore be segmented by material conditions such as language, channel, tool, authority level or customer type. The analysis must preserve critical failures as release gates even when the overall average is high.
The evaluation team should record the versions of the agent, prompt, model, tools, data, environment and graders. This version history allows the team to reproduce a finding and identify which change caused a later result.
Example: testing an order-cancellation agent
Consider an agent that cancels orders before dispatch. The operating population includes valid cancellations, shipped orders, split shipments, duplicate requests, missing identity evidence and customers whose payment status changes during the conversation.
The suite includes ordinary successful cancellations and close boundary pairs. One case places the order immediately before dispatch; another places it immediately after. A third introduces a tool timeout after the cancellation succeeds, testing whether the agent repeats an irreversible action. A fourth gives the agent a persuasive customer message but no verified identity.
Each trial starts with a known order, payment and permission state. Deterministic checks inspect the order record, refund ledger and number of cancellation calls. A bounded rubric assesses whether the explanation matches the final state. Repeated trials establish whether success is consistent, while segmented results reveal whether tool failures or state changes produce the largest risk.
The resulting evidence supports a release decision because the cases represent the operating boundary, the graders inspect the relevant outcome and the analysis distinguishes routine quality from critical failure.
What Marketways delivers
Marketways defines the evaluation decision, maps the operating population and converts material workflow failures into reproducible cases. We develop coverage maps, boundary pairs, reference solutions, controlled environments and held-out release sets without confusing evaluation data with the operational information available to the agent.
We then specify trial counts, grading evidence, release gates and segmented analysis. The final test set shows what the evaluation covers, what remains outside its scope and which findings management can use to approve, restrict or redesign the agent.