Statistical Design for AI Agent Evaluations: Repeated Trials and Reliable Conclusions
An agent can pass a case once and fail the same case on another run. A test score can also change because the case mix or grader changed rather than because the agent improved. Statistical design separates these sources of variation so that management can decide what the evaluation actually supports.
State the release claim before selecting a metric
The analysis should begin with the decision. Management may need to know whether the agent completes routine cases consistently, avoids a critical action, improves on the current workflow or performs acceptably for a defined customer group. These claims require different evidence.
The evaluation should name the action, operating population, consequence and tolerance. It should also define what would restrict or stop release. A single overall score cannot answer several release questions unless the underlying results preserve the relevant cases and failure types.
Define the unit of analysis
The observation may be a user turn, complete case, workflow trajectory, customer, asset or repeated trial. The correct unit follows the claim. For example, a case-completion decision cannot be supported by counting each message in the same conversation as an independent success.
Measurements from the same customer, case or simulated environment can share information and conditions. Treating them as unrelated observations exaggerates the amount of evidence. The analysis should preserve these groupings in sampling, comparison and uncertainty estimates.
Build a sampling frame that reflects intended use
The sampling frame should represent the routes, customer groups, data quality and operating conditions included in the proposed release. It should also include cases where the agent should not act. A test set containing only intervention cases can reward a workflow that intervenes indiscriminately.
Rare but consequential conditions may require deliberate oversampling. This design improves the ability to examine those failures, but the synthetic case mix does not estimate their production prevalence. The report should distinguish coverage from frequency and state which populations or conditions remain outside the evidence.
Repeat cases when behaviour can vary
Model sampling, tool timing and external state can change the route even when the starting case is the same. Repeated trials estimate how consistently the agent satisfies the requirement under controlled conditions.
The number of repetitions should follow the management claim and expected variation. Record the model settings, tool state and versions held constant. Report the number of trials and successes for each important case or group rather than presenting a percentage without its denominator.
Distinguish occasional success from dependable performance
Different summaries answer different questions. The probability of at least one success across several attempts may matter when a user can try several independent solutions. By contrast, a customer-facing or financial workflow may require every repeated trial to respect the same constraint.
Management should choose the reliability measure before seeing the result. Otherwise, a team can select the summary that makes an unstable agent look successful. Critical failures should also remain visible as events even when the average completion rate is high.
Compare versions on the same cases
Two versions tested on different cases can appear different because one received an easier sample. A paired design gives both versions the same evidence, constraints and starting state. The comparison then focuses on how their results differ for each case.
Record every component that can change the result, including the model, prompt, retrieval source, data, tool, permission and workflow. Review cases that improve, regress or change route, not only the average difference. A gain in routine completion does not justify a new critical failure.
Preserve dependence in multi-step and multi-agent tests
One workflow trajectory can contain several tool calls, handovers and decisions. These events reveal where the route failed, but they do not automatically create several independent tests of the overall workflow.
Similarly, multiple agents may share the same retrieval source or external state. Their errors can therefore be correlated. The analysis should report component findings while keeping the case or trajectory as the unit for the release claim that covers the complete route.
Measure the reliability of the evaluator
Evaluation results depend on the measurement process. Deterministic checks can be wrong when the expected state is incomplete. Human reviewers can interpret an ambiguous rubric differently. Model-based graders can vary between runs or disagree systematically with qualified reviewers.
Use overlapping human ratings to measure agreement and resolve weak anchors. Test automated graders against representative reviewed cases, including borderline and adverse examples. Where judge variability matters, repeat the grading and report disagreement. An unreliable evaluator cannot establish that the agent is reliable.
Analyse missed and unnecessary intervention together
A detector can capture every material issue by escalating every case. That design may have high sensitivity but little operating value. Evaluation must therefore report both missed issues and unnecessary interventions.
The preferred threshold depends on severity, prevalence, review capacity, delay and the cost of acting or failing to act. These factors belong to the management decision rather than to the model score alone. Subgroup results may also require different review arrangements when consequences or evidence quality differ.
Report uncertainty without false precision
A sample result is an estimate, not a permanent property of the agent. The report should state the number of cases, repetitions, missing results, exclusions and uncertainty. Very small subgroups may support a cautious restriction or a need for more evidence rather than a precise performance claim.
Controlled tests can establish a release boundary without estimating future savings or live failure prevalence. Production evidence should update the conclusion as the case mix and workflow change. The statistical review should make that limit explicit instead of extending the test beyond what it measured.
What management should receive
The release pack should state the decision, unit of analysis, sampling frame, subgroup coverage, repetitions, paired comparisons, evaluator agreement, critical events and uncertainty. It should distinguish limited evidence from unstable behaviour because those problems require different responses.
Marketways can design the experiment, analyse repeated and paired results, assess evaluator reliability and connect decision thresholds with the consequences of missed and unnecessary action. The result is a defensible release boundary and evidence plan, not a decorative score.