How to Test and Evaluate Agentic AI Workflows

Begin with the release claim

An evaluation cannot determine whether a workflow is suitable until management states what the release is meant to achieve. For a maintenance workflow, the claim may be that the agent prepares a valid work order for eligible alarms, assigns an authorised technician and refers safety-critical cases before dispatch. Each part describes an observable condition rather than a general expectation that the agent should be helpful or accurate.

The release claim must also name unacceptable outcomes. The agent may not invent an asset identifier, assign a technician without the required skill, disclose restricted records or mark equipment as restored without verification. These prohibitions determine which failures can block release even when average task completion is high.

Build cases from the operating population

Common cases establish whether the workflow can handle its intended volume. Difficult cases establish whether it remains dependable when records are missing, sources conflict, policies change, tools fail or users provide ambiguous or hostile instructions. The case set should preserve meaningful differences in route, value, consequence and affected user because an average score can conceal failure in a small but important segment.

The evaluation case set is not the operational dataset that the agent uses while completing a task. It contains test instructions, initial conditions and expected outcomes. The sandbox may supply copies or synthetic versions of operational records, but evaluators must keep those records isolated from live systems and protect any production data used to construct a case.

Test inside a controlled environment

An action-taking agent changes records as it proceeds. A valid evaluation must therefore begin from a known state and inspect the state after the run. A sandbox can contain a test asset register, mock scheduling service and work-order database so that the agent can call realistic tools without dispatching a technician or altering a live maintenance record.

Evaluators can then introduce conditions that are difficult or unsafe to wait for in production. A tool can time out, return stale information or reject a permission. A record can omit the technician's certification. An event can arrive after the agent has planned its route. Because each change is controlled, the finding identifies the condition that caused the behaviour rather than relying on an anecdote from an uncontrolled demonstration.

Judge the completed state first

The primary question is whether the workflow left the business process in an acceptable state. For the maintenance example, the evaluator can inspect the selected asset, priority, assigned technician, approvals, work-order status and evidence of restoration. Deterministic checks should assess exact facts and hard constraints whenever the system exposes them.

A completed state can still contain a serious side effect. The agent may create a correct work order and also send an unauthorised message, duplicate an existing task or expose a restricted field. The evaluator must inspect required outcomes and prohibited changes together. Task completion alone cannot establish safe completion.

Use the trajectory to explain the result

The trajectory records the evidence retrieved, tools selected, parameters supplied, approvals requested and recovery steps attempted. Reviewing this path can reveal that a correct final state depended on an unsafe tool call or that a failed task resulted from an unavailable service rather than a reasoning error.

However, evaluators should not require one exact path when several paths are valid. They should specify mandatory constraints, prohibited actions and material checkpoints, then allow harmless variation. This distinction prevents the test from rejecting a better route merely because it differs from the route imagined by the test designer.

Match each judgement to a suitable grader

Exact identifiers, permissions, transaction values and database state can usually be checked with deterministic rules. Expert reviewers are better suited to judgements that depend on engineering, clinical, financial or operational context. A model-based judge can review larger volumes of language or trajectories, but its score remains another model output rather than ground truth.

Before a model-based judge controls a release decision, evaluators should compare its decisions with a set of expert-labelled cases. They should examine disagreement by case type, test whether superficial wording changes alter the score and retain human review for consequential or ambiguous findings. A scalable judge is useful only when its known error is acceptable for the decision it supports.

Measure variation across repeated runs

The same case can produce different plans, tool calls or outcomes across repeated runs. One successful trial therefore gives weak evidence about reliability. Important cases should be repeated enough to estimate the rate of successful completion, prohibited action, human intervention, cost and latency under the same conditions.

Management should set release thresholds before seeing the results and attach uncertainty to the estimates. A workflow that succeeds in 19 of 20 trials has not proved a 95 per cent success rate for every future case. The evidence is especially weak for rare harmful events, which require targeted cases, stronger controls and continued production monitoring.

Compare the agent with realistic alternatives

An agentic design should be compared with the current human route and with the strongest simpler solution. In maintenance triage, a deterministic rule may route common alarms faster and more consistently, while an agent adds value only for cases that require evidence gathering and contextual judgement.

The comparison should use the same eligible cases and measure completed outcomes, rework, escalation, cost and elapsed time. Otherwise, the agent may appear superior because it receives easier cases or because the evaluation counts preparation as completion. A fair baseline determines whether the agent improves the decision rather than merely demonstrating technical capability.

Apply the evaluation to maintenance coordination

Consider an intermittent pump alarm that requires an inspection rather than an immediate shutdown. The test environment contains the asset history, current alarm, technician roster, access restrictions and spare-part record. The evaluator varies asset identity, alarm order, technician certification, access status and tool availability while preserving the expected safety rules.

One case gives the agent an apparently suitable technician whose certification expired the previous day. A correct run detects the expiry, withholds dispatch and refers the case. A superficially successful run may create the work order and obtain a quick acknowledgement, yet still fail because the assignment breached the release condition.

The final-state checks confirm the asset, priority, technician, approval and work-order status. The trajectory checks show which record supplied the certification date and whether the agent attempted an unauthorised fallback. Repeated runs reveal instability, while a rules-only route and the existing human process establish whether the agent improves difficult cases enough to justify its additional cost and control burden.

Convert findings into release conditions

Every confirmed failure should lead to a decision about the system. The remedy may change a prompt, tool, permission, knowledge source, operating rule or human handover. The case should then become a regression test linked to that change. Production failures and near misses can extend the suite, but they should be reproduced in a controlled environment before the team concludes that it has found the cause.

Marketways can define the release claim, construct representative and difficult cases, design the sandbox, select and validate graders, compare alternatives and set evidence thresholds. Management receives findings that distinguish outcome failure, prohibited action, unstable behaviour, tool failure and weak evaluation evidence. That distinction directs investment towards the part of the workflow that actually needs correction.

Continue through the agentic workflow series

• Deployment and monitoring
• Failure and incident response
• Human-in-the-loop controls
• Single-agent versus multi-agent design