Machine Learning Metrics vs AI Agent Evals: Accuracy Is Only the Beginning
Machine learning metrics measure whether a model performs a defined prediction task. An AI agent evaluation must answer a wider question: did the complete system make the right decision, use its authority correctly and produce the required business outcome? Management needs both forms of evidence because a reliable prediction can still sit inside an unsafe or ineffective workflow.
Begin with the decision that the model supports
A predictive model estimates an outcome that the business cannot observe in advance. For example, a model may estimate whether a transaction is fraudulent, a customer will leave or a machine will fail. The prediction matters only when it changes a decision, such as holding a payment, contacting a customer or scheduling maintenance.
Managers must therefore judge the model against the consequences of its errors. A missed fraudulent payment creates a loss, while an incorrect fraud alert delays a legitimate customer. Analysts must distinguish these errors because the business may tolerate them at different rates.
Choose model metrics that reveal those errors
Analysts begin by comparing predicted and actual classes. Accuracy reports the overall proportion of correct predictions. However, it can conceal serious errors when one outcome is rare. A model that predicts every transaction as legitimate may appear accurate in a dataset with very little fraud while failing to detect the cases that matter.
Precision shows how often a positive prediction is correct. Recall shows how many actual positive cases the model detects. A confusion matrix identifies the classes involved in false alarms and missed cases. If management uses predicted probabilities to set a decision threshold, calibration tests whether those probabilities correspond to observed frequencies.
No metric is inherently the right one. Management must select the measures and thresholds that reflect the cost of false positives, missed cases and unreliable probabilities. The model evaluation can then establish how well the prediction supports the defined decision.
An agent introduces additional ways to fail
An AI agent can use a model output, retrieve records, apply policies, call tools and change business systems. Each step introduces a failure that a model score cannot observe. The agent may classify a request correctly but retrieve the wrong account, use an invalid parameter, exceed its authority or stop before the transaction is complete.
These failures require evidence from the full trial. The transcript shows what the agent saw and which tools it called. State checks show whether the underlying system changed as intended. Policy checks establish whether the action stayed within the agent's authority. The final response shows whether the agent explained the actual outcome accurately.
Preserve component metrics inside the agent evaluation
The wider evaluation does not make model metrics obsolete. If an agent routes cases, detects risk or decides when to escalate, false-positive and false-negative analysis still reveals which decision errors occur. Calibration remains necessary when a probability controls an approval, review or intervention threshold.
However, the evaluation must keep component performance separate from end-to-end performance. Otherwise, a strong classifier can hide tool failures, and a successful workflow average can hide deterioration in the model. Separate results allow the team to locate the failure and choose the right remedy.
Measure the trajectory and the business outcome
Agent evaluation should test whether the system selected suitable tools, supplied valid arguments, respected permissions, handled intermediate failures and completed the task. Microsoft Foundry, for example, distinguishes task completion, task adherence, tool selection, tool input accuracy and use of tool outputs. These measures describe different parts of the trajectory.
The outcome remains decisive. An agent may say that it issued a refund even though the ledger contains no refund. Conversely, an agent may reach the right final state through an unsafe shortcut. The evaluator must therefore inspect both the resulting business state and any prohibited action taken along the way.
Several trajectories may be legitimate. The evaluator should not require one exact sequence unless the sequence itself is a control. It should instead define required evidence, prohibited actions and the final state that constitutes successful completion.
Repeated trials reveal reliability
A standard classification evaluation may score one prediction for each labelled observation. Analysts can and do use resampling or repeated experiments when the model question requires them. An agent creates an additional reliability problem because the same business case can follow a different multi-step trajectory on each run. One successful trial therefore proves possibility rather than dependability.
Important cases should run repeatedly. A customer-facing agent may need to succeed consistently, whereas a research agent may be useful when one of several attempts finds a valid answer. The reported measure must match that use. Anthropic distinguishes the probability of at least one success across repeated attempts from the probability that every attempt succeeds.
Repeated trials also require a stable environment. If shared state or infrastructure failures affect several runs, the results are not independent evidence of agent performance. The evaluation team must reset the environment and record the model, prompt, tool, data and grader versions.
Connect every threshold to a consequence
An average score can combine errors that demand different decisions. A slow response may justify optimisation. An unsupported statement may require a retrieval change. An unauthorised payment may block release even if every other case passes.
The scorecard should therefore separate component quality, workflow completion, control breaches and business consequences. Management can set minimum model performance, minimum task completion and zero-tolerance gates for specified critical events. This structure prevents a large number of easy successes from cancelling one unacceptable failure.
Example: fraud prediction inside a payment-review agent
A fraud model assigns a risk probability to each payment. The business chooses a threshold above which a payment requires review. Precision, recall and calibration establish whether the probabilities and resulting alerts support that decision.
The surrounding agent then retrieves the account, checks exemptions, places an authorised hold, records the reason and refers the case to a reviewer. A well-calibrated model cannot establish whether the agent used the correct account, respected the hold limit or created a review case. Those claims require trace, permission and state evidence.
The evaluation runs boundary cases above and below the risk threshold, including exempt accounts and tool failures. It reports model errors separately from agent errors. Management can then tell whether a missed intervention came from the prediction, the policy rule, the tool call or incomplete follow-through.
What Marketways establishes
Marketways begins with the business decision and identifies the errors that could change its outcome. We select model metrics that reveal prediction errors, then design agent evaluations for tool use, authority, trajectory and final business state.
The resulting assurance record distinguishes component performance from system performance and links each threshold to a management consequence. It shows whether the model is suitable, whether the agent uses it safely and whether the complete workflow produces a dependable result.