How to Design Graders for AI Agent Evaluations
Evaluation results cannot be more reliable than the rule used to decide whether each trial passed. A grader must inspect evidence that can establish the required behaviour, apply observable criteria and reach decisions that agree with qualified human judgement often enough for the intended use. The grading design should combine deterministic checks, model-based assessment and human review according to the evidence required for each claim.
Begin with the property the grader must establish
Teams often begin by choosing a grading technology. However, the first question is what the evaluation must prove. A refund test may need to establish that the agent identified the correct transaction, stayed within an approval limit, changed the ledger once and explained the outcome accurately. These are different properties and may require different evidence.
The grading plan should state each property, its consequence and the decision it supports. A permission breach may block release, while an awkward explanation may support a wording change. Combining both into one quality score would hide the difference and make remediation harder.
Use deterministic checks for observable state
Code-based checks are appropriate when the expected state, calculation, schema or tool argument can be specified exactly. A database query can establish whether one refund exists. A policy function can establish whether the amount exceeded the agent’s authority. These checks are reproducible and usually easier to audit than subjective ratings.
Deterministic does not mean infallible. A test can encode the wrong account, require unnecessary wording or reject a legitimate alternative path. The team should create a reference solution and confirm that the check accepts every intended route while rejecting prohibited outcomes. Otherwise, the grader measures compliance with a faulty test rather than successful completion of the task.
Use model graders for defined qualitative judgements
Some outcomes have several valid forms. A customer response may need to preserve uncertainty, explain the next step and avoid an unsupported promise. A model grader can assess these qualities at scale when the rubric identifies what evidence supports each judgement.
The rubric should separate dimensions that can fail independently. It should describe observable anchors for pass, partial and fail, and allow the grader to return “insufficient evidence” when the supplied record cannot support a decision. Labels such as “good reasoning” or “professional quality” are too vague unless the rubric states what the assessor must observe.
A model grader should receive only the evidence required for its criterion. A final response may support an assessment of clarity, but it cannot reveal a permission breach hidden in a tool call. Conversely, a full trace may expose confidential information or distract the grader with irrelevant detail. The grading input must match the claim.
Keep human judgement where automation lacks authority
Qualified reviewers remain necessary when the criterion depends on context, consequence, novelty or a professional judgement that the automated grader cannot establish. Human review is also needed to create calibration labels and investigate disagreements.
The reviewer guide should use the same observable anchors as the automated rubric. Reviewers should score an overlapping sample independently before discussing the cases. Agreement indicates whether the instructions produce consistent decisions. However, high agreement does not prove that the criterion is valid; several reviewers can apply the same flawed rule consistently.
Calibrate the model grader against reviewed cases
A model grader can sound decisive while disagreeing systematically with domain experts. Calibration compares its decisions with a reviewed set that includes clear passes, clear failures and difficult boundary cases. The set should also include correct refusals, polished but incorrect answers and terse answers that contain the necessary evidence.
The team should examine false approvals and false rejections separately because their consequences differ. A grader that approves unauthorised actions creates a different risk from one that rejects acceptable responses. The acceptable error balance depends on how the score will affect release, monitoring or customer treatment.
Agreement statistics can summarise performance, but the disagreements provide the explanation. Reviewers should inspect whether the model missed a material fact, followed an ambiguous anchor or preferred presentation over substance. The resulting revisions may change the rubric, evidence supplied, grader model or decision threshold.
The calibration set should also contain cases that the team did not use to write the rubric or tune the grader prompt. A grader can appear reliable when it is tested only on examples that shaped its design. A held-out set shows whether the grading rule transfers to unfamiliar cases before management relies on its scores.
Test the grader for presentation bias
Model judges can react to features unrelated to the required behaviour. Research has documented position, verbosity and self-preference effects. A reliable validation set should therefore include controlled pairs that preserve the substantive answer while changing order, length, formatting or apparent authorship.
The expected score should remain stable when the change is irrelevant to the criterion. If reversing two answers changes a pairwise judgement, the team can randomise order, judge each response independently or revise the prompt. If a longer answer receives a higher score without adding evidence, the rubric should direct the judge to cite the exact criterion and supporting text.
These tests do not prove the absence of every bias. They reveal whether common presentation changes influence the proposed judge enough to affect the evaluation decision.
Test whether an agent can game the grader
An agent may satisfy the scoring rule without fulfilling the business task. It may copy a preferred phrase, exploit a rigid string match or place persuasive instructions in text shown to a model judge. NIST describes this failure as grader gaming when the agent exploits a gap in automated scoring.
The evaluator should construct cases that target the grader rather than the workflow. It can add rubric language to the candidate response, produce an impressive explanation after an incorrect action or exploit an accepted shortcut that bypasses the intended task. Outcome checks and trace evidence should prevent rhetorical quality from covering a failed business state.
Version and monitor the grader
A changed rubric, reference answer, threshold or judge model can change scores while the agent remains identical. The evaluation team must therefore record the version of every grading component alongside the test suite. Before comparing releases, the team should rescore a stable calibration set and explain any break in comparability.
Production examples can reveal new valid responses and new ways to exploit the grading rule. The team should add reviewed cases to the calibration set, but it must not silently edit historical criteria. A recorded version change preserves the meaning of earlier results.
Example: grading a duplicate-charge refund
A customer asks an agent to refund a duplicate card charge. The business rule requires a verified customer, two matching charges, one authorised refund and an updated ledger. It also requires the agent to explain when the evidence is insufficient and refer amounts above its limit.
Deterministic checks establish identity, charge matching, approval status and final ledger state. A model rubric assesses whether the explanation accurately reflects the evidence and next step. Human reviewers label the boundary cases, including similar but non-duplicate charges, a polite explanation after an unauthorised refund and a correct refusal when the second charge is still pending.
The model grader is tested with reordered answers, additional but irrelevant detail and wording that imitates the rubric. A high language score cannot override an incorrect ledger state or a missing approval. The combined grading design therefore preserves the distinction between completing the transaction correctly and explaining it well.
What Marketways delivers
Marketways defines the property, required evidence, accepted alternatives, severity, anchors and threshold for each grader. We build deterministic checks for observable state, model rubrics for bounded qualitative judgements and reviewer guides for decisions that require human expertise.
We then calibrate model graders against independently reviewed cases, measure decision errors, investigate disagreement and test presentation bias and gaming. The validation record states where the grader is reliable, where human review remains necessary and whether its errors are acceptable for the proposed evaluation decision.