How Marketways Evaluates AI Agents Under Real Business Conditions

Start with the responsibility the agent will receive

An organisation may ask an AI agent to resolve a complaint, amend a booking, review an invoice or investigate an operational exception. Each task carries a different consequence if the agent acts incorrectly or fails to act. Marketways first identifies the decision, record or transaction entrusted to the agent. We then establish who remains accountable, which actions require approval and which failures would prevent release.

This definition gives the evaluation a precise purpose. A complaint-handling agent, for example, may need to identify the customer and transaction, apply the approved recovery policy and refer an exceptional claim to a manager. Polite language does not prove that the agent completed any of those responsibilities.

Convert the responsibility into observable requirements

Broad aims such as “improve service” or “reduce handling time” cannot determine whether one run passed. Marketways converts the proposed responsibility into observable requirements. These requirements state what evidence the agent must obtain, which rules it must apply, what it may change, when it must stop and what outcome the authorised system must record.

The requirements also distinguish a helpful answer from a completed task. If an agent tells a customer that a booking has changed, the evidence must show that the correct booking changed once, the new option complied with the fare or rate rules and the customer received an accurate confirmation. The spoken or written answer is only one part of that proof.

Reproduce the conditions that can change the correct action

The same request can require different actions when the customer, policy, inventory, permission or prior event changes. Marketways therefore creates controlled cases that vary the business conditions rather than merely rephrasing the prompt. Ordinary cases establish basic competence. Boundary, adverse and interrupted cases show whether the agent can recognise when its usual action is no longer permitted.

Some cases reveal all relevant information at the start. Others introduce a delayed event, a conflicting record, a tool failure or a new instruction during the workflow. This variation tests whether the agent discovers and uses material evidence. It also exposes agents that succeed only when the prompt has already organised the problem and disclosed the answer.

Observe the path and verify the business outcome

Marketways records the agent's retrievals, decisions, tool calls, approvals, handovers and retries. This trace shows how the agent reached its answer. However, a plausible trace does not prove that an external system accepted the request or recorded the intended change.

We therefore compare the trace with the controlled business state. A successful tool message does not constitute success if the agent changed the wrong record, created a duplicate transaction or promised a follow-up that never entered the queue. The evaluation passes a consequential task only when the authorised action and final state agree with the declared requirement.

Match each claim to the right form of evidence

Known rules and final states support exact checks. Language quality or the adequacy of an explanation may require a reviewed rubric and a model-based judge. Consequential judgement, novel cases and disputed assessments may require a qualified human reviewer. Marketways assigns each claim to the evidence source capable of testing it instead of forcing every question into one score.

Automated judges also require evaluation. We compare their decisions with reviewed examples, examine material disagreements and preserve the judge version used for each run. Otherwise, a revised rubric or judge model could change the reported score even when the agent's behaviour remained the same.

Repeat uncertain behaviour and preserve serious failures

An AI agent may respond differently when the model samples another path, a retrieval result changes or an external service responds later. Marketways repeats cases where variation could alter the business outcome. We then separate systematic defects from intermittent failures and conditions that the test environment did not reproduce.

Average performance cannot erase a serious failure. One unauthorised payment, disclosure or record change may matter more than hundreds of correct low-risk answers. The evaluation therefore reports critical events and affected conditions separately, so management can see the failure that governs the release decision.

Turn production evidence into continuing assurance

Pre-release testing cannot reproduce every customer, event and system interaction. Once an approved agent enters controlled operation, Marketways uses reviewed incidents, overrides, complaints and near misses to identify missing cases. A production trace becomes a regression test only after a qualified reviewer establishes what happened, what should have happened and which conditions caused the difference.

This process allows the test collection to develop with the operating system. It also prevents one incident from becoming an unreliable test through an incomplete trace or an unverified assumption. Each accepted regression case retains the evidence needed to reproduce the failure after a model, prompt, policy, tool or workflow changes.

State the release boundary clearly

Marketways does not reduce the conclusion to a single quality score. The evaluation states which responsibilities the evidence supports, which conditions require supervision or additional controls and which actions the agent must not perform. Management can then grant authority that matches the demonstrated capability.

The client receives the requirements, controlled cases, evidence records, material findings, recommended controls and reusable regression tests. These assets explain why a release is supported, conditional or blocked. They also provide the baseline needed to investigate a later change without exposing Marketways' proprietary evaluation framework or selection logic.

Related evaluation guides

• AI agent evaluation test data is not operational data: assign cases and business-state fixtures to the right evidence layer
• Test AI agents with sandboxes, mock tools and seeded data: reproduce consequential actions without exposing live systems
• Validate an LLM judge before relying on its scores: establish whether an automated judge measures the intended quality
• Convert a production failure into a regression test: preserve a verified incident as repeatable release evidence