How Marketways Evaluates AI Agents Under Real Business Conditions
Marketways evaluates an AI agent as part of a business system. The evidence must cover what the agent said, what it did, what changed and whether the intended business task was completed within authority.
The object of evaluation is the operating system
An agent rarely works alone. It receives a request, retrieves evidence, interprets policy, calls tools, changes records and hands work to people or other agents. A test that observes only the final message cannot establish whether the surrounding process worked. Marketways begins by defining the workflow, operating context, users, systems, permissions and decision rights that make the agent useful or dangerous.
Business claims become observable requirements
Statements such as improves onboarding or reduces service effort are too broad to test directly. They must become observable claims about completeness, evidence, timing, authority, correction and outcome. For each claim, the evaluation records what the agent may do, what it must do, when it must stop and what evidence would justify management reliance.
Scenarios reproduce the conditions that matter
The test collection represents ordinary work, difficult cases, rare consequences and plausible false alarms. Some tests present all evidence at once. Others reveal information over several turns, change a tool response or add a delayed event. The evaluation shows whether the agent discovers and uses important information rather than succeeding only when the prompt has already organised the problem.
The trace is compared with the business state
Marketways inspects the conversation and the sequence of retrievals, decisions, tool calls, handovers and retries. The trace is then reconciled with the controlled external state. A tool call marked successful is not enough if the customer record remained unchanged, the wrong item was reserved or the promised follow-up never occurred.
Judges are evidence sources, not unquestioned authorities
Exact checks are used where the required state or rule is known. Model-based judges are useful where several answers may be acceptable. Human reviewers remain necessary for consequential judgement, novel cases and calibration. Automated judges are checked against reviewed examples and versioned with the evaluation so a change in the judge is not mistaken for a change in the agent.
The conclusion is conditional and decision-ready
The result states where the agent is supported by evidence, where additional controls or supervision are required, and where deployment is not justified. Material failures remain visible rather than disappearing inside one average score. The client receives recommendations and regression tests that can be repeated after changes to the model, prompt, knowledge source, tool, permission or workflow.
What to read next
- Test data and operational data are different: understand the two evidence layers
- Controlled environment testing: see how external conditions are introduced safely
- From production failure to regression test: carry operating evidence into future releases