AI Agent Evaluation Test Data Is Not the Agent's Operational Data
An evaluation test collection defines the cases used to assess an AI agent. The operational environment contains the records, permissions, tools and events the agent uses while attempting those cases. Treating these assets as one dataset can produce a convincing score without proving that the agent can operate safely in the business system.
The test collection defines the question
An evaluation item may contain a user request, scenario, expected behaviour, reference evidence, severity and scoring rule. A collection of such items allows the organisation to compare agent versions against a stable set of questions. Evaluation platforms commonly manage these cases, experiments and scores.
The collection should explain why each case exists and which decision its result supports. It does not need to copy every underlying business record. Instead, it identifies the conditions that the operational fixture must provide when the agent runs the case.
Operational data creates the conditions for action
Operational data includes customer records, policies, inventory, transactions, messages, work queues, permissions and event history. Some items are read-only evidence; others change when the agent completes an action. Their relationships and current state determine whether the agent's decision is correct.
Consider a test item that asks an agent to change a hotel booking. The item can describe the guest's request and expected conduct. The operating fixture must establish that the booking exists, the guest is authorised, the new room is available, the revised terms are accepted and the change reaches the booking system exactly once.
One case may require several operational fixtures
The same evaluation question can run against different starting states. A booking change may be permitted for one rate, require additional payment for another and be impossible after the stay begins. These fixtures test whether the agent responds to business conditions rather than repeating a memorised answer.
Each fixture should identify the source, version, starting records, permissions, events and permitted final states. A mock service may be enough for a tool-selection test, while a seeded application may be necessary when the release decision depends on linked records and committed changes.
Production traces supply candidates, not automatic truth
Production traces can reveal recurring requests, rare failures and operating conditions that designers did not anticipate. Microsoft Foundry, Langfuse and Braintrust document routes for turning selected traces or production examples into evaluation datasets. However, the recorded outcome may be incomplete, wrong or affected by evidence that the trace omitted.
A qualified reviewer must determine what happened, what should have happened and which information can be retained safely. The reviewed incident can then supply a test case and, where needed, a controlled operational fixture. The historical agent response or human action should not become the expected answer without this adjudication.
Protect production information without weakening the case
Evaluation cases and fixtures may contain personal, confidential or commercially sensitive information. The team should minimise fields, replace identifiers, use synthetic records where they preserve the relevant relationship and restrict access according to the case's purpose.
De-identification should not remove the condition being tested. If two similar customer names caused a record-selection failure, a fixture that replaces both with one generic identity destroys the risk. The reviewer must preserve the structural feature while removing information that the evaluation does not need.
Version the case and the environment separately
A stable evaluation item can produce a different result after a policy, tool schema, inventory snapshot or permission model changes. Conversely, a revised case can change the score even when the agent and operational fixture remain unchanged. Record both versions for every experiment.
This separation allows reviewers to distinguish an agent regression from an environment or measurement change. The experiment record should also identify the agent, prompt, model, evaluators and run conditions so that another team can reproduce a material result.
The distinction changes platform and architecture decisions
An evaluation platform may provide strong dataset, experiment, grader and trace functions while relying on the client to reproduce operational systems. That division is not necessarily a product weakness. It becomes a risk only when buyers assume that case storage also supplies realistic permissions, business records, event timing and side-effect verification.
Selection should therefore assess two capability groups separately. First, determine how the platform governs cases, labels, experiments and scores. Then determine how the evaluation architecture will seed operational state, control tools, inject events, inspect side effects and reset the environment.
How Marketways designs the complete evidence system
Marketways maps every release claim to the test case, operational conditions and final-state evidence required to assess it. We define which information belongs in the reusable test collection, which records and services belong in the controlled environment, and which production evidence remains necessary after pre-release testing.
The client receives a case schema, fixture specification, data-handling rules, version model, platform responsibility map and evidence boundary. This design prevents a platform score from standing in for business-state proof and preserves a clear path from evaluation question to management decision.