From Production Failure to Reproducible AI Agent Regression Test
An AI agent may fail under one specific set of operating conditions. The organisation gains lasting protection only when investigators can explain that failure, reproduce its cause and turn the verified case into a release test. Copying the original prompt is rarely enough because the outcome may also depend on retrieved evidence, tool responses, permissions, timing and external records.
Preserve evidence before changing the agent
When an AI agent causes or contributes to an operational incident, the immediate priority is to contain the consequence. The investigation must also preserve the evidence needed to explain what happened. That evidence can include the user's request, conversation history, retrieved sources, model and prompt versions, tool inputs and outputs, permissions, timestamps, approvals, errors and the business records that existed before and after the run.
An immediate prompt change may remove the visible symptom. However, it may also make the original behaviour difficult to reproduce and leave the underlying cause unproven. The incident owner should therefore preserve the relevant trace and external state before the implementation team changes the system, subject to the organisation's privacy, security and retention rules.
Define the failure in business terms
Investigators can use a trace to reconstruct what the system recorded. The trace does not determine which behaviour was correct. Investigators must compare the agent's actions with the policy, authority, workflow and business state that applied when the incident occurred. They must then state the failure precisely: what the agent did, what it should have done and which consequence followed.
This distinction matters when the operating rule was unclear or the source data was wrong. A historical human response may offer useful context, but it is not automatically the correct reference. The regression case should protect an approved business rule rather than preserve an accidental practice or a disputed interpretation.
Reproduce the material operating conditions
The investigation must recreate the conditions that could have caused the failure. For a conversational error, the prompt and relevant context may be sufficient. For an agent that uses tools, the case may also require seeded records, permission settings, mock tool responses, scheduled events and the same retry or timeout sequence.
The test should run in a controlled environment that cannot repeat the production consequence. If the team cannot reproduce the material failure, it should retain the incident as an unresolved investigation case. An unreproduced incident can guide monitoring and further inquiry, but it does not prove that a proposed change has corrected the cause.
Isolate the cause without removing it
A complete incident record often contains details that did not influence the outcome. Investigators can remove or vary one condition at a time and rerun the case. This process tests whether the failure depends on a particular instruction, stale record, permission, tool response, event order or interaction between components.
The smallest useful case is not necessarily the shortest prompt. It is the smallest controlled fixture that still produces the same material failure for the same reason. That fixture must retain every causal condition, including external state. Otherwise, the test may pass because the team removed the source of the failure rather than because the agent now handles it correctly.
Specify expected behaviour and prohibited outcomes
Once the cause is sufficiently supported, the test owner must define what a passing run requires. Some cases have one correct output. Others allow several acceptable actions, such as checking authoritative state, asking for clarification or escalating to a named role. The case should record those accepted alternatives instead of rewarding one preferred sentence.
The test should also name outcomes that must never occur. Examples include a duplicate instruction, an unauthorised disclosure, an unsupported assurance or an unrecorded change to a customer account. Deterministic checks can verify identifiers, permissions and final state. Human reviewers or validated model judges may assess qualities that require interpretation. The evaluation method should follow the claim the organisation needs to prove.
Test the correction and its side effects
The implementation team should rerun the verified fixture after making the correction. It should also run neighbouring cases that exercise legitimate behaviour near the new boundary. A stricter refusal rule, for example, may prevent one prohibited action while blocking ordinary requests that the agent should complete.
Compare the corrected version with the approved baseline under the same dataset, evaluator definitions and operating conditions. Because model outputs can vary, consequential cases may require repeated runs. A higher average score cannot excuse failure on a case that protects a critical financial, safety, privacy or authority rule.
Example: duplicate payment after an uncertain response
Consider a finance agent that sends an approved supplier-payment instruction. The payment endpoint completes the instruction but returns a timeout, so the agent cannot tell whether the payment succeeded. The agent retries, creates a duplicate instruction and tells the operator that the supplier has been paid once.
The investigation preserves both request identifiers, the timeout, the payment records before and after each call, the agent's permissions and the message sent to the operator. A controlled fixture then seeds one approved invoice and makes the first mock payment call succeed while returning an uncertain response. The accepted behaviour is to check authoritative payment state and escalate if the agent cannot establish that state. The test fails if the agent submits a second instruction or makes an unsupported statement about settlement.
Keep the incident as a versioned release gate
A verified incident becomes a regression asset when the organisation records its starting state, event sequence, expected behaviour, prohibited outcomes, evaluators, severity, owner and review date. The asset should link to the original incident without copying unnecessary personal or commercially sensitive production data.
Run the case whenever a relevant model, prompt, knowledge source, tool, permission, policy or workflow changes. Periodically confirm that the fixture still represents the current operating rule. Current evaluation platforms can link production traces to datasets and compare experiments, but the organisation remains responsible for reproducing the external business state and approving the release decision.
How Marketways converts incidents into assurance evidence
Marketways reconstructs the business decision, system state and sequence of events behind a material agent failure. We distinguish confirmed evidence from assumptions, test competing explanations and build a controlled fixture that reproduces the supported cause without exposing the production system to the same consequence.
We then define accepted behaviour, prohibited outcomes and the checks required to assess both. The client receives a versioned regression case, evidence record, correction assessment and release condition. The resulting regression asset changes future assurance practice instead of leaving the incident as an isolated production report.