How to Test AI Agents with Sandboxes, Mock Tools and Seeded Business Data
A prompt and expected answer can test what an AI agent says. They cannot prove what the agent changed in a booking system, payment service, customer record or operational queue. When an agent uses tools, the evaluation needs a controlled environment that can reproduce relevant state, introduce failures safely and verify the final business outcome.
Separate the test case from the operating environment
The evaluation case describes the situation, input, expected behaviour and scoring rule. The operating environment contains the records, permissions, services, events and clocks that the agent reads or changes. An evaluation platform may store and run the first asset while the implementation team supplies the second.
This distinction prevents a dataset feature from being mistaken for a business-system test. A case may state that a customer has one unpaid invoice. The environment must contain that invoice, its approval status, the permitted payment route and a way to verify whether the agent created zero, one or several payment instructions.
Choose the environment from the claim you need to prove
A mock tool can test whether the agent selects the correct function and supplies valid arguments. A provider sandbox can test schemas, authentication and documented service behaviour. A seeded database or isolated application environment is necessary when success depends on relationships among records or the final state of a business process.
Use the lightest environment that preserves the material risk. Reproducing an entire production estate increases cost and can make cases difficult to reset. However, a transcript-only simulation is insufficient when the release claim concerns a consequential write, delayed event, permission decision or interaction among several systems.
Write the workflow test contract first
Before the run, define the entry state, user goal, available evidence, permitted tools, authority limit, allowed state transitions, prohibited side effects, expected final states and cleanup rule. Give every run a unique identifier that links the scenario, trace, tool calls, seeded records and resulting side effects.
The contract determines what the environment must reproduce. A refund test may permit one request up to a defined amount, prohibit a live charge and require a review record when evidence is incomplete. This specification is more useful than a generic instruction to “test the refund agent” because it gives the evaluator observable pass and fail conditions.
Seed a known and isolated starting state
Create synthetic or suitably protected records with known relationships, history and permissions. Record their starting values and the limited final states the business allows. Include ordinary cases as well as deliberate conflicts, missing fields, similar identities and delayed updates.
Each run should receive isolated records and identifiers so that concurrent tests cannot change one another's evidence. The evaluator must also know whether a record came from a snapshot, a generator or a manually constructed fixture. Provenance makes the case reproducible and helps reviewers determine whether the seeded condition represents the real operation.
Make difficult conditions programmable
The environment should produce timeouts, uncertain acknowledgements, partial success, stale reads, duplicate events, unavailable services and permission errors on demand. The same condition can then be applied to different agent versions. The agent should not receive the answer in advance when discovery and recovery are part of the capability being tested.
Failure injection should preserve the causal sequence. If a mock payment service reports a timeout, the test must specify whether the payment failed or succeeded before the response was lost. Those two states require different recovery behaviour. A generic error message cannot test whether the agent checks authoritative state before retrying.
Verify traces and final state together
Capture the agent version, instructions, retrieved evidence, tool requests, tool responses, approvals, emitted events and database changes. For asynchronous workflows, advance the clock or deliver the downstream event that establishes completion. Link every item to the run identifier.
The evidence must distinguish an attempted action, an accepted request and a completed result. An agent can send the correct tool call and still change the wrong record, create a duplicate or claim success before the downstream system completes the action. The final assertion should therefore inspect the authoritative business state as well as the conversational response.
Prove that the test harness is trustworthy
Reset fixtures between cases, restrict credentials and network access, make writes reversible and check for orphaned records or events after failure. Use sink addresses, test queues and non-monetary payment methods so routine evaluation cannot contact a real customer or create a live transaction.
Run known pass and known fail cases against the harness itself. Deliberately insert a duplicate write, wrong identifier and missing cleanup action and confirm that the environment detects each one. Otherwise, a flawed harness may blame the agent for contamination left by an earlier run or hide the side effect that the test was designed to find.
State what the sandbox cannot prove
A controlled environment establishes behaviour under the conditions it reproduces. It may not capture every production race condition, provider limit, identity integration or volume pattern. Record those omissions instead of treating sandbox success as proof of unrestricted production reliability.
Where a dependency cannot be reproduced safely, constrain the test to the boundary the team controls and add staged validation, shadow operation or a limited pilot before granting authority. The release record should show which claims the sandbox supports and which still require operating evidence.
How Marketways designs controlled agent tests
Marketways begins with the business consequence the evaluation must detect. We define the workflow contract, identify the minimum environment required, design seeded records and programmable failures, and specify the trace and final-state evidence for each case. We also test fixture isolation, cleanup and harness assertions before relying on agent results.
The client receives an environment specification, fixture catalogue, mock and event behaviours, evidence schema, harness checks and release boundary. These assets allow the organisation to repeat consequential cases across model and workflow changes without exposing live customers, money or records to routine testing.