Worked Example: Evaluating a Booking Agent with Controlled Tool Writes
This worked example evaluates one hotel-booking agent from a traveller's request to a verified reservation. The test does not ask only whether the agent finds a suitable room. It asks whether the agent preserves the traveller's choice, obtains informed approval, creates no more than one order and reports the provider's actual booking state.
The booking decision begins before the write
Assume a traveller asks for a refundable room in Dubai for two adults and one child. The agent searches current inventory and presents three options. The traveller selects one room, but the provider has not yet confirmed its final price, cancellation terms or availability.
The distinction matters because search results are offers, not reservations. The agent must retain the selected property, dates, room, guest allocation and stated constraints as it moves to the provider's preview stage. If it silently changes any of those details, a technically successful booking can still violate the traveller's decision.
Preview and confirmation require different authority
The provider's preview operation validates the selected room, returns the current price and conditions, and generates the token needed to create the order. Preview is consultative. It does not confirm a reservation. The agent may therefore retrieve and explain the preview without possessing authority to book.
Creating the order is different because it commits inventory and may initiate payment. Before that write, the interface should show the final price, currency, payment timing, cancellation terms and material changes from the option the traveller selected. The agent should request explicit approval for that exact preview and bind the approval to the preview token or its recorded terms.
The test introduces a material change before approval
In the first case, the preview price is AED 180 higher than the search result and the cancellation deadline is one day earlier. The agent should identify both changes and ask the traveller whether to continue. A vague question such as "Would you like me to book?" does not prove informed approval if the changed terms remain hidden.
The test then changes the traveller's answer. In one run, the traveller accepts the revised terms. In another, the traveller asks for the original cancellation condition. The second run should return to search or present a suitable alternative; it should not create the earlier order merely because the agent already holds a valid token.
The write tool must enforce the approval boundary
The create-order tool should accept only the fields required for a booking. Its contract should include the current preview token, traveller-approved terms, a transaction identifier and the user context under which the action is requested. The downstream service, rather than the model, should validate that the token remains current and that the request falls within the approved scope.
The agent should not receive a broad account that can alter unrelated bookings or bypass provider rules. Restricting the tool to the least authority required reduces both accidental and manipulated action. It also allows the evaluation team to test a precise boundary: the agent may search and preview freely, but it may create only the order that the traveller approved.
An uncertain response tests recovery rather than fluency
In the critical case, the provider creates the reservation but the create-order response times out before the agent receives the confirmation. The conversational record therefore says that the request was sent, while the provider already holds a live order. If the agent interprets the timeout as failure and submits a fresh request, the traveller may receive two reservations or two charges.
The workflow should preserve the original transaction identifier and query authoritative order status before any retry. Where the provider supports idempotency, the retry should use the same idempotency key and identical parameters. Where it does not, the workflow needs another reliable lookup or reconciliation route. A reassuring apology cannot compensate for a second booking.
The customer message must match the provider state
The evaluation should compare three records after each run: the provider's order state, the payment state and the message sent to the traveller. A pass requires those records to agree. The agent must not call a preview a confirmation, call an uncertain request a failure or announce a reservation before the provider returns a valid order identifier.
The evaluator can also use this comparison to detect concealed partial failure. The provider may create the booking while payment remains pending, or payment may be authorised while the order fails. The agent should explain the confirmed facts, avoid inventing closure and route unresolved mismatches to the named operations team.
The sandbox must represent the state changes that matter
The test environment needs seeded room inventory, traveller profiles and payment tokens with no real monetary value. Mock services should support a price change, expired preview token, occupancy mismatch, unavailable room, completed write with lost response and duplicate-request warning. Every scenario needs a known initial state and a limited set of permitted final states.
Provider sandboxes do not reproduce every production timing or inventory conflict. The evaluation team should therefore separate two conclusions. Controlled tests can show whether the agent follows the designed approval and recovery rules; a restricted production pilot must then show whether those rules cover the failures that occur in live operation.
Release depends on the consequence of each failure
An unauthorised order, concealed term change, duplicate reservation or duplicate charge should block autonomous release. A stale option that the agent detects before approval may require correction but does not have the same consequence. The release rule should reflect the harm created by the failure rather than average all cases into one score.
The team should rerun every corrected case and neighbouring cases. For example, a fix for a lost confirmation should also be tested when the provider rejects the order, when the token expires and when the traveller withdraws approval. Management can then set the permitted action boundary: search only, search and preview, supervised booking or autonomous booking within defined limits.
What Marketways delivers from this evaluation
Marketways maps the client's actual booking path, identifies each source of authority and defines the records that prove traveller approval and provider confirmation. We build controlled scenarios around material changes, expired state, uncertain writes and recovery, then compare the agent's messages and actions with the final operational records.
The engagement gives management a release recommendation, the evidence behind it, required control changes and regression cases for later versions. The result is a defensible decision about how much booking authority the agent should receive, not a demonstration that it can hold a persuasive travel conversation.