How to Evaluate Multi-Agent Workflows, Handoffs and Coordination
A multi-agent workflow divides a business case among specialised agents. Each agent may complete its assigned task correctly while the overall case still fails. Evaluation must therefore test who owns the case, what crosses each handoff, how agents resolve conflict and whether the completed workflow achieves the authorised business outcome.
More agents create a coordination problem
Organisations usually introduce multiple agents because one agent cannot handle every instruction, tool and specialist decision reliably. A triage agent may route a request, a specialist may analyse it and another agent may approve or execute the response. The design can reduce complexity within each agent.
However, decomposition moves part of the difficulty into coordination. The workflow must allocate every necessary task, avoid duplicated action and preserve the user’s objective as control changes. A local score for each agent cannot prove that the group met those conditions. The evaluation must treat the full case, rather than the individual response, as the unit of success.
Establish who owns the case
Two common architectures assign responsibility differently. In a handoff, one agent transfers control and the receiving agent takes ownership. In an agent-as-tool design, the primary agent retains ownership while a specialist returns a bounded result. Microsoft’s Agent Framework makes this distinction explicit because the two patterns require different context and control flows.
The evaluation team must first record the chosen ownership model. It should name the agent that owns the case before and after each transfer, the agents that may delegate, the authority that approves a consequential action and the component that verifies completion. Without this record, an apparent coordination failure may reflect an undefined operating design rather than poor model performance.
Ownership also needs an acceptance event. Sending a message does not prove that another agent received the task, accepted responsibility or could act on it. The test should therefore include a destination that is unavailable, rejects the task or returns an incomplete acknowledgement. The operating design must keep a named agent or employee accountable until another agent has accepted the case.
Test whether the handoff preserves the decision
A handoff should carry the business objective, material evidence, constraints, uncertainty, completed actions and next required action. More context is not automatically better. Excess history can obscure the decisive facts or expose information that the receiving agent does not need. The evaluation must test whether the transfer contains the minimum sufficient context for the specialist’s decision.
Boundary cases reveal whether the routing and handoff rules work. The test set should include requests that fit one specialist, cross two domains, change intent midway or contain conflicting signals. It should also include a case that no specialist can resolve. The evaluator then checks whether the sending agent chose the correct destination, preserved the original objective and retained every material constraint.
The receiving agent must validate the handoff rather than accept its summary as fact. It should retrieve primary evidence when the decision requires it, challenge unsupported assumptions and return an incomplete case for correction. Otherwise, one agent can compress an error into a confident summary that every later agent treats as established truth.
The evaluation must also verify what the chosen orchestration framework actually transfers. Context preservation is an implementation property, not a promise created by the word “handoff”. For example, Microsoft documents that its handoff orchestration synchronises user and agent messages while filtering tool calls, tool results and approval payloads. A test should therefore confirm that the receiving agent can recover any decision evidence that exists only in tool state instead of assuming that the conversation history contains it.
Observe shared state and simultaneous action
Multi-agent workflows may share a case record, memory store or business system. Shared access allows one agent to continue another agent’s work. Yet it can also create stale reads, conflicting updates and duplicated side effects when two agents act at the same time.
The evaluation should vary the timing and order of events. It can delay an acknowledgement, update a record during a handoff, revoke a permission after allocation or allow two agents to propose the same action. The evaluator then inspects the external state, not only the agents’ messages. The test fails if both agents issue the refund, neither agent acts, or a later write silently removes an earlier correction.
Shared state does not imply shared authority. A specialist may need to read a case without gaining permission to approve or execute it. Tests should confirm that an agent cannot expand its authority by asking another agent to perform the restricted action. The evaluator must therefore test coordination together with identity, permission and transaction controls.
Reconcile disagreement before the workflow acts
Specialists can produce different conclusions because they use different evidence, assumptions or decision rules. A supervisor agent that selects the majority answer may conceal the disagreement rather than resolve it. The workflow needs an explicit reconciliation rule that matches the consequence of the decision.
The evaluation should create cases in which agents disagree for known reasons. It can remove one record, change a threshold or give each specialist a different data version. The evaluator then checks whether the workflow detects the conflict, compares the relevant evidence and refers the case to an authorised person when the rule cannot resolve it. The final decision should follow the evidence and authority model, not the confidence or rank of one agent.
Test stopping and recovery
A coordinated workflow also needs to know when its obligation has ended. Agents may continue delegating, reviewing or retrying after the authorised action is complete. They may also stop after producing an answer while leaving the business record unchanged.
The test must define completion through observable state. For a refund case, completion may require an approved decision, one payment instruction, an updated case record and a customer notification. The evaluator should then inject a delayed response, a failed tool call and a duplicate event. The workflow passes only if it resumes the unfinished obligation without repeating a completed side effect.
Example: a disputed insurance claim
An insurer uses a triage agent to classify an incoming claim, a policy agent to check coverage and a fraud agent to examine unusual evidence. The triage agent owns the case until a specialist accepts it. The policy and fraud agents may recommend a decision, but neither may approve payment.
The test case contains a covered loss, an image that appears inconsistent with the claim and a corrected policy endorsement received during the review. The evaluator delays the fraud agent’s acknowledgement and gives the policy agent the earlier policy version. These conditions test allocation, acceptance, evidence freshness and disagreement without inventing an unrealistic task.
A dependable workflow keeps the triage agent responsible while the fraud handoff is pending. It attaches the relevant image and claim identifiers, records which policy version each agent used and detects the conflicting conclusions. The workflow then refreshes the policy evidence and sends the unresolved decision to the authorised claims manager. No agent issues or duplicates a payment.
This example cannot pass through high specialist scores alone. The evaluator must inspect the route, handoff contents, acceptance event, evidence versions, authority checks and final claim state. Those observations show whether the agents coordinated a complete case rather than producing several individually plausible answers.
Judge the completed case
The scorecard should begin with the authorised business outcome. Supporting measures can then identify why a case passed or failed. Relevant measures include routing accuracy, handoff completeness, acceptance latency, lost constraints, duplicated actions, unresolved conflicts, authority breaches, completion time and cost.
The evaluator must preserve the end-to-end trace and the external starting and final states. OpenAI’s evaluation guidance recommends trace grading for workflow-level questions such as tool choice and handoff timing. Trace evidence also prevents an average local score from hiding a missing action or duplicated side effect.
Repeated trials remain necessary because routing, handoff and reconciliation decisions can vary. Management can compare architectures only when the test set, environment and scoring rules remain stable across the alternatives. The resulting evidence can justify a multi-agent design, reveal that a simpler single-agent workflow performs better or show which coordination control needs revision.
What Marketways delivers
Marketways maps the client’s case ownership, specialist boundaries, authority rules and completion states before building the evaluation. We then construct boundary cases, manipulate timing and shared state, run repeated trials and inspect the complete trajectory and business outcome.
In the resulting report, Marketways separates agent quality from coordination quality. It identifies where allocation, context transfer, acceptance, state consistency, reconciliation or stopping failed. Confirmed cases become regression tests, allowing management to improve the multi-agent design without assuming that more agents necessarily produce a better system.