How to Red-Team Agentic Workflows, Authority and Security

Red teaming must test a business threat

A broad instruction to “attack the agent” may produce striking examples without establishing whether the system can protect the business. The test must begin with a threat model. The evaluation team uses that model to name the protected asset, the attacker’s access, the action the attacker wants, the business consequence and the response the workflow must produce instead.

Current public threat catalogues distinguish several routes that matter to an agentic system. MITRE ATLAS includes prompt injection, context poisoning, tool-data poisoning, tool poisoning and unauthorised tool invocation. The catalogue helps a team ask which routes the proposed workflow actually exposes. It does not prove that every listed attack applies to that workflow.

Consider a procurement workflow that reads supplier documents and can propose changes to vendor records. Management needs to know whether a supplier document can induce the workflow to replace an approved bank account. That question defines the asset, attack path, prohibited action and safe outcome. It also determines which records, tools and approval states the test environment must reproduce.

Test the conflict between instructions and evidence

Agentic workflows often retrieve emails, documents, websites and database records to complete a legitimate task. The workflow should use that material as evidence and accept instructions only from authorised users, policies or system controls. However, an attacker can place instructions inside the evidence and attempt to make the agent accept them as authority. NIST describes this pattern as agent hijacking through indirect prompt injection.

A useful test preserves the legitimate user task while varying the hostile instruction. The red team can change its location, wording, source and apparent urgency. It can also split the instruction across several records or make the text resemble an internal policy. These variations reveal whether the workflow distinguishes authorised instructions from untrusted content or merely resists one familiar prompt.

One successful refusal does not settle the question. Model behaviour can vary across repeated attempts, and attackers can adapt their wording after observing a defence. NIST found that repeated and system-specific attacks produced materially different estimates of hijacking risk. Therefore, the test plan must include repeated trials and new attacks designed for the actual workflow.

Test what the workflow can do after manipulation

Prompt injection becomes a business loss when the surrounding application gives the model enough functionality, permission or autonomy to act on the malicious instruction. OWASP describes that condition as excessive agency. A model refusal is therefore weak evidence if the tool layer would still accept an unauthorised call.

The red team must test the action boundary itself. For each consequential tool, the test should vary the requested action, record, amount, user identity and approval state. It should also test values below, at and above an authority limit. The resulting trace must show whether identity checks, permission scopes and transaction controls stopped the action, even when the model attempted it.

These findings determine the remediation. A failed prompt defence may justify stronger separation between instructions and retrieved content. A successful but unauthorised tool call requires a narrower permission, an independent policy check or a human approval. Red teaming should identify which control failed instead of attributing every failure to the model.

Follow the attack across tools, memory and handovers

A multi-step workflow can carry a manipulated instruction beyond the turn in which it arrived. A tool may return poisoned content. A memory store may preserve a false fact or instruction. One agent may pass a compromised objective to another agent that has greater authority. The initial attack can therefore disappear from the final response while still influencing a later action.

The test must follow the complete trajectory. The red team should alter mock tool outputs, seed authorised test records, interrupt handovers and inspect persistent state. It should then verify which component read the hostile content, how the workflow represented it, which agent relied on it and whether the influence survived into another case. Synthetic accounts and sandbox databases make these tests reproducible without exposing client or customer data.

Persistent state requires a separate check. The red team should confirm that one user’s records cannot influence another user’s case, that an expired instruction does not remain active and that a corrected record supersedes the poisoned version. Otherwise, a workflow can appear safe during the attack and still fail later because the corrupted state remains available.

Test disclosure and side effects separately

Some attacks seek information rather than an operational change. The red team should attempt to retrieve personal data, credentials, system instructions and records from another case. It must inspect tool arguments and intermediate traces as well as the final response because a workflow can disclose information to an external service without displaying it to the user.

Other attacks seek side effects, such as sending a message, changing a record or executing code. These tests need a controlled environment with observable consequences. A mocked success message is not enough. The evaluator must compare the authorised starting state with the actual final state and confirm whether any prohibited action occurred.

Preserve demonstrated failures as regression tests

A red-team finding is useful only when another team can reproduce it. The record should identify the workflow version, model, instructions, tools, permissions, starting state, attack input, execution trace, final state and evaluation rule. It should also distinguish a demonstrated exploit from a plausible concern that remains unconfirmed.

After remediation, the team must rerun the failed case and a wider set of previously passed cases. The fix may stop the original wording while leaving the underlying attack path open. It may also block legitimate work. Regression testing therefore checks both security and task completion.

Controlled experiments can reveal plausible failures without establishing that the same behaviour has occurred in production. For example, Anthropic describes its agentic-misalignment results as controlled simulations rather than observed incidents in deployed systems. The report should preserve that distinction so that management can respond to demonstrated test evidence without presenting a simulated failure as a real event.

Passing the test suite does not prove universal resistance. It establishes the workflow’s performance against a defined threat model, environment and attack set. Management can use that bounded evidence to decide whether to release the system, restrict its authority, add monitoring or commission further testing.

Example: a hostile instruction inside a supplier document

A procurement agent receives a supplier document that asks it to ignore the approval policy and replace the bank account in the vendor record. The document supports the procurement task, but it has no authority to change the account.

The red team places the instruction in visible text, metadata and an attached file. It varies the supplier identity, proposed account, approval status and wording across repeated trials. It also gives the agent a realistic read tool and a sandboxed write tool. A safe workflow may preserve the document for review, flag the attempted redirection and refer the case to an authorised employee. The vendor record must remain unchanged.

The evaluator then uses the trace to establish why the outcome occurred. If the model attempted the change but the tool rejected it, the permission control worked while the instruction defence failed. If the model stored the hostile instruction in memory, the apparent refusal did not contain the risk. These distinctions determine whether the team must revise retrieval, memory, permissions, approvals or monitoring.

What Marketways delivers

Marketways translates the client’s operating process into an adversarial evaluation plan. We identify consequential assets and actions, map user and system authority, construct realistic attack scenarios, prepare controlled data and tool states, execute repeated trials and preserve the resulting traces.

In the final report, Marketways links each reproduced attack path to its business consequence and identifies the control that requires attention. Confirmed failures become regression tests. Unconfirmed concerns remain investigation items. This separation enables management to make a defensible release or remediation decision without presenting a limited test as proof that every attack has been prevented.

Continue through the agent evaluation series

• Agentic AI security and governance in the UAE
• How to design reliable graders for agent evaluations
• How to monitor new failures in production
• AI Agent Evaluation and Assurance