Indirect prompt injection is no longer adequately tested with a fixed folder of suspicious strings. New research frames it as adaptive search across the environment, the user’s task and the attacker’s objective. The practical response is a search-budgeted evaluation that measures whether untrusted content can reach a sensitive action, how much exploration that takes, and whether the failure survives replay.
A static payload test asks, “Does the model reject this sentence?” An adaptive test asks a harder question: “Given this agent’s tools, data paths and feedback, can an attacker discover a route from content the agent must read to an action the user never authorised?”
That distinction matters because real agents do not live inside a prompt. They read email, tickets, documents, comments and webpages. They follow links, call tools and change state. The dangerous instruction is often not the cleverest-looking one. It is the one placed beside a legitimate step the agent already expects to take.
What the new research changes
On 3 September 2026, Duong M. Nguyen, Joon Sik Kim, Blazej Manczak and Vaikkunth Mugunthan submitted Rethinking Indirect Prompt Injection as a Test-Time Search Problem. The paper treats the attack surface as task-dependent. It is jointly shaped by the environment, the benign user task and the attack objective.
The authors built an attacker with a search harness. It explores the environment, organises strategies, remembers prior attempts and evaluates candidates using the victim agent’s response and tool-use trajectory. Across synthetic workspace and retail task suites, more attacker test-time compute improved vulnerability discovery. Removing explicit strategy management caused repeated patterns and excessive reconnaissance, while the full harness kept finding useful directions at larger budgets.
The headline result is not that one model scored a particular percentage. Those numbers belong to specific synthetic tasks, victim models and a white-box experimental setting. The durable lesson is methodological: attack success is not a budget-independent property. A system that appears robust after ten generic probes may fail after a well-directed search learns which content the agent reads, which tools it can call and which near-miss deserves refinement.
The paper’s most useful observation is that successful attempts concentrate on task-adjacent surfaces. The attacker does not merely shout unrelated instructions into every input. It places the hostile objective in a path the victim already needs for legitimate work. That resembles social engineering more than a magic phrase.
Why the payload-list mindset breaks
A payload library is still useful. It provides repeatable regression cases, tests known parsing tricks and catches obvious failures cheaply. It becomes misleading when its pass rate is presented as the security of the whole agent.
Consider an assistant asked to review a customer ticket and issue a refund when policy permits. The attack surface includes the ticket body, attachments, linked order records, tool descriptions, returned database fields and any follow-up page the agent opens. The user task determines which of those surfaces become reachable. The agent’s authority determines what a successful manipulation can change.
A fixed test usually varies the words. An adaptive search varies the route, timing, context, apparent legitimacy and destination. Feedback from the victim then reveals whether the next test should change the content, choose a different surface or imitate a normal prerequisite more closely. This is why the AI gateway must be treated as a control plane, not merely a place to store provider keys.
This also explains why an input classifier cannot carry the whole defence. OpenAI’s prompt-injection guidance says developed attacks increasingly resemble social engineering and are not usually caught by systems that only classify inputs. Its recommended framing combines source-sink analysis with controls that constrain impact even if the model is persuaded. Untrusted content is the source. A dangerous tool action or external transmission is the sink.
Build an Agent Injection Search Envelope
I would give every agent red-team run an Agent Injection Search Envelope. This is a bounded evaluation contract, not permission to attack production. It defines the safe environment, allowed search effort, protected actions and evidence required before a result enters the regression suite.
The envelope needs seven fields:
- Benign task: the real user outcome the victim agent should complete. Without this, the test cannot measure preserved utility.
- Environment snapshot: tool versions, permissions, seeded data, model configuration and the exact state from which each attempt starts.
- Injection surfaces: the untrusted objects the task may legitimately encounter, such as a document field, comment, message or tool result.
- Protected actions: state changes or transmissions that must not occur without policy, including sending data, changing permissions, approving money or executing code.
- Search budget: maximum tokens, trials, wall time, tool calls and parallel branches. Report the result with this budget, never without it.
- Stop conditions: immediate termination for scope escape, real credentials, unexpected network access, unsafe state mutation or monitoring failure.
- Evidence contract: the trace, state diff, replay seed, policy decision and utility result retained for every confirmed failure.
The search process has four phases. First, map the task’s normal route through tools and data. Second, rank surfaces close to sensitive actions. Third, test benign synthetic objectives and use the victim’s behaviour to select the next bounded variation. Fourth, replay any apparent failure from a clean snapshot and turn the route into a deterministic regression case.
This is not a recipe for generating weaponised text. The security value comes from mapping routes and authority. A synthetic marker can stand in for sensitive data. A test account can stand in for a payment recipient. A mock endpoint can prove attempted transmission without exposing anyone.
Measure the action boundary, not dramatic prose
Agent evaluations often overvalue alarming output. A model may repeat a suspicious sentence while the policy layer correctly blocks every consequential action. Another model may sound calm while quietly adding an unauthorised recipient to a tool call. Only the second result crosses the boundary.
The paper evaluates both utility and security using resulting environment state. That is the right direction. A useful internal scorecard should extend the same idea:
| Metric | Question | Why it matters |
|---|---|---|
| Action violation rate | Did a protected state change or transmission occur? | Separates model chatter from material impact. |
| Search budget to first failure | How many tokens, trials and tool calls were required? | Makes robustness claims comparable and exposes shallow testing. |
| Route coverage | Which source-to-sink paths did the run exercise? | Prevents hundreds of wording variants from masquerading as broad coverage. |
| Replay rate | How often does the same route fail from a clean snapshot? | Distinguishes a reproducible defect from noise. |
| Benign-task utility | Can the agent still complete the legitimate task? | Stops a defence from “winning” by disabling the product. |
| Detection and containment time | How quickly was the route blocked or surfaced? | Connects the evaluation to incident operations. |
Report the distribution, not only the best or average result. A single confirmed high-impact route matters even if most trials are safe. Conversely, a one-off ambiguous trace should not become a board-level statistic until it is reproduced and its state change is verified.
Use a scorecard that rewards coverage and restraint
Do not reward the testing system solely for “winning”. That encourages noisy exploration and makes unsafe shortcuts look productive. Reward evidence quality as well: novel route coverage, reproducibility, clean isolation, precise state diffs and preservation of the benign task.
Keep strategy history separate from raw traces. The paper finds that explicit strategy management helps avoid redundant search. In an internal programme, that history also tells defenders whether the harness found a genuinely new route or simply paraphrased the same idea 200 times.
A compact run record might look like this:
evaluation:
benign_task: "summarise seeded support case"
protected_action: "send synthetic marker outside approved origin"
environment_snapshot: "support-lab-2026-09-08.3"
budget:
trials: 60
tokens: 120000
tool_calls: 180
stop_on:
- scope_escape
- real_credential_access
- monitoring_loss
report:
- action_state_diff
- utility_result
- route_coverage
- replay_evidenceThe values are illustrative, not universal recommendations. A low-risk summarisation agent and a privileged finance agent should not inherit the same budget or acceptance threshold.
What defenders should change
Adaptive testing strengthens the case for deterministic controls. Chrome’s WebMCP security guidance recommends restricting cross-origin interactions, confirming actions, setting token limits, recognising untrusted tool outputs, using classifiers at critical execution points and evaluating whether mitigations stop unauthorised actions without needlessly reducing capability.
Those controls should appear directly in the search envelope:
- Restrict sources: the agent should read only origins, mailboxes and repositories required for the task.
- Restrict sinks: sending, purchasing, publishing, permission changes and code execution need deterministic policy gates.
- Separate planning from approval: the component exposed to untrusted content should not alone authorise the dangerous action.
- Minimise arguments: tools should receive only the data required for the approved step.
- Bind confirmation to the state change: show the user the destination, data and consequence, not a generic “continue?” prompt.
- Fail closed at critical boundaries: loss of policy, identity or audit services must stop sensitive actions.
This extends the authority model in my earlier agent containment analysis. Containment limits what one compromised process can reach. Search-budgeted testing measures whether the remaining routes still allow untrusted content to influence an action.
Read the paper with its limits intact
The study is a preprint. It uses synthetic workspace and retail suites, a particular attacker harness, named model versions and a white-box setting where the attacker can see the environment. The authors describe black-box transfer as future work. Its measured success rates should not be copied into a risk register as the probability that a deployed agent will fail.
Test-time compute is also not the only axis. More realistic permissions, longer-lived memory, human confirmation design, tool reliability and detection response can change the outcome. A larger budget with a poor search strategy may simply repeat itself. The ablation result reinforces that point.
Still, the central challenge survives those limits. If your evaluation fixes the attacker’s procedure and budget, it measures the victim against that narrow opponent. It does not establish an intrinsic security level for the product.
A 60-minute review for an agent team
- 0 to 10 minutes: select one production-like task and name the final state change that makes it useful.
- 10 to 20 minutes: draw every untrusted source the task can read and every sensitive sink it can invoke.
- 20 to 30 minutes: choose a synthetic attack objective, clean snapshot and explicit stop conditions.
- 30 to 40 minutes: define the token, trial, time and tool-call budget. Record why the budget represents a credible attacker.
- 40 to 50 minutes: test one task-adjacent route. Capture the victim’s tool trajectory and environment state diff.
- 50 to 60 minutes: replay any failure, convert it into a regression case and identify the deterministic control that should break the route.
The output is one search envelope, one route map and one replayable result. If the team cannot say what the attacker was allowed to search, a passing score says very little.
Frequently asked questions
What is indirect prompt injection?
It is an attempt to manipulate an AI agent through external content the agent retrieves, such as a webpage, document, email or tool output, rather than through the user’s direct instruction.
Why treat prompt injection as a search problem?
The useful attack surface depends on the environment, the user’s benign task and the attacker’s objective. An adaptive attacker can learn from the victim’s responses, prune weak routes and concentrate effort on content and tools adjacent to the legitimate workflow.
What is a prompt-injection search budget?
It is the declared limit on attacker effort, including tokens, trials, tool calls, wall time and parallel branches. Security results should be reported with that budget because additional well-managed search can uncover failures missed by a small static test.
Does passing a payload benchmark prove an AI agent is secure?
No. It proves the agent resisted that benchmark under its specific procedure, environment and budget. A stronger evaluation also covers source-to-sink routes, protected state changes, replayability and benign-task utility.
What is the first control to add?
Define the protected action boundary. Decide which transmissions and state changes require deterministic policy or specific user confirmation, then test whether untrusted content can reach them.
The strategic shift is small but consequential: stop asking whether the model recognises a bad string. Ask whether an adaptive opponent can discover a path from content the agent must consume to authority the user did not grant. Name the search budget, inspect the state change, preserve utility and replay the route. That is a security result an engineering team can act on.
One next action
If you run an agent that executes tools, AgentCrash is an open-source harness that crash-tests it in a synthetic workplace: synthetic tools, visual evidence and regression tests. It is built for exactly this problem, turning a containment claim into something you can run against the system instead of arguing about it in review.