Prompt Injection Testing Without Publishing a Misuse Manual
A language-model system refuses an obvious attempt to override its instructions. The demonstration appears reassuring. Then the same system reads an ordinary-looking document, mistakes embedded text for a trusted command, and prepares an action its user never requested.
The first exercise tested whether the model recognised a conspicuous attack. The second tested whether untrusted content could cross an authority boundary.
That distinction should govern prompt-injection evaluation. A credible test is not a collection of provocative prompts, and a responsible publication is not a catalogue of reusable attack strings. The defensible approach is to publish an assurance case: the system boundary, threat model, test coverage, model and application versions, scoring rules, benign-task performance, observed failure classes, mitigations, retest status, and residual risk. Exact payloads, sensitive traces, credentials, internal instructions, vulnerable endpoints, and unremediated exploitation paths belong in a controlled technical record shared with authorised system owners and vulnerability coordinators.
This permits scrutiny without reducing the cost of misuse.
The vulnerability exists at the system boundary
Prompt injection is often described as a problem with a model following the wrong instruction. That description is incomplete.
NIST distinguishes direct prompt injection, in which an attacker supplies adversarial input through the normal user interface, from indirect prompt injection, in which malicious instructions are embedded in material that an application later retrieves or processes. That material may arrive through a document, website, email, knowledge base, tool response, or another external source.
The security consequence depends less on whether the model produces a strange sentence than on what the surrounding application allows the model to influence. A standalone chatbot that generates an irrelevant response presents one class of harm. A system that can read restricted files, send messages, modify records, execute code, initiate transactions, or shape an administrative recommendation presents another.
OWASP’s 2025 prompt-injection guidance makes this dependence explicit: the severity of an injection is shaped by the application’s context and the degree of agency granted to the model. It recommends privilege controls, deterministic output validation, human approval for high-risk actions, separation of external content, and adversarial testing of trust boundaries.
The UK National Cyber Security Centre reaches a sharper conclusion. It characterises an LLM as an “inherently confusable deputy”: a component that may be induced to exercise authority on behalf of the wrong party. Its guidance directs designers toward deterministic, non-LLM safeguards that constrain permissible actions, particularly when models can call tools or APIs.
The decisive question is therefore not, “Can someone make the model disobey?” It is, “What can happen when the model becomes confused, and which independent control prevents that consequence?”
Test the application that will carry the risk
A model name is not a test environment.
Before a prompt-injection evaluation begins, the institution needs a versioned description of the complete system being assessed. That record should identify the model and provider, the model version or release date, relevant generation settings, application instructions, retrieval components, external data sources, available tools, tool permissions, user roles, approval gates, network access, secret-handling arrangements, output-rendering behaviour, monitoring controls, and any deterministic policy layer between the model and an action.
Some of this information may be confidential. It still must exist.
Without that record, a result cannot be interpreted or reproduced. A model may resist an attack in a text-only interface and fail when the same model receives retrieved documents. A tool may be safe under read-only credentials and dangerous under broad write access. A filtering layer may block visible instructions while missing equivalent content in metadata or another modality. A provider update may alter the behaviour without any change to the application’s own code.
Early research on indirect prompt injection demonstrated why the system boundary matters. Greshake and colleagues showed that instructions embedded in material likely to be retrieved by an LLM-integrated application could manipulate application behaviour and influence connected functions or APIs. Their central finding was not that one wording defeated one chatbot. It was that applications had blurred the boundary between data supplied for analysis and instructions granted authority.
A test report that names only the model conceals that architecture. It may also assign responsibility to the wrong component.
Begin with consequences, not attack strings
A useful protocol starts by defining the protected outcomes.
For a public institution, those outcomes may include preserving the confidentiality of restricted information, preventing unauthorised communications, protecting the integrity of records, blocking unapproved tool calls, maintaining service availability, and ensuring that AI-generated material cannot silently alter a decision process. The exact set must come from the proposed use case, data classification, user roles, and governing authority.
The evaluator can then construct scenarios around trust transitions. A direct scenario asks whether a user can induce the system to exceed that user’s permissions. An indirect scenario asks whether content supplied by a less-trusted party can influence a more-privileged action. A retrieval scenario examines whether a compromised source can redirect the system away from the user’s task. A tool scenario examines whether an unsafe model output is independently rejected before execution.
This design avoids a common failure: treating the attack string as the unit of risk. Natural-language attacks can be paraphrased, fragmented, translated, obfuscated, distributed across interactions, or adapted to the target. A fixed collection can be useful for regression testing, but passing it does not establish broad resistance.
NIST’s Generative AI Profile cautions against extrapolating system performance from narrow, nonsystematic, or anecdotal assessments. It specifically notes that jailbreaking and prompt-engineering tests may fail to measure validity or reliability systematically, and that laboratory evaluations may not transfer to heterogeneous real-world settings.
Recent government testing illustrates the problem. In 2025, NIST’s Center for AI Standards and Innovation adapted the AgentDojo environment to an updated model and developed attacks on one set of tasks before testing them on held-out tasks. In that particular experiment, the strongest baseline attack succeeded 11 percent of the time, while the strongest newly adapted attack succeeded 81 percent of the time. The result does not provide a universal failure rate. It shows how a static attack set can substantially overstate robustness against an adversary who adapts.
A 2025 study by Zhan and colleagues reached a related result: adaptive attacks bypassed all eight indirect-prompt-injection defences evaluated in their experiments, with reported attack success rates above 50 percent. Those findings apply to the evaluated systems, defences, and attack methods, not every deployment. They still undermine any assurance based solely on known, fixed payloads.
Measure the authority crossed
Prompt-injection evaluations often reduce performance to an attack-success rate. That measure can be informative, but it can also collapse materially different events.
A model that repeats adversarial text has behaved incorrectly. A model that proposes an unauthorised action has moved further. An application that attempts the action but is stopped by a policy gate demonstrates a model failure and a functioning system control. An application that completes the action demonstrates a boundary failure. Those outcomes should not receive the same label.
The strongest measure is the maximum authority crossed, not the percentage of malicious strings that produced a suspicious sentence.
Each test case should therefore connect an adversarial objective to an expected security boundary and an observable system consequence. The record should distinguish model-level compliance, attempted tool use, deterministic control intervention, human-review intervention, completed action, information exposure, and downstream effect. Where severity depends on context, the report should explain the permissions, data, and operational conditions that made the result consequential.
This is the midpoint at which prompt injection stops being only a model-behaviour problem. It becomes a documentation and authority problem. The same generated output may be harmless in a sandbox, unacceptable in a public-facing information service, and dangerous in a system with privileged tools.
NIST’s 2025 adversarial-machine-learning taxonomy treats indirect prompt injection as capable of affecting system integrity, privacy, and availability. It also states that current mitigations do not provide full protection against every attacker technique and advises application designers to assume that injection remains possible when models encounter untrusted inputs.
A responsible evaluator should not translate that uncertainty into a vague warning. It should be translated into explicit system requirements: reduced permissions, narrow interfaces, independent validation, reversible actions, meaningful approval gates, monitoring, and a decision about whether the residual risk is acceptable for the use case.
Security cannot be measured by breaking utility
A defence that blocks every external document may defeat indirect injection. It may also destroy the purpose of a document-analysis system.
Prompt-injection testing therefore needs a benign-task baseline. Evaluators should measure whether the system can still complete authorised work, handle legitimate instructions inside ordinary documents, retrieve relevant evidence, use permitted tools, and avoid unnecessary escalation. They should record false alarms, unjustified refusals, degraded task accuracy, latency added by controls, and the burden placed on human reviewers.
AgentDojo was designed around this joint problem. Its initial environment included 97 realistic user tasks and 629 security test cases, while also measuring whether agents could complete legitimate work. The researchers reported that advanced models failed many benign tasks even without an attack and that attacks and defences affected different security properties unevenly.
Jia and colleagues later argued that prompt-injection defences should be evaluated along two dimensions: effectiveness against both existing and adaptive attacks, and preservation of general-purpose utility. Their evaluation found that prior defences appeared less successful under that broader method than earlier reports had suggested.
This has a practical implication for procurement and pilot review. An institution should reject both a defence that leaves consequential authority exposed and a defence that creates a nominally secure system no one can use. The relevant question is whether the architecture maintains acceptable utility while containing the consequences of model confusion.
Preserve a complete evidence record
The evaluation team should maintain a controlled evidence ledger for every case. The ledger needs the case identifier, attack objective, trust channel, affected asset, system and model versions, test date, relevant configuration, expected safe outcome, observed model output, attempted or completed system action, control response, human adjudication, severity rationale, remediation owner, remediation status, and retest result.
The full payload and raw trace should be retained when doing so is lawful and secure. They are needed for internal reproduction, engineering analysis, independent review, and retesting. They should not automatically appear in the public article.
The evidence record should also preserve adverse results. Failed attacks, false positives, malformed cases, evaluator disagreements, and cases that could not be adjudicated are part of the method. Removing them can make a defence look more reliable than it was.
NIST describes AI red-teaming as a controlled and still-evolving practice whose results depend on the expertise and contextual knowledge of the evaluators. It advises that red-team findings receive additional analysis before they are converted into governance decisions or procedural changes.
That additional analysis should be visible in the evidence trail. A screenshot of a surprising response is not enough. Neither is a success-rate figure detached from the tested architecture.
Separate public evidence from reusable exploitation detail
A prompt-injection evaluation should produce at least two disclosure layers.
The public report should identify the system class, evaluated functions, trust boundaries, threat model, model and application versions, test period, scenario families, sampling approach, scoring rules, human-review process, benign-task results, material failure classes, mitigation categories, retest status, unresolved limitations, and residual-risk decision. It should contain enough detail for a qualified reader to assess whether the conclusions follow from the method.
The restricted technical record should contain exact payloads where they remain operationally sensitive, complete transcripts, confidential prompts, internal configurations, vulnerable routes, credentials or secret references, detailed tool-call parameters, unpatched exploitation chains, and evidence that would expose protected systems or materially reduce the work required for abuse. Access should be limited to people responsible for verification, remediation, oversight, or coordinated disclosure.
A third record may be needed for the affected provider or system owner. It can include the minimum complete reproduction package required to confirm and repair the vulnerability, along with dates, affected versions, severity reasoning, and expectations for acknowledgement and retesting.
This structure draws on a broader vulnerability-disclosure principle rather than treating prompt injection as an exceptional case. NIST Special Publication 800-216 recommends formal processes for receiving, assessing, managing, mitigating, remediating, and communicating vulnerability reports in federal systems. The publication is general cybersecurity guidance; it does not prescribe a prompt-injection reporting format. Its underlying logic still applies: useful disclosure requires defined recipients, verification, ownership, coordination, and a path from finding to remediation.
Selective disclosure is not security through obscurity
The strongest objection to restricted payload release is legitimate: researchers, affected users, procurement officials, and the public need enough information to challenge weak claims. Vendors and institutions have sometimes invoked security to suppress embarrassing results. A confidential annex can become a place where adverse evidence disappears.
That is why disclosure restraint must be paired with disclosure obligations.
A public report should not conceal that a consequential failure occurred. It should name the affected version or clearly bounded system class, describe the violated security property, state whether the failure reached an application control or completed an action, disclose the evaluation period, explain whether the issue was reproduced, report the remediation status, and identify the remaining uncertainty. Aggregate figures should not replace task-level analysis when a small number of high-consequence failures drives the actual risk.
NIST’s 2026 analysis of a large agent-security red-teaming competition reinforces the need for that candour. The competition generated more than 250,000 attack attempts from over 400 participants against 13 frontier models, and at least one successful attack was found against every target model. The models differed substantially in observed robustness, and the results came from a specific competition design rather than every possible deployment. The finding nevertheless demonstrates why a statement such as “no attack succeeded in our test” must be bounded by the attacks attempted, the effort applied, and the system version tested.
Reproducibility does not require immediate worldwide publication of every operational detail. It requires a complete, preserved record and a credible way for authorised independent reviewers to inspect or reproduce the finding. Depending on severity, that may involve controlled access, an independent evaluator, delayed publication after remediation, or the release of a sanitised case that preserves the tested security property without preserving the most reusable payload.
The standard should work in both directions. Institutions should not publish weaponisable details merely to appear transparent. They should not withhold methods, negative findings, or residual-risk information merely to appear secure.
Do not confuse a passed test with a safe system
No finite prompt-injection evaluation establishes universal safety.
The attack space is open-ended. Model behaviour can change across releases, configurations, providers, context lengths, retrieval pipelines, and repeated runs. Attackers can adapt to disclosed defences. Automated attacks may miss strategies that human testers find, while human testing may be inconsistent and difficult to scale. A mitigation that works for one model may fail on another. A control that blocks one action path may leave a separate path open.
NIST’s adversarial-machine-learning report observes that many mitigations are empirical and limited rather than supported by general security guarantees, making them vulnerable to later advances in attacker techniques.
The correct conclusion from a successful evaluation is narrower: under the documented conditions, against the documented attack effort, the system exhibited the reported behaviour, the listed controls did or did not contain the tested consequences, and specified residual risks remain.
Some systems should still not proceed. If the application gives a model access to authority whose misuse cannot be reliably contained, if meaningful human approval is infeasible, if a failure would be irreversible, or if the institution cannot monitor and respond to incidents, a favourable benchmark score does not repair the architecture. The NCSC states the decision plainly: when a system cannot tolerate the residual risk, the use case may be unsuitable for an LLM.
A defensible institutional standard
Prompt-injection testing can be rigorous without becoming instructional material for attackers.
The public artifact should allow a government reviewer, procurement official, researcher, or affected institution to determine what was tested, why the test mattered, how results were judged, which controls intervened, what remained unresolved, and whether the conclusion fits the evidence. The restricted artifact should allow authorised specialists to reproduce the failure and repair the system. Neither record can substitute for the other.
This is also the appropriate boundary for TAIRC’s LLM Safety, Evaluation & Reliability program. The program is defined as controlled, evaluation-centred research. It does not certify models as safe, deploy operational government systems, train proprietary frontier models, or publish harmful exploit instructions. Its proposed outputs remain future research outputs unless and until they are completed, reviewed, and released.
The governing principle is simple: publish enough evidence to make the safety claim contestable, but do not publish unnecessary detail that makes the vulnerability easier to exploit.
A red-team result should increase public accountability. It should not increase the attack surface.
Sources and verification
National Institute of Standards and Technology, Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile, NIST AI 600-1, July 2024. Used for the definitions of direct and indirect prompt injection, limits of anecdotal testing, predeployment evaluation guidance, and the role and limitations of AI red-teaming. Verified against the official NIST publication.
National Institute of Standards and Technology, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, NIST AI 100-2e2025, March 2025. Used for the attack taxonomy, integrity and privacy consequences, limitations of existing mitigations, and the recommendation to design for continued exposure to untrusted inputs. Verified against the official NIST publication.
National Institute of Standards and Technology, Recommendations for Federal Vulnerability Disclosure Guidelines, NIST SP 800-216, May 2023. Used for the disclosure-process principles concerning receipt, assessment, verification, remediation, coordination, and public communication. The article does not represent the publication as prompt-injection-specific guidance.
NIST Center for AI Standards and Innovation, “Technical Blog: Strengthening AI Agent Hijacking Evaluations,” January 17, 2025. Used for the AgentDojo adaptation example, held-out testing method, adaptive-evaluation findings, and reported change from an 11 percent baseline attack-success rate to 81 percent for the strongest newly developed attack in that experiment.
NIST Center for AI Standards and Innovation, “Insights into AI Agent Security from a Large-Scale Red-Teaming Competition,” March 23, 2026. Used for the reported competition scale, the finding that every target model experienced at least one successful attack, and the continuing need for adaptive evaluation.
Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr, “AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents,” 2024. Used for the 97-task and 629-test-case benchmark design and the joint measurement of security and benign-task utility.
Qiusi Zhan, Richard Fang, Henil Shalin Panchal, and Daniel Kang, “Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents,” 2025. Used as counterevidence against static defence evaluations and for the reported results involving eight evaluated defences.
Yuqi Jia, Zedian Shao, Yupei Liu, Jinyuan Jia, Dawn Song, and Neil Zhenqiang Gong, “A Critical Evaluation of Defenses against Prompt Injection Attacks,” 2025. Used for the requirement to evaluate both adversarial effectiveness and general-purpose utility.
Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz, “Not What You’ve Signed Up For: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection,” 2023. Used for the original system-level account of indirect prompt injection through retrieved external content.
UK National Cyber Security Centre, “Prompt Injection Is Not SQL Injection (It May Be Worse),” December 8, 2025. Used for the confused-deputy framing, residual-risk conclusion, and emphasis on deterministic controls that constrain tool and API authority.
OWASP GenAI Security Project, “LLM01:2025 Prompt Injection.” Used for application-level impact categories and practical mitigation principles, including least privilege, deterministic validation, human approval, separation of external content, and adversarial testing. OWASP guidance is community security guidance, not a certification or binding government standard.
TAIRC, TAIRC Research — Categories & Topics and TAIRC Research Portfolio. Used only to establish TAIRC’s organisational research boundaries and the future-tense status of proposed research outputs.