A jailbreak tries to bypass a model's behavioural safeguards. Prompt injection changes which instructions an application follows. An indirect injection places those instructions in data the application retrieves, such as a web page, email or document. Once the application can call tools, this becomes a system-security problem involving permissions, data flow and the consequences of an action.
Indirect prompt injection
A direct injection appears in the user's input. An indirect injection is embedded in external content that an application later gives to the model. Greshake et al. (2023) demonstrated attacks through retrieved web pages and other application data. The model receives trusted instructions and untrusted content in the same token stream, and may treat text inside the content as an instruction.
A worked threat model
Consider an assistant asked to summarise an email. The email contains hidden text instructing the assistant to send private correspondence to an external address. The attacker controls the email, the application supplies it as untrusted content, and success means an unauthorised tool call or disclosure. A test that checks only the quality of the summary misses the security outcome.
Automated red-teaming
Perez et al. (2022) used one language model to generate tests for another, with a classifier scoring the target's responses. On the model and offensive-language classifier they studied, reinforcement-learning optimisation of the red model increased the estimated offensive-reply rate from about 3.7% for zero-shot generation to above 40%. The comparison shows that an evaluation's attack-generation method can dominate its result. Session 11 returns to elicitation and validity in more detail.
Bounded guarantees
Two defence families come from image-domain adversarial ML. Adversarial training trains against the strongest perturbation found inside an allowed set:
The inner optimisation approximates the strongest available attack inside the specified set. Certified defences prove a guarantee without relying on attack search. For randomised smoothing, if class \(A\) has lower-bounded probability \(p_A\) and every competing class has upper-bounded probability at most \(p_B\), the smoothed classifier is constant within an \(L_2\) radius
The certificate covers that classifier, noise distribution, norm and radius. It does not cover a translation, a multi-turn conversation or an instruction embedded in a document.
Changing the attack
Zou et al. note that image-domain adversarial training and certification are rarely used in practice because of computational cost, lower clean performance and narrow attack models. Input filtering has the same scope problem. A perplexity filter may detect GCG's conspicuous suffixes, while AutoDAN explicitly optimises readability and PAIR uses ordinary-looking language. A defence result should name the attacks used during development and include adaptive attacks designed with knowledge of the defence.
Layered controls for applications
Model-level refusal is one control in an application. System controls reduce what a successful injection can do:
- Separate trust levels. Mark retrieved text as untrusted data and keep it distinct from developer instructions in the application protocol.
- Minimise privileges. Give each tool only the data and actions required for the current task.
- Validate actions. Check tool arguments against application rules and require confirmation for consequential actions.
- Constrain data flow. Prevent untrusted content from selecting destinations or carrying private data into outbound requests.
- Evaluate adaptively. Test the complete application against attackers who know the proposed defence.
These controls do not make the model immune to injection. They reduce the authority available to an injected instruction and create places where the application can reject unsafe actions.
Coverage and resources
Multilingual red-teaming requires speakers who can judge translation, comprehension and harm in context. Compute is one cost; evaluation data and skilled review determine which languages a defence has tested. A vendor should report that coverage directly rather than extending an English result to languages absent from the evaluation.
Questions to bring to class
- In an application that summarises email and can send replies, list the assets, untrusted inputs and tool permissions. Which control limits the damage if the model follows an injected instruction?
- Write the adversarial-training objective for translation. Why does its allowed set not fit the \(\max_{\|\delta\|\le\varepsilon}\) form?
- Perez et al. found that changing the red-team method changed the measured failure rate sharply. What does that imply for an evaluation using only zero-shot prompts?
- Write a robustness claim that names the model, attack set, application boundary, languages and success measure. Which part is a guarantee and which part is empirical?
Readings
Core
• Greshake et al. (2023), “Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection” (arXiv:2302.12173). The attack class and threat taxonomy.
• Zhan et al. (2025), “Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents” (arXiv:2503.00061). Tests eight defences using attacks adapted to each one.
Supplementary
• Perez et al. (2022), “Red Teaming Language Models with Language Models” (arXiv:2202.03286). Automated red-teaming and the elicitation effect.
• Madry et al. (2018), “Towards Deep Learning Models Resistant to Adversarial Attacks” (arXiv:1706.06083). Robust optimisation and PGD.
• Cohen, Rosenfeld & Kolter (2019), “Certified Adversarial Robustness via Randomized Smoothing” (arXiv:1902.02918). The certified-radius result.
Next
Session 10 turns to machine unlearning, backdoors and the AI-control agenda. Its control protocols assume that the model itself may act adversarially, then ask how much harm a surrounding system can prevent.