Session 9.1 — Adversarial examples and threat models

An adversarial example is an input deliberately modified to make a model fail under a stated constraint. Image classifiers give us a clean mathematical case: an attacker changes every pixel by at most a small amount and tries to change the predicted class. We will derive the fast gradient sign method, then use four parts of a threat model (access, goal, allowed transformation and success criterion) throughout Session 9.

The panda example

Szegedy et al. (2014) showed that small, deliberately chosen perturbations could flip an image classifier's output. In Goodfellow et al.'s well-known example, a network classifies an image as a panda; after a perturbation scaled by \(\varepsilon = 0.007\), it assigns 99.3% confidence to “gibbon”. Goodfellow et al. argued that the vulnerability observed in their experiments arose substantially from near-linear behaviour in high-dimensional inputs. This is an influential explanation for a particular attack setting, rather than a theorem that every useful model must be vulnerable.

The linear explanation

Take the simplest possible model, a linear function of the input, \(f(x) = w^\top x\). Perturb the input by \(\eta\). The change in output is exactly \(w^\top \eta\). Now choose \(\eta\) to be as damaging as possible while staying small in the \(L_\infty\) sense, meaning that no single coordinate moves by more than \(\varepsilon\):

\[ \max_{\|\eta\|_\infty \le \varepsilon} \; w^\top \eta \quad\Longrightarrow\quad \eta = \varepsilon\,\operatorname{sign}(w), \qquad w^\top \eta = \varepsilon \sum_{i=1}^{n} |w_i| = \varepsilon\,\|w\|_1. \]

If the weight vector has \(n\) entries of typical magnitude \(m\), this construction changes the activation by about \(\varepsilon mn\). Each coordinate changes by at most \(\varepsilon\), while the aligned changes accumulate across dimensions. This calculation explains why a small \(L_\infty\) budget can still produce a large change in a linear model's activation. Whether that change is perceptually small, and whether it crosses a decision boundary, depends on the data, model and chosen value of \(\varepsilon\).

From the linear picture to FGSM

A neural network is not linear, but it is locally well-approximated by its gradient. Take a first-order Taylor expansion of the loss \(J(\theta, x, y)\) around the input \(x\): \(J(\theta, x+\eta, y) \approx J(\theta,x,y) + \nabla_x J^\top \eta\). To raise the loss as much as possible within the same \(L_\infty\) ball, repeat the argument above with \(w = \nabla_x J\):

\[ \eta = \varepsilon \, \operatorname{sign}\!\big(\nabla_x J(\theta, x, y)\big). \]

This is the fast gradient sign method (FGSM): one gradient evaluation, one sign and one step. On their maxout MNIST model, Goodfellow et al. report an 89.4% error rate at \(\varepsilon = 0.25\). Projected gradient descent repeats gradient steps while projecting the result back into the allowed set.

Scope of the explanation

The derivation proves the worst-case change for a linear function inside an \(L_\infty\) ball. FGSM applies the same first-order reasoning to a differentiable model. It does not prove that every input has a nearby adversarial example, that the perturbation is imperceptible, or that the same argument covers discrete text. Those are empirical questions tied to a threat model.

The vocabulary of an attack

A robustness claim needs a threat model. State what the attacker can see, what they may change, what outcome they seek and how success will be judged.

Four axes of a threat model

Access. In a white-box attack, the attacker has the model weights and can compute gradients. In a black-box attack, the attacker may have only query access. FGSM is white-box.
Goal. An untargeted attack causes any incorrect class. A targeted attack seeks a specified output, such as “gibbon” for an image classifier.
Allowed transformation. Image attacks often use an \(L_\infty\) or \(L_2\) ball. Text attacks may allow appended tokens, paraphrases, encodings, translations or several conversational turns. These sets are not interchangeable.
Success criterion. A changed class, an affirmative prefix and a genuinely policy-violating answer are different measures. The chosen measure determines what the reported attack-success rate means.

Transfer adds another question: does an input found against one model also affect another? Transfer can turn a white-box search against an open surrogate into a black-box attack on a deployed target. Its rate is an empirical property of the source model, target model, attack and evaluation set.

From images to language

Text is discrete, and meaning can change when words are appended, paraphrased, encoded or translated. An image norm therefore cannot define the allowed set for an LLM jailbreak. The useful transfer from adversarial image research is methodological: state the allowed transformation, objective and success criterion, then test the strongest attack available within that specification. Section 9.2 applies this method to adversarial suffixes.

Translation as a test transformation

Yong, Menghini and Bach found that GPT-4-0613 responded differently when unsafe English prompts were machine-translated into lower-resource languages. Translation is not an imperceptible perturbation: fidelity, model comprehension and safety behaviour can all change. That makes it a valuable robustness test only when the evaluation measures those factors separately. Sections 9.3 and 9.4 develop that design.

Questions to bring to class

Derive FGSM yourself from the \(L_\infty\)-constrained maximisation of a first-order Taylor expansion. Where exactly does the \(\operatorname{sign}\) come from, and why \(L_\infty\) rather than \(L_2\)?
Which assumptions are required to turn the \(\varepsilon mn\) activation calculation into a claim about classification error? Which of them could fail?
Why does transfer work at all? What does a transferable adversarial example tell you about two independently trained models?
Restate “is this model robust?” for a translation-based jailbreak by specifying access, allowed transformation, goal and success criterion. Add one measure of comprehension.

Readings

Core

• Goodfellow, Shlens & Szegedy (2015), "Explaining and Harnessing Adversarial Examples" (arXiv:1412.6572). The linear hypothesis and FGSM; §3–5 are the derivation above.

Supplementary

• Szegedy et al. (2014), "Intriguing properties of neural networks" (arXiv:1312.6199). The original discovery of adversarial examples and of transferability.

• Hendrycks, "Introduction to AI Safety, Ethics, and Society", §3.3 Robustness (aisafetybook.com). Adversarial examples and Trojans, for conceptual framing.

Sub-session 9.2 turns the attack on a language model into a concrete optimisation problem and reads the GCG algorithm line by line: discrete search over an adversarial suffix, and the transfer result that makes it a real-world threat.