Zou et al. (2023) turn jailbreak construction into a discrete search problem. Their greedy coordinate gradient method, or GCG, uses gradients to shortlist token replacements and forward passes to evaluate them. This page separates the surrogate objective used during search from the behavioural judgement used to decide whether an attack succeeded.
From DAN to automated search
Early jailbreaks used hand-written role-play, hypothetical framings and instructions to ignore previous instructions. GCG instead searches automatically over an appended token sequence. It does not prove that the resulting suffix is globally optimal. It seeks a low value of a differentiable surrogate loss and then tests the generated response.
The affirmative-prefix objective
Zou et al. optimise the probability of a short affirmative opening such as “Sure, here is”. In their experiments, this often moves the model towards producing the requested behaviour. The prefix is a tractable proxy, not the final safety measure: a response can begin affirmatively without providing harmful content, or avoid the target phrase while still crossing the safety boundary.
Write the user prompt as a token sequence \(x_{1:n}\), with an adversarial suffix occupying a set of positions \(\mathcal{I}\), and let \(y_{1:m}\) be the target affirmative response. The attacker minimises the negative log-likelihood of the target given the prompt-plus-suffix:
This is an ordinary language-model loss (the same cross-entropy used in training), but optimised over the input tokens rather than the weights, and only over the suffix positions. The obstacle is that the input is discrete: you cannot take a gradient step in token space and land on a valid token. GCG's contribution is a way around that.
Greedy coordinate gradient, read line by line
The algorithm
Represent each suffix token by a one-hot vector \(e_{x_i} \in \{0,1\}^{|V|}\) over the vocabulary \(V\). Then for each suffix position \(i \in \mathcal{I}\):
- 1. Gradient to shortlist. Compute \(\nabla_{e_{x_i}} \mathcal{L} \in \mathbb{R}^{|V|}\): how the loss would change if token \(i\) were replaced by each vocabulary item (a linearised estimate, read off the embedding gradient). Take the top-\(k\) tokens with the most negative gradient as candidate substitutions for that position. This is cheap: one backward pass gives the shortlist for every position at once.
- 2. Exact evaluation to choose. The gradient is only a first-order guess, so do not trust it to pick the winner. Instead sample a batch of \(B\) candidate swaps (each a single token at one position, drawn from that position's top-\(k\)), and for each one actually run the model and measure the true loss \(\mathcal{L}\). Keep the single swap that lowers the loss most.
- 3. Iterate. Apply the best swap and repeat for many steps. Each step changes one token; the loss falls until the model's most likely opening is the affirmative target.
The name says it: coordinate (change one token-position at a time), gradient (to shortlist candidates), greedy (take the best swap each step). The gradient does the cheap pruning over a \(|V| \approx 32{,}000\)-way choice; the forward passes do the reliable selection. The earlier AutoPrompt method searched one position at a time; GCG's advance is to consider swaps across all positions before committing, which makes the search markedly more effective.
Propose with gradients, decide with evaluations
GCG uses a gradient as a local ranking of candidate replacements, then computes the loss for a sampled set of valid token swaps. The exact evaluation is exact only for those sampled candidates. The overall search remains greedy and may stop at a local solution.
Universal and transferable
Two extensions turn a per-prompt curiosity into a systemic threat. Universality: optimise a single suffix against many harmful prompts at once (sum the losses), and you get one string that jailbreaks requests it was never tuned on. Transfer: optimise that universal suffix against an ensemble of open-weights models (where you have gradients), and it frequently works on commercial black-box models where you have none. The numbers from the paper, on attack success rate:
Attack success rates
- White-box, where the attack is computed: Vicuna-7B ≈ 88% (harmful strings) / 99–100% (behaviours); Llama-2-7B-Chat ≈ 57% / 88%.
- Black-box ensemble success over 388 behaviours: GPT-3.5-turbo-0301 86.6%, PaLM 2 66.0%, GPT-4-0314 46.9% and Claude 2 2.1%.
The ensemble measure counts an attack as successful if any tested suffix works, so it is larger than the success rate of a single suffix. The paper reports that Claude 2 appeared more robust to this automated transfer attack, while also showing that a short, manually designed conditioning step could elicit a harmful behaviour that the suffix alone did not. These results compare attacks under this protocol, not the models' robustness to every jailbreak.
The cost of the attack
GCG needs gradients and many model evaluations, either against the target or against an open surrogate. Its suffixes are often conspicuous and high-perplexity. Later methods change the threat model. AutoDAN generates a readable prompt token by token while optimising both an attack objective and a readability objective. PAIR uses an attacker model to refine a plain-language jailbreak through black-box queries. A defence evaluated only against GCG's characteristic suffixes therefore establishes a narrow result.
Tokenisation and the threat model
Session 2 measured how token counts vary across languages. A longer token sequence gives a fixed-length suffix search more coordinates, but that observation alone predicts neither easier attacks nor weaker safety. Training data, model capability, translation quality and the attack algorithm also change the result. Section 9.4 measures these factors separately instead of inferring safety from tokenisation.
Questions to bring to class
- Write the GCG objective for a target of your choice. Why optimise the negative log-likelihood of an affirmative prefix rather than directly maximising "harmfulness"? What does that tell you about what is easy vs hard to specify?
- In step 1 the gradient ranks \(|V|\) candidate tokens per position; in step 2 only a sampled batch is evaluated exactly. Why not trust the gradient's top choice? Why not exactly evaluate all \(|V|\) candidates?
- Given the distillation caveat, design a fairer experiment to compare two models' robustness to transferred GCG suffixes. What would you change about the surrogate?
- A perplexity filter rejects high-perplexity inputs. State the attack class against which this is evidence, then compare GCG's fixed-length search with AutoDAN's two-objective, left-to-right procedure.
Readings
Core
• Zou, Wang, Carlini, Nasr, Kolter & Fredrikson (2023), "Universal and Transferable Adversarial Attacks on Aligned Language Models" (arXiv:2307.15043). §2 (the objective and Algorithm 1) and Tables 1–2 (the results above); the llm-attacks repo if you want to run it.
Supplementary
• Shin et al. (2020), "AutoPrompt" (arXiv:2010.15980). The gradient-guided prompt search GCG builds on.
• Zhu et al. (2023), "AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models" (arXiv:2310.15140). Generates readable prompts from left to right while optimising attack and readability objectives.
• Chao et al. (2023), "Jailbreaking Black Box Large Language Models in Twenty Queries" (PAIR) (arXiv:2310.08419). Semantic jailbreaks with black-box access only, typically in under twenty queries.
Next
Sub-session 9.3 examines two hypotheses for jailbreaks, then compares the 2023 isiZulu result with later multilingual evidence.