Illusions of Understanding
Contents
Key terms
What We'll Cover
Versions of AI come and go. The capabilities listed in 9.1 will date within months. The patched failures in 9.2(a) will be replaced by new patched failures. The strong cases in 9.3 will be joined by new strong cases.
What does not date as fast is the deeper question of what happens to your own understanding when AI is doing more of the cognitive work in your research. That question is the subject of this sub-lesson, and its centrepiece is Messeri & Crockett's 2024 Nature paper, which lays out four distinct illusions of understanding that AI use can produce.
The argument is durable because it is about epistemics, not about specific model behaviour. A frontier 2030 model that's right 99% of the time will produce these illusions more easily, not less.
Messeri & Crockett: The Four Illusions
Required Reading (Centrepiece)
Messeri, L. & Crockett, M. J. (2024). Artificial intelligence and illusions of understanding in scientific research. Nature 627, 49–58. DOI:10.1038/s41586-024-07146-0.
A 10-page Nature Perspective. Read in full before working through the sub-lesson; the four illusions below summarise but do not replace the original argument.
Messeri (Yale, anthropology) and Crockett (Princeton, neuroscience) argue that AI use in research generates four specific illusions of understanding. Each looks like progress; each can mask a real loss of epistemic ground.
1. Illusion of Explanatory Breadth
Researchers using AI to summarise a literature feel they've covered the whole field. They haven't. The AI summary reflects which sources are well-represented in training data, which papers attracted attention, which arguments propagate easily. Marginal voices, dissenting views, and recent work all systematically under-appear. The feeling of breadth is real; the breadth is not.
2. Illusion of Exploratory Objectivity
AI outputs feel neutral; they don't come with a visible agenda. But they inherit the biases of training data, the values encoded by RLHF, the framings the model finds most natural. The neutrality is performative, not actual. A researcher using AI to generate hypotheses thinks they are exploring without prejudice; they are exploring the priors of the model.
3. Monocultures of Knowing
If everyone in a field uses the same frontier models, ideas and framings homogenise. Different researchers asking similar questions of the same model get convergent answers. The field becomes narrower in its diversity of approaches, even as it produces more output. Connect to the “scientific monoculture” finding in Week 6 (Traberg, Roozenbeek & van der Linden, Communications Psychology, 2026): AI use by researchers is measurably narrowing the diversity of ideas in circulation.
4. Doing More But Understanding Less
AI accelerates output. More papers can be written, more analyses run, more literature reviewed. But the cognitive work that produces deep understanding (the slow grappling with a problem, the dead-ends that turn out to matter, the synthesis that emerges from struggle) can be skipped. Output rises; depth falls. Connect to Week 6's “writing as thinking” argument and Kosmyna et al. (2025)'s MIT Media Lab study showing reduced neural engagement in AI-assisted essay writing.
A 2026 Concrete Example: “Vibe Coding”
The term “vibe coding” has emerged in 2025–26 to describe a development style in which programmers describe what they want to an AI assistant in plain English and accept the output without deeply reading the resulting code. The AI handles the syntax; the human handles the intent.
For straightforward tasks like quick scripts and well-trodden patterns, this is genuinely productive. For research code, it produces exactly the illusion Messeri & Crockett describe: output rises (more analyses run, more pipelines built), depth falls (the developer's actual understanding of what the code is doing in detail is shallower than it would have been if they had written it themselves).
This connects directly to the “silent error” problem from Week 7: code that runs without errors but produces wrong results is precisely the failure mode that vibe coding amplifies. If you don't read the code closely, you don't catch the silent errors. The illusion of doing analysis is real; the analysis itself may not be.
The pedagogical point: vibe coding is not wrong. It is a real productivity gain for many tasks. But applying it without verification (Sub-Lesson 9.5) to research-grade work is the contemporary face of doing-more-understanding-less.
Why this argument matters more in 2026
A natural response to the 2024 paper is: well, that was when AI was unreliable. Now that frontier models are good at maths and physics (9.3), surely the illusions are less of a problem, since the AI is more often right.
The argument runs the other way. The illusions are more dangerous when the AI is more often right, because the surface fluency of AI outputs masks the remaining gaps more effectively. A model that's right 60% of the time is obviously to-be-checked. A model that's right 95% of the time looks trustworthy, and the 5% wrong is harder to spot because it doesn't come surrounded by other obvious errors.
The Calibration Trap
If your overall experience with a model is that it's usually right, you start trusting it more. Trust calibrated to overall accuracy is then mis-calibrated for the specific cases where the model fails, which by structural argument (9.2c) cluster in the long tail, where novel research happens.
This is the AI version of the Dunning–Kruger trap: the less you know about a topic, the harder it is to spot the AI's errors in that topic, but the more useful the AI feels precisely because you couldn't do the work yourself. Both cuts of the knife point the same way.
How This Connects to Earlier Weeks
Several earlier weeks have touched on pieces of the illusions argument. The Messeri & Crockett framework gives you a way of seeing them together.
| Earlier week | Concrete instance of an illusion | Which Messeri & Crockett illusion |
|---|---|---|
| Week 5 (Literature reviews) | AI literature summaries miss niche work; over-represent canonical papers | Explanatory breadth |
| Week 6 (Writing & ideation) | Si et al. (2024): LLM-generated research ideas more novel-seeming but less diverse than human-generated | Monoculture; exploratory objectivity |
| Week 6 (Writing as thinking) | Kosmyna et al. (2025): EEG study showing reduced neural engagement during AI-assisted writing | Doing more but understanding less |
| Week 6 (Homogenisation) | Agarwal, Naaman & Vashistha (CHI 2025): AI writing tools homogenise text toward Western norms | Monoculture |
| Week 6 (Scientific monoculture) | Traberg, Roozenbeek & van der Linden (Comms Psych, 2026): heavy AI use narrows scientific diversity | Monoculture |
| Week 7 (Silent errors) | AI-generated code that runs but produces wrong results; researcher feels they've done analysis but hasn't verified meaning | Doing more but understanding less |
| Week 8 (Multimodal) | Jin et al. (2024): GPT-4V medical image accuracy 81.6% but 35.5% of correct answers via flawed reasoning | Exploratory objectivity (apparent precision masking flawed process) |
Each week has given you fragments. Messeri & Crockett is the unifying frame, and the reason the previous fragments matter even as the specific models change.
What to Do About It
The Messeri & Crockett argument is not a counsel of despair. They explicitly argue that AI can be a useful tool in research provided researchers maintain practices that resist the four illusions. Their suggested practices, paraphrased and operationalised:
- For explanatory breadth: deliberately seek out sources that AI summaries miss. Use AI for first-pass familiarisation; then read primary sources, dissenting views, recent work that may post-date training cutoffs.
- For exploratory objectivity: ask the same question of multiple models with different training. When they agree, ask: is this a property of the world or a property of overlapping training data? When they disagree, that disagreement is evidence.
- For monoculture: consciously vary your prompts and your problem framings. Talk to colleagues who don't use the same tools. Read literature outside your immediate field. Treat AI ideas as one input among several, not the privileged starting point.
- For doing-more-but-understanding-less: impose deliberate slowness on parts of the work. Read papers manually before asking AI to summarise them. Write paragraphs without AI before refining with it. Run analyses by hand on small datasets to check intuitions before scaling.
The argument that doesn't go stale
Note what the Messeri & Crockett argument is not. It is not “current AI is unreliable, so don't use it.” It is not “AI is overhyped.” It is not anti-AI. The argument is strictly about what happens to human understanding when AI does more of the cognitive work, regardless of how good the AI is.
That is why the argument doesn't go stale when the next model releases. A better model produces more output; the question of whether the researcher understands that output more or less than they would have without AI assistance is unchanged.
This is the durable epistemic concern. Sub-Lesson 9.5 takes it into practical verification protocols.
What Comes Next
Sub-Lesson 9.5 — Verification Protocols for a Moving Target. Practical techniques for verifying AI output in your own research, including the meta-skill of how to read AI capability claims critically when the artefact under study changes every six months.
Drafted with Claude (Anthropic) and reviewed by Jonathan Shock before publication. AI-generated errors are possible — if you spot one, please email jonathan.shock@uct.ac.za. Full detail: AI Content Disclaimer.
© 2026 Jonathan Shock · MAM5020F: Generative AI for Research · CC BY 4.0