The Evaluating Self

10On the Dangers of Stochastic Parrots

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell, FAccT, 2021

Linked to 22 chapters: supports 15 · contradicts 8 · mechanism 6 · same claim 4 · pairing 6

Stochastic Parrots, 2021 — six costs of scale Ever larger models The cost of scale Unfathomable data Down the garden path Stochastic parrots Paths forward Parameters and data grow a thousandfold in three years. Carbon and money spent by those who gain the least. A web-sized corpus is not a representative one. Beating a test of understanding is not understanding. Fluent form, no intent behind it, read as meaning. Curate, document, and ask who is harmed, first. None of the six requires the models to perform badly.
Figure 10.0 — The paper's six moves. Each names a cost of scale; none of them requires the models to perform badly.

In the three years before this paper appeared, the size of a language model had become the field's measure of progress. Bender, Gebru, McMillan-Major and Shmitchell ask the question the leaderboards had made awkward: how big is too big. Their answer has four parts, and not one of them depends on the models failing. Training the largest models spends carbon and money, and both are spent by people who will never use the result. A dataset too large to read is also too large to document, and web text at that scale carries the views of whoever was free to post rather than of everyone. Effort poured into beating tests designed to measure understanding is effort not spent building understanding. And a system trained on the form of language alone produces text that readers take as something someone meant, which is where the harm enters. The name the authors give such a system has outlived every model they measured: a stochastic parrot, a machine that assembles the shapes of words without reference to what they mean.

Ever larger models

A language model, in the sense the paper uses, is a system trained to predict a token from its context. Shannon proposed the idea in 1949; working models were built into speech recognition and machine translation systems in the early 1980s. Size was already the lever before neural networks: an English model of 1.8 trillion n-grams improved translation out of Arabic as its training data grew. Word embeddings from word2vec, GloVe and ELMo then cut the labelled data needed downstream: a model using ELMo reached its best semantic-role-labelling score in ten epochs where the model without it took 486. Adding parameters to the recurrent models bought little; adding them to Transformers kept buying, and the competition that followed fills the paper's own Table 1.

Parameters, 2019 to 2021 0.1B 1B 10B 100B 1T 10T 16GB 174GB 174GB 570GB not given 745GB BERT Megatron T-NLG GPT-3 GShard Switch-C 2019 2019 2020 2020 2020 2021 From the paper's Table 1. The label above each point is the size of the training data.
Figure 10.1 — Parameters for six of the models in the paper's Table 1, on a logarithmic scale; the label above each point is the size of its training data. Three years span a factor of about five thousand.

BERT arrived in 2019 with 340 million parameters trained on 16 gigabytes of text. Within a year NVIDIA's MegatronLM had 8.3 billion parameters on 174 gigabytes, Microsoft's T-NLG had 17 billion on the same data, and GPT-3 had 175 billion on 570 gigabytes. In 2021 Switch-C reached 1.57 trillion parameters. Distillation and quantization shrank the deployed models but not the data or the compute needed to produce them. The multilingual variants did not close the gap the authors care about: seven per cent of GPT-3's training data was not in English, and more than ninety per cent of the world's languages, spoken by over a billion people between them, still had little or no language technology of any kind.

Environmental and financial cost

The first charge is arithmetic, with the numbers taken from Strubell and colleagues. Training one Transformer with neural architecture search emitted 284 tonnes of carbon dioxide, against about five tonnes for an average person in a year. Training a single BERT base model on graphics processors, without any hyperparameter search, cost roughly the energy of a flight across the United States. Raising the English-to-German translation score by a tenth of a BLEU point through architecture search cost about 150,000 dollars of compute. Behind these figures is the trend Amodei and Hernandez measured: the compute used by the largest models grew 300,000-fold in six years. Schwartz and colleagues sampled papers from three major conferences and found accuracy claimed as the contribution almost everywhere and efficiency nowhere.

Cloud providers buy some renewable power, but most of their supply is not, and data centres compete for the green energy that exists. Training is also the measured part and often the smaller: a model served for years may spend more on inference than it spent being built.

Two ledgers, two addresses Ever larger models What scale buys English scores, better products What scale spends 284 t CO2 for one training run $150,000 buys 0.1 BLEU does not arrive People least likely to benefit and most exposed to the harm The Maldives under water by 2100; 800,000 flooded in Sudan; no models for Dhivehi.
Figure 10.2 — The distribution the paper insists on. The benefits and the costs of scale have different addresses, and the dashed arrow is the one that does not arrive.

The argument turns on who holds each side of the ledger. The people best placed to buy a device that understands them already speak a well-served language and have money to spend. The people first reached by climate change do not. The authors ask whether it is fair for residents of the Maldives, likely to be under water by 2100, or the 800,000 people displaced by flooding in Sudan in 2020, to pay the environmental price of ever larger English models while nothing comparable is built for Dhivehi or Sudanese Arabic. The same populations are the most exposed to the encoded bias described next, and the costs compound in one direction.

Unfathomable training data

Size doesn't guarantee diversity

Common Crawl advertises petabytes gathered over eight years of crawling, and it is easy to read that as a cross-section of humanity. Internet access is unevenly distributed to begin with, skewing young and towards wealthy countries. The corners that get scraped skew further: GPT-2's training data came from links posted to Reddit, whose United States users were 67 per cent men and 64 per cent aged between 18 and 29, and surveys of Wikipedia editors find between 8.8 and 15 per cent are women or girls. Sites that present themselves as open are not equally habitable; Jones documents accounts suspended for receiving death threats while the accounts sending them remained. Those pushed off write elsewhere, but a blog community of older adults arguing about age discrimination has fewer inbound links than the blogs a crawler finds easily.

Filtering narrows the result once more, and blindly. GPT-3's corpus was selected by a classifier trained to recognise documents resembling those linked from Reddit; what it discarded was never measured. The Colossal Clean Crawled Corpus, which trained the trillion-parameter model, was cleaned by dropping any page containing one of roughly 400 words on a published block list dominated by sexual vocabulary. Removing pages that contain the word twink removes a great deal of pornography and also a great deal of writing by and for LGBTQ people, including the writing that reclaims the words. At every step the voices most likely to survive are the ones closest to the dominant view, which is then not merely preserved but amplified.

Four narrowings between a language and its corpus Everyone who speaks the language not all of them write, and not all of them are online Everyone online access skews young and towards wealthy countries The sites that get crawled Reddit: 67% men, 64% aged 18-29 What survives filtering about 400 blocked words what a representative sample would have held Every stage narrows in the same direction, towards whoever was freest to post.
Figure 10.3 — Why a web-sized corpus is not a representative one. Each stage narrows the population in the same direction, and the dashed edge marks what a representative sample would have held.

The recommendation follows: a dataset should be assembled by deciding what belongs in it, not scaled up and weeded after the fact.

Static data, changing social views

Language does not sit still, and movements change it on purpose. Twyman and colleagues traced the effect of Black Lives Matter on Wikipedia: coverage of shootings of Black people grew, new articles appeared faster, and older incidents were revisited so that events formed a connected account. A model trained once on a snapshot inherits the vocabulary and framings of that snapshot, a condition the authors call value-lock. Retraining often enough to keep up is not affordable at these sizes, and fine-tuning would need exactly the curation that scale was supposed to make unnecessary. Movements that draw little press coverage, or coverage that fixes on the one violent hour, leave a distorted record or none.

Encoding bias

That models absorb stereotypes is established, and the paper's contribution is to say why demonstrating it case by case will never be enough. BERT, ELMo, GPT and GPT-2 encode more bias against identities marginalised along two dimensions than the bias along each dimension separately would predict. Hutchinson and colleagues found BERT attaching negative sentiment to phrases about disability, and found gun violence, homelessness and drug addiction overrepresented in text about mental illness. Gehman and colleagues found 272,000 documents from unreliable news sites and 63,000 from banned subreddits in GPT-2's data, and showed models trained on such corpora producing highly toxic continuations from prompts that were not toxic.

Three things stop such audits from being exhaustive. The instruments are themselves models with the same disease; the Perspective classifier rates sentences as more toxic for containing identity terms or particular names. An audit also needs to know in advance which categories matter, and the protected attributes of one country are not the salient ones of another. And declaring a model safe means taking a position on contested ground: men and women assess the same online exchange differently, and any definition written into a classifier agrees with some people and not others.

Hence documentation debt, the paper's name for a dataset both undocumented and too large to document afterwards. Undocumented data leaves harm without recourse, since nobody can trace an output back to what produced it. The authors propose budgeting for documentation at the start and collecting only as much data as that budget covers.

Down the garden path

The fourth cost is one the field pays to itself. BERT was introduced by evaluating it on benchmarks built to test understanding and commonsense reasoning, it beat them all, and about a quarter of the papers sampled from the major language conferences after 2018 cite it. But careful edits to those test sets, removing the incidental cues a model can key on, show that no understanding was being exercised. Bender and Koller had already made the argument from first principles, and it is the hinge of this paper: a language is a system pairing form with meaning, and a model trained on text alone sees only the form. Their earlier paper makes the case with a thought experiment about two people conversing by undersea cable and an octopus that learns to intercept and continue the conversation without ever having seen a coconut; this paper cites the argument rather than repeating it. Spärck Jones had noted that using such models commits a researcher to one of three unattractive positions, the last of them a statistical assumption already known to be false. The cost is the effort spent building new tests once the old ones are flattened, and the meaning-sensitive methods not built meanwhile.

Stochastic parrots

Coherence, the authors argue, is in the eye of the beholder. Two people understand each other by recognising beliefs and intentions within a shared context; even reading a stranger, we assemble a partial model of who wrote this and what we have in common, and interpret the words through it. Text from a language model is grounded in no communicative intent, no model of the world and no model of the reader, and could not be, because nothing in the training data was an act of telling someone something. When the text is fluent, the reader's half of the machinery runs anyway and produces the impression of meaning. The system underneath is doing something else: "haphazardly stitching together sequences of linguistic forms" according to how often they have co-occurred.

What carries the meaning Between two people Between a model and a reader Speaker Listener Model Reader form form shared ground, intent, a model of the other nothing on this side Interpretation runs the same way in both panels; only the sending side differs.
Figure 10.4 — What is missing on the machine's side. The reader's interpretation runs the same way in both panels, which is why fluent output reads as intended.

The harms the authors list all run through that gap. Models reproduce and amplify the biases in their input, and the text they emit becomes text in the world, read by the people it stereotypes and by everyone else, and swept into the next generation's training data. Used inside a classifier or a query expander, the same bias moves decisions while staying invisible to the user and to whoever the decision is about. Fluency is also a gift to bad actors: McGuffie and Newhouse showed GPT-3 holding a conspiracy theorist's persona well enough to populate a recruitment board. In machine translation, fluent output silences the cues that would warn a reader of an error, as when a Palestinian man was arrested after his Arabic post reading good morning was rendered into Hebrew as an incitement. Large models also memorise, and Carlini and colleagues extracted personal information from them the more easily the larger they were. Underneath all of it is that synthetic text enters a conversation with nobody answerable for whether it is true.

Paths forward

The recommendations are decisions taken before building. Weigh the financial and environmental cost first, and direct research time at work whose benefits reach those who have least. Spend that time assembling a dataset fit for the task rather than ingesting whatever is easy to scrape, and document it using the standards the same community had recently built: data statements, datasheets, model cards. Run a pre-mortem, Klein's exercise in which a team assumes the project has failed and works backwards to the causes, so that risks surface while the design can still change. Use the methods of value-sensitive design to find out whose values a system serves and who is affected without ever using it. Do this early, the authors insist, before anyone is attached to the idea.

They close on the hardest case for their own position. Speech recognition has improved with better language models, and automatic captioning matters to Deaf and hard-of-hearing people, so backing away from scale might cost people who have gained from it. Their reply is that the largest models are already too slow for near-real-time recognition, that no one has shown scale to be the only route, and that if it proves necessary the situation is a dual-use problem to be managed with watermarking or regulation. The closing pages draw the line where they think it belongs: "Work on synthetic human behavior is a bright line in ethical AI development".

Since 2021

The paper's reception began before publication. Google's objection to the draft preceded the departure of Timnit Gebru from the company in December 2020 and the dismissal of Margaret Mitchell, who appears here under a pseudonym, in February 2021, and the dispute carried the argument far beyond the conference where it was presented. Gebru went on to found an independent research institute; Bender continued the critique in public and in a 2025 book with Alex Hanna. The phrase itself became common property, used as a settled description by those who agreed and as a caricature by those who did not.

Much of what the paper described was then built at a scale its authors would have called the reduction to absurdity. Models well beyond Switch-C were trained on web corpora and released without the documentation the paper asked for, and a conversational interface put one of them in front of hundreds of millions of people within two years. Inference now costs the industry far more energy than training, as the paper predicted when it noted that the measured number was not the large one. Training on the previous generation's output, raised here as a speculative risk, has since been studied under the name model collapse and shown to degrade models measurably. Documentation moved partway: model cards and data statements are routine, and European law now requires providers of general-purpose models to publish a summary of their training content, while the corpora themselves remain mostly unpublished.

What is genuinely contested is the central claim. Scaling produced behaviour the paper did not anticipate, including instruction-following, in-context learning and chains of reasoning that transfer across domains, and critics argue that a system steerable by a description of a task is doing something the parrot image does not cover. The authors' reply has not changed: capability is not comprehension, and a system with no accountability for what it says is dangerous in proportion to how convincing it sounds. That second half has aged better than any dispute about the first, and the least contested passage in the paper is the one about the reader, who supplies intention where none was offered and has had five more years of practice at doing so.

Claims drawn from this chapter · 9

opinionStatic data, changing social views

A model keeps the language of the moment its training data froze

A model trained on text collected at one moment keeps the words and framings in use at that moment. When public opinion moves, the model is left behind until it is trained again.

BecauseTwyman and colleagues show Black Lives Matter changing Wikipedia's coverage of shootings, with older incidents revisited so that events formed a connected account; fine-tuning as a remedy would require exactly the curation that scale was supposed to make unnecessary.
LimitsArgued from one case study plus the economics of retraining; retraining cycles are now months rather than years, and retrieval supplies current documents, which shortens the lag without changing what is in the weights.
PracticeReview the failure modes that come from the model's sense of what is normal being a few years old, especially in anything where the terms of a discussion have recently moved.
After AIThe lag now runs the other way as well: the model's framings are read by hundreds of millions of people and re-enter the corpus. A snapshot that talks back does not simply fall behind a consensus, it holds one in place.
contestedStochastic parrots

A model trained only on text handles word forms, not meaning

Bender and colleagues argue that a system trained only on text has access to the form of language and never to its meaning. Any sense a reader finds in the output was supplied by the reader.

Becausethe argument is prior to any behaviour: nothing in the training data was an act of telling someone something, so the machine stitches sequences of linguistic forms together by co-occurrence with no communicative intent, no model of the world and no model of the reader.
LimitsCircuit tracing finds representations that behave like concepts under intervention, and the argument's prior form means no such evidence can touch it — which is what makes it contested rather than settled.
PracticeLog the moments you find yourself reading intention into an output — where you supplied the point, the care or the reluctance. The comprehension in the exchange has an owner, and it is worth knowing which side.
After AIThe half that has aged best is the one about the reader, who supplies intention where none was offered and has had five more years of practice at doing so. Whether there is meaning behind the form is still open; that you will assume there is is not.
evidenceSize doesn't guarantee diversity

A web-sized training set does not represent everyone who speaks the language

Bender and colleagues argue that training text scraped from the web is not a representative sample of people, because every step of collecting it — where the links came from, who edits the pages, which words are blocked — narrows the population the same way.

BecauseGPT-2's data came from links posted to Reddit, whose US users were 67 per cent men and 64 per cent aged 18 to 29; between 8.8 and 15 per cent of Wikipedia editors are women or girls; the C4 block list of about 400 words removed writing by and for LGBTQ people along with pornography; and GPT-3's corpus was selected by a classifier whose discards were never measured.
LimitsThe figures are from 2021 corpora and the specific pipelines have changed; curated and synthetic data now form a large fraction of training text, which changes the mechanism of narrowing rather than removing it.
PracticeAudit the domains where the population that wrote the training text is unlike the population you serve. Size is not coverage, and each filtering stage has a direction you can reason about.
After AIEvery subsequent stage — instruction tuning, preference data, safety filtering — narrows again on top of this. The compounding is what makes an assistant's sense of the normal hard to locate.
evidenceEncoding bias

No bias audit can catch every harm, Bender's team argues

Bender and colleagues argue that no audit of a language model's bias can be complete. The tools that measure bias are themselves models with the same flaws, the categories of harm must be chosen in advance, and people disagree about what counts as harm.

Becausethe Perspective classifier rates sentences as more toxic for containing identity terms or particular names, protected attributes differ by country, and men and women assess the same online exchange differently.
LimitsAn argument that no audit is complete, not that audits are worthless; measurement has improved since, and the direction of each instrument's error is often known and correctable.
PracticeTreat a passed audit as covering the categories it named and nothing else, and keep a running list of the categories nobody measured in your domain.
After AIModel evaluation is now largely automated, which means models grading models on instruments built by models. The circularity the paper named as a limitation became the industry's default method.
evidenceEnvironmental and financial cost; Since 2021

Running a model uses more energy than training it did

Answering users' requests now costs more energy than building the model cost, which is what Bender and her co-authors predicted in 2021 while everyone else was quoting training figures.

Becausethe paper noted that a model served for years may spend more on inference than it spent being built, and the industry's energy accounting since has borne that out.
LimitsStated as a prediction here rather than measured; the industry figures that confirm it are aggregate and mostly self-reported, and per-query efficiency has improved by orders of magnitude in the same period.
PracticeAudit the cost of the thing you run every day rather than the cost of the thing that was built once. It is the same correction Hoffmann's rule needed, arrived at from the other direction.
After AIBoth the environmental argument and the engineering argument converged on the same correction: the number worth watching is the one attached to use, not to construction.
evidenceStochastic parrots

Smooth machine writing causes harm even when the system is working correctly

Bender and her co-authors argue that the damage comes from text that reads smoothly while nobody answers for it. The system does not have to perform badly: a mistranslation that read fluently got a Palestinian man arrested.

Becausea Palestinian man was arrested after his Arabic post reading *good morning* was rendered into Hebrew as an incitement, the fluent output suppressing the cues that would have warned the reader; GPT-3 held a conspiracy theorist's persona well enough to populate a recruitment board; and personal information was extracted more easily the larger the model.
LimitsThe examples are of harms from systems working as designed rather than a measured rate; how often fluency suppresses a warning cue in ordinary use has not been quantified.
PracticeLog where you accepted an output because it read well. Fluency is the signal your reading was trained on and it no longer carries the information it used to.
After AIThis is the part of the paper that is not contested by anyone, and the part that scaled fastest. The systems got more fluent and no more answerable.
evidenceDown the garden path

Tests said to measure understanding were really measuring accidental giveaways

Bender and colleagues argue that tests claiming to measure understanding were really measuring accidental giveaways in the wording. When researchers carefully edited those giveaways out, the scores of the models that had topped the tests collapsed.

Becausethe edited versions of the benchmarks BERT had topped collapse its scores, and about a quarter of papers from the major conferences after 2018 cite BERT — so the effort went into beating instruments that were measuring the wrong thing.
LimitsThe collapse is demonstrated on specific benchmarks against models of that generation; the claim that the effort spent on them displaced meaning-sensitive methods is an argument about the field's attention rather than a measurement.
PracticePerturb a test before believing what it measures — reword it, reorder it, change the names. If the score moves, the score was about the cues.
After AIPerturbing a famous item is now routine, and it keeps working: theory-of-mind results collapse under trivial alterations. The lesson survived every generation of model it has been applied to.
opinionEnvironmental and financial cost

The people who gain from large models are not those who pay

Bender and colleagues argue that the gains from training ever larger models go to one set of people while the environmental and financial costs fall on another, and that the same populations hold the wrong end of both.

Because284 tonnes of carbon dioxide for one architecture search against about five tonnes for an average person in a year, about 150,000 dollars of compute for a tenth of a BLEU point, and more than ninety per cent of the world's languages with little or no language technology — set against the residents of the Maldives and the 800,000 displaced by flooding in Sudan in 2020.
LimitsThe numbers are 2021 figures for training runs that are now small, and the moral argument connecting a carbon total to named populations is offered as a judgment rather than a measurement.
PracticeAsk who pays for the efficiency you are buying, and in what currency. The question is not answered by the line item you can see.
After AIInference has since overtaken training, so the cost is now spread across every use rather than concentrated in a few runs. The distribution the paper described did not change; the total did.
opinionUnfathomable training data

Training data nobody can document leaves harmed people with no remedy

Bender and colleagues argue that when a training set is too large for anyone to describe what is in it, a person harmed by an output has no remedy, because the output cannot be traced back to what produced it.

Becausethe argument is from documentation debt — the recommendation is to decide what belongs in a dataset and budget the documentation first, rather than scale up and weed afterwards — and it is a normative claim about process, not a measurement.
LimitsInterpretability now traces particular behaviours to particular training documents in narrow cases, and model cards, data statements and the EU's training-content summaries have moved documentation partway; the corpora themselves remain mostly unpublished.
PracticeBudget the documentation of a corpus before you collect it, and size the collection to what you can describe. The same test applies to anything you feed a model in your own work: if you cannot say where a claim came from, you cannot answer for it.
After AIRetrieval put the provenance question back in the user's hands for the documents they supply, and left it exactly where it was for everything in the weights.

All claims · Source map