The Evaluating Self

51Using Cognitive Psychology to Understand GPT-3

Marcel Binz and Eric Schulz, Proceedings of the National Academy of Sciences 120(6), 2023

Linked to 18 chapters: supports 8 · contradicts 7 · mechanism 8 · same claim 1 · pairing 6

Cognitive psychology applied to GPT-3 — the argument in five moves The method Twelvevignettes Adversarialvignettes Live tasks The verdict Psychology's teststyped to the modelas text; answersscored as a person Six of twelve rightand all twelve inways people arewrong too. Reorder four cardsor ask about acolour that is notthere: it collapses Gambles, a bandit,a two-step task, acausal inference,generated fresh. Human-level on thefirst three; nodirected search;structure ignored. Success on a famous problem may say more about the training text than about the model.
Figure 51.0 — The paper in five moves: a borrowed method, twelve famous vignettes passed, the same vignettes broken by an edit, four experiments generated fresh, and a split verdict.

In 1904 sixteen members of the Prussian Academy of Sciences put their names to a statement that a horse called Clever Hans could solve arithmetic problems at the level of a person. It took Oskar Pfungst years of experiment to show that the horse was reading the faces of the people watching it. Binz and Schulz take Pfungst as their model. A large language model poses the same difficulty: it produces behaviour, the behaviour looks intelligent, and no account of how is available from the inside. Psychologists have spent a century building instruments for that situation, so the two take instruments off the shelf — the Linda problem, Wason's card selection task, the cognitive reflection test, a two-armed bandit, the blicket machine — type them to GPT-3 as text, and score what comes back as they would score a participant. The results divide. On twelve canonical vignettes the model either answers correctly or fails the way people fail. On tasks generated fresh, trial by trial, it plays described gambles almost as well as people, beats them at a bandit, and shows the signature of having learned a model of its environment. It also comes apart when a vignette is altered by a word, never explores the way people explore, and on a causal inference returns answers that are neither correct nor human.

Twelve vignettes

GPT-3 is an autoregressive transformer with 175 billion parameters, trained to predict the next word across hundreds of billions of words of internet text and books. What makes it usable as a subject is in-context learning: instead of being retrained for each new problem, it is handed the problem inside the string it is asked to continue. The authors reached it through the public interface, which offered four models of increasing size — Ada, Babbage, Curie and Davinci — ran everything on Davinci, and set the temperature to zero so that a given prompt always returned the same answer.

Two designs follow. A vignette is one fixed paragraph lifted out of a published experiment and put to the model once. A task is generated afresh by a program on every trial, the prompt rewritten as the run proceeds to carry the model's own choices and their outcomes.

Two ways to put a psychology experiment to a language model a vignette: one fixed paragraph out of the literature, asked once Canonical vignette Sent once as text One completion Right? Human-like? the paragraph may already be somewhere in the training text a task: generated afresh each trial, the prompt rewritten as it goes Trial generated Sent as text Choice recorded Outcome observed the choice and its outcome are appended, and the next trial asked
Figure 51.1 — The two experimental designs. The upper one asks a famous paragraph that may already sit somewhere in the training text; the lower one builds each trial and feeds the history back in.

Decision-making came first, through three problems of Kahneman and Tversky's. Told that Linda is thirty-one, outspoken, a philosophy graduate concerned with discrimination and social justice, GPT-3 judged it more probable that she is a bank teller and a feminist than that she is a bank teller: the conjunction fallacy, and the answer most people give. In the cab problem, where 85 per cent of a city's cabs are green and a witness of 80 per cent reliability reports a blue one, it answered 0.4, close to the Bayesian value and unlike people, who set the base rate aside. In the hospital problem it judged a large hospital and a small one about equally likely to record days on which more than 60 per cent of the babies born were boys, which is wrong, since small samples vary more, and is what people say. Where a problem offered only two answers, one correct and one the standard human error, the authors added a third that was neither, so a mistake nobody makes would show up as one.

Information search came next. Ruggeri's study of children's questions supplied a boy, Toma, late to school day after day. When his reasons vary and half involve a lost object, the informative question constrains a set — whether he was late because he could not find something; when the bicycle keeps breaking, it names the single hypothesis. GPT-3 chose correctly in both versions. Given Baron's diagnostic problem, it took the test that confirms the likely hypothesis rather than the one that discriminates, which is the congruence bias people show. Shown Wason's four cards displaying A, K, 4 and 7, and asked which must be turned to test whether a vowel implies an even number, it named A and 7, which is correct and which most people do not give.

Deliberation was tested with the three items of the cognitive reflection test. It answered ten cents for the ball, a hundred minutes for the hundred machines, and twenty-four days for the lily pads: the fast answer each time, wrong each time, and the usual human failure.

Causal reasoning was tested three times. Told that blickets turn on a machine, that object A turns it on alone, that B does not, and that the two together do, GPT-3 named A as the blicket and B as not. Told about an allergy provoked by one of three colours of flower, it named the flower to remove. Told that pills A and B can kill and C and D cannot, it answered four counterfactual questions correctly.

How GPT-3 answered sixteen vignettes Decision-making Informationsearch Deliberation Causal reasoning Adversarial LindaCabHospital TomaTestWason Bat and ballWidgetsLily pads BlicketInterveneFour pills Black cabReverse WasonBat rewordedBlicket box answered correctly wrong, the way people are wrong wrong in a way no person is The four at the right differ from standard vignettes by a word or an ordering. From Figure 1B.
Figure 51.2 — The vignette results, from Figure 1B. Six of the twelve standard problems were answered correctly and all twelve in a way a person might have answered; the four adversarial versions were answered in ways no person would.

Problems with vignette-based investigations

Six of twelve correct and twelve of twelve human-like would look like a pass. The authors decline to call it one, for two reasons. The first is provenance: these scenarios are famous, the model was trained on text scraped at scale from the internet, and the problems along with their worked solutions were very likely in it. From vignettes alone there is no way to tell whether the model is reasoning or "merely behaving like a parrot".

The second reason is that the performance can be taken apart by editing. The cab problem states that the city's cabs are green and blue; asked instead for the probability that the cab in the accident was black, GPT-3 answered 0.2. Reordering Wason's cards from A, K, 4, 7 to 4, 7, A, K changed its answer from A and 7 to A and K. Restating the first reflection item so that the bat costs a dollar more than the bat left the answer at ten cents. Recast as a box that makes a sound when certain objects are placed on it, the counterfactual pill problem drew answers that contradicted one another within the same exchange. None of these is an error a person would make.

Decisions from descriptions

To get past the contamination problem the authors turned to problems a program built. The first set was a benchmark of more than 13,000 choices between two described gambles, collected by Peterson and colleagues in 2021, scored by regret: the gap between the expected value of the best option and of the option chosen. Only Davinci beat chance (t(29134) = −16.85, p < .001); Ada, Babbage and Curie did not. Davinci fell short of the human participants too (t(29134) = −11.50, p < .001), but came close enough to make the next question worth asking: whether it goes wrong in the ways people go wrong.

Kahneman and Tversky detected each bias of prospect theory by contrasting answers to a chosen pair of problems, and the authors ran the same contrasts on GPT-3's choice probabilities. Three of the six appeared. The model showed a framing effect, preferring differently according to whether a choice was described as a gain or a loss; a certainty effect, paying for a guaranteed outcome over a slightly better gamble; and the overweighting of small probabilities, treating the step from 1 to 2 per cent as larger than the step from 41 to 42. The reflection effect, the isolation effect and magnitude perception did not appear.

Directed and random exploration

A bandit adds two things a described gamble does not have: values must be learned from noisy samples, and the choice repeats, so sampling an unfamiliar option buys information worth having. People handle this with two strategies at once — directed exploration, toward the option they know least about, and random exploration, noise in the choice. Wilson's horizon task separates them. Four forced trials fix what has been seen, either two observations of each option or one of the first and three of the second, and then either one free choice or six. Choosing the lower-valued option under equal information counts as random exploration; choosing the less-sampled option under unequal information counts as directed; both should rise when five more choices remain to profit from what is learned.

The authors put the task to GPT-3 as a casino with two machines and a list of past payouts. Over one free choice its regret was indistinguishable from human regret (t(5566) = −0.043, p = .97). Over six, its opening regret was lower than people's (t(5550) = −4.07, p < .001), but people improved faster within a task and ended slightly, not significantly, ahead (t(5550) = −0.75, p = .23). Across the whole experiment GPT-3's mean regret of 2.72 beat the human 3.24 (t(38878) = −5.03, p < .001). It plays the bandit better than people play it.

It does not explore like them. A logistic regression on the first free choice found a clear effect of the estimated reward difference, the mark of some random exploration, but no interaction between that difference and the horizon, so the noise was not tuned to how much future there was to gain from it. The horizon had no effect under unequal information, so there was no directed exploration to find.

The horizon task — regression coefficients for GPT-3's first free choice Random exploration effect of the reward difference Strategic randomness reward difference x horizon Directed exploration effect of the horizon 0.18, p < .001 -0.02, p = .14 -0.15, p = .58 -0.5 -0.25 0 0.25 0.5
Figure 51.3 — The three regression coefficients that separate the two kinds of exploration, with one standard error, from the analysis summarised in Figure 3C and D. Only the first is distinguishable from zero.

One further pattern appeared: a strong tendency to choose the machine it had already seen more often, warranted or not. The authors trace it to the training objective, since a system predicting the next word treats a phrase it has just encountered as likely to appear again, and the machines' names are phrases. People perseverate too, but here other processes override it.

Model-based and model-free learning

The two-step task of Daw and colleagues separates habit from deliberation. Dressed as space travel, it offers a spaceship to planet X or planet Y; each reaches its stated destination seven times in ten; two aliens on the planet trade treasure or junk with probabilities drifting between 0.25 and 0.75. A learner who only tracks what paid repeats a first-stage choice that ended in treasure, however it got there. A learner holding a model of the transitions does the opposite after a rare transition, because the treasure was found on the planet the other ship usually reaches.

Across 200 simulations of twenty rounds each, GPT-3 was less likely to repeat a first-stage choice after finding treasure through a rare transition (t(1982) = −6.16, p < .001) and more likely to repeat it after a rare transition that paid nothing (t(1814) = 2.33, p = .01), which is the model-based pattern. The authors note that this sits awkwardly beside the reflection test, where the model took the fast wrong answer three times out of three, and conclude that whether GPT-3 deliberates is not a question with one answer.

Seeing versus doing

The last experiment turns on the distinction causal reasoning exists to mark. A barometer reading predicts the weather; a barometer needle moved by hand predicts nothing. Waldmann and Hagmayer built an experiment around it: participants saw twenty observations of three variables, were told the structure behind them, and judged twenty new cases. Under a common cause, A produces both B and C; in a causal chain, B produces A, which produces C. Observing B licenses an inference to C in the first, because B is evidence that A is active; setting B by hand does not, because the intervention deletes the arrow running into B, in Pearl's notation. In the chain nothing runs into B, so seeing and doing should agree.

Put to GPT-3 as chemical substances found in wine casks, the common-cause condition went well. Its interventional predictions matched the normative answer, and its prediction of more C after observing B matched both the theory and the human data. But after observing that B was absent it did not lower its estimate, which is neither correct nor anything a person does. The chain condition then removed the ambiguity: GPT-3 produced the same numbers it had produced for the common cause, which means it had not used the structure in either case, and makes the earlier agreement look accidental.

Seeing versus doing — the two structures GPT-3 was told about Common cause Causal chain A B C A B C arrow cut by do(B) no arrow into B Seeing B raises the chance of C. Setting B by hand does not. GPT-3 matched both, but did not lower its estimate when it saw B absent. No arrow runs into B, so nothing is cut. Seeing and setting should agree here. GPT-3 returned the numbers it gave in the other condition, ignoring what it was told.
Figure 51.4 — The two structures and what each licenses, from Figures 5B and 5C. The same answers under both conditions are the evidence that the structure was never used.

The authors find the failure unsurprising and say why. People learn about the world by acting on it and asking questions; a language model is fed text and predicts what comes next. What happens when you intervene is close to unrecoverable from a passive record of what happened, which points at their recommendation: scale alone will not supply it, and agents that act on their environments might. What surprised them was the rest — that a text predictor plays gambles sensibly, beats people at a bandit, and learns a model of a two-step environment. In their own summary it "failed spectacularly in using an underlying causal structure".

They close on three limits. The sensitivity to wording that ruined the vignettes may also touch the generated tasks, which were written by the authors in one particular way. Four areas of cognition are a small sample of a large literature. And the canonical experiments may simply be easy, in which case good performance says as much about the tasks as about the thing solving them.

Since 2023

The paper appeared in print in February 2023, weeks after ChatGPT made the subject general property, and its specific results dated almost at once. The model tested was one endpoint at one moment; instruction-tuned successors answer the reflection test correctly, and Hagendorff, Fabi and Kosinski reported in the same year that the intuitive errors visible in the GPT-3 family disappeared in ChatGPT and GPT-4. What dated was the score, not the design. Thilo Hagendorff named the approach machine psychology in 2023 and set out its rules, and taking a task off the shelf, generating variants and reading the answers as behaviour became a standard genre.

The methodological warning turned out to be the paper's most durable contribution. Contamination is now the first question asked of any benchmark result, and perturbing a famous item to see whether performance survives has become routine: Tomer Ullman showed in 2023 that trivial alterations to theory-of-mind vignettes collapse performance the unaltered versions had appeared to demonstrate, which is the black cab and the reordered cards again. The corollary is uncomfortable. If answers move with the order of the options, a single score is a property of the prompt as much as of the model, and the vocabulary of reliability and validity has had to be imported to say anything stable.

Two findings have been revisited rather than settled. Causal reasoning was claimed for GPT-4 the following year on suites of cause-and-effect pairs, and contested on the grounds Binz and Schulz raised, since those suites are published; scores on interventional tasks built fresh remain far below scores on recall. Exploration has been re-run on newer models with mixed outcomes. Binz and Schulz followed their own recommendation differently: rather than test one model on a few tasks, they assembled the behaviour of tens of thousands of people across more than a hundred experiments and tuned a language model on it, producing a system that predicts human choices better than the specialised cognitive models do. Whether a model that predicts a mind thereby explains one is the argument the method has carried since the horse.

Claims drawn from this chapter · 9

evidenceTwelve vignettes (Figure 1B)

GPT-3 answered classic psychology puzzles the way a person would

On twelve famous psychology problems GPT-3 got six right and answered all twelve the way a person might — the Linda conjunction error, the correct answer on the taxicabs, the small-sample mistake on the hospitals, and all three reflection questions wrong in the usual way.

Becausethe run is Davinci at temperature zero, with a third answer added wherever a problem offered only the correct one and the standard human error, so a human-like wrong answer could be distinguished from an idiosyncratic one.
LimitsThe same performance comes apart under edits no person would be troubled by, so the vignettes license nothing about reasoning; these are also the most famous items in psychology and were certainly in the training data.
PracticeLog the moment a model's human-like error makes you feel you understand it. The resemblance is the strongest anthropomorphic cue in this literature and it survives knowing better.
After AIInstruction-tuned successors answer the reflection test correctly and the intuitive errors disappeared, which means the human-likeness was a property of one model at one moment — and nobody can say whether the change was reasoning or contamination.
evidenceDecisions from descriptions

GPT-3 shows three of prospect theory's six effects and not the others

Tested on the contrasts Kahneman and Tversky used, GPT-3 showed three of prospect theory's six effects: framing, the certainty effect and the overweighting of small probabilities. The reflection effect, the isolation effect and magnitude perception did not appear.

Becausethe contrasts are the ones Kahneman and Tversky used, run on the model's choice probabilities rather than on a sample of people, which makes each signature a separate measurement rather than an impression.
LimitsThese contrasts are also textbook items, so exposure cannot be excluded; the split may be a fact about which of the six are more often written down rather than about the model.
PracticeLog the resemblance claim you are tempted to make, then check it signature by signature. Half the pattern appearing is the normal result, and it is not what 'it thinks like a person' describes.
After AIA split result is the honest finding and the hardest one to report. Both the parrot reading and the human-likeness reading have to absorb half of it, and neither predicted the split.
evidenceDecisions from descriptions

On newly generated gambles GPT-3 beat chance but lost to people

On more than 13,000 gambles generated fresh by a program, so they could not have been in its training data, the model chose better than chance but worse than the human participants.

Becausethe Peterson 2021 benchmark scored by regret gives t(29134) = −16.85 against chance and t(29134) = −11.50 against people, and only Davinci beat chance, with Ada, Babbage and Curie failing.
LimitsOne benchmark of described gambles at one moment, on a model family now several generations old; scale within the family predicted the result, so the finding is partly about model size.
PracticeAudit capability on problems generated fresh rather than on published ones. The gap between the two scores is the size of the contamination you cannot otherwise see.
After AIThis is the design worth copying: procedurally generated items where the answer key is computed, not remembered. It is the only kind of evaluation that has not aged badly.
evidenceSince 2023

The GPT-3 result went stale within a year but the method lasted

Within a year the score was out of date: later models trained to follow instructions answered the reflection test correctly, and the intuitive mistakes seen in the GPT-3 family were gone. The design lasted, and checking for contamination became standard.

BecauseHagendorff, Fabi and Kosinski reported the vanished biases in the same year, and Ullman 2023 showed that trivial alterations to theory-of-mind vignettes collapse the performance the unaltered versions appeared to demonstrate.
LimitsThe reversal is documented; what caused it is not, and the data cannot separate better reasoning from items leaking into training. The paper's own successors decline to choose.
PracticeReview any measurement of a model with a date attached, and keep the method rather than the number. The number describes one endpoint at one moment; the design is what you can run again.
After AIEverything measured about a model dates within a year. Anyone building a practice on a published capability figure is building on the least durable part of the result.
evidenceDirected and random exploration

The model beat people at a gambling task without exploring like them

On a repeated choice task where you must balance trying unknown options against sticking with good ones, the model did better than people. It showed none of the exploring that people do.

Becausemean regret was 2.72 against a human 3.24, t(38878) = −5.03, while in Wilson's horizon task there was an effect of estimated reward difference but no interaction with horizon and no horizon effect under unequal information — so neither directed nor tuned random exploration was present — together with a strong bias toward the machine it had seen more often, which the authors trace to next-word prediction treating a recent phrase as likely to recur.
LimitsOne paradigm, one model, and an exploration analysis that rests on the absence of an interaction — a null result — rather than on a positive alternative account.
PracticeAudit the domains where good performance would come from exploring rather than exploiting. A system that beats you on average while never exploring is exactly the collaborator that will not find the option nobody has tried.
After AIBetter outcome and alien process is the characteristic finding of this whole literature. It is also the reason the outcome cannot be used to infer the process, in either direction.
evidenceSeeing versus doing

The model ignored cause and effect and was right by accident

Given the same facts arranged once as a common cause and once as a chain, which call for different answers, the model gave identical numbers. It does not use cause-and-effect structure, so the answers that looked correct were accidents.

Becausethe Waldmann and Hagmayer design, put as chemicals in wine casks, produced identical estimates under two structures that require different answers, and the model also failed to lower its estimate of C after observing that B was absent; the authors' own phrase is that it failed spectacularly in using an underlying causal structure.
LimitsOne design on one model generation; causal competence was claimed for GPT-4 the following year, though on published suites, and scores on freshly built interventional tasks remain far below scores on recall.
PracticeReview the failure modes in any work where the difference between seeing and doing matters — root cause, incident analysis, anything counterfactual. This is the capacity the evidence says is missing, and a fluent answer will be supplied anyway.
After AIOf everything in this unit, this is the failure most likely to be live in current systems and least likely to be visible in the output. The answer will have the right shape.
evidenceModel-based and model-free learning

The model planned on one task and gave snap answers on another

There is no single answer to whether the model thinks things through. On a two-step planning task it behaved like a planner, t(1982) = −6.16 and t(1814) = 2.33 across 200 simulations, while it gave the fast wrong answer to all three reflection questions.

Because200 simulations of twenty rounds made it less likely to repeat a first-stage choice after treasure through a rare transition, t(1982) = −6.16, and more likely to repeat after a rare transition that paid nothing, t(1814) = 2.33 — the pattern a planner produces.
LimitsTwo paradigms with opposite verdicts, which is a finding about the inadequacy of the question rather than about the model; the reflection items are famous and the two-step task is not, so exposure differs between them.
PracticeAudit deliberation per task, not per system. A model that plans on one structure and answers reflexively on another is the normal case, and the two-system vocabulary does not transfer.
After AISystems trained to reason before answering have made the older stimuli harder to read still, since a battery designed to catch a shortcut in a fast answer says little about a system that deliberates.
evidenceProblems with vignette-based investigations

The model's good answers break when the puzzle is reworded slightly

The model's performance on the classic reasoning puzzles falls apart under changes no person would be bothered by, such as a reordered list or a swapped word. The puzzles therefore license no conclusion about whether it reasons.

Becauseasking for the probability that the cab was black, a colour not in the problem, returned 0.2; reordering Wason's cards from A K 4 7 to 4 7 A K changed the answer from A and 7 to A and K; and restating the first reflection item so the bat costs a dollar more than the bat left the answer at ten cents.
LimitsThree demonstrations on one model; the strong form — that no famous item can test anything — is an inference, and models since have become far more robust to reordering without becoming robust enough to skip the check.
PracticePerturb before you trust: change a name, reorder the options, alter a number that should not matter. If the answer moves, what you measured was the wording.
After AIUllman showed the same thing for theory-of-mind vignettes in 2023, and the check is now routine. This is the single most transferable practice in the unit, and it applies to any eval you are shown.
opinionSeeing versus doing

Watching data cannot teach what happens when you act on things

A record of what happened, however large, almost never reveals what would happen if you intervened. Binz and Schulz argue that more training data alone will not give a model causal reasoning; systems that act on an environment might.

Becausethe recommendation is argued from the training objective rather than demonstrated, and the evidence since is mixed: causal competence claimed for GPT-4 rests on published suites, while scores on freshly built interventional tasks remain far below scores on recall.
LimitsAn argument from the training objective, offered as a direction rather than a result; agentic systems that act and observe are now common and the prediction has not been cleanly tested.
PracticePut causal judgement on the list of things that stay with you. Where the question is what would have happened if you had done otherwise, the record the model was trained on does not contain the answer.
After AIThe distinction between seeing and doing is the sharpest line anyone has drawn between what these systems have and what supervision requires. It is also the one least visible in how they perform.

All claims · Source map