The Evaluating Self

52Machine Psychology

Thilo Hagendorff, Ishita Dasgupta, Marcel Binz, Stephanie C. Y. Chan, Andrew Lampinen, Jane X. Wang, Zeynep Akata and Eric Schulz, "Machine Psychology", arXiv 2303.13988, 2023; revised 2024

Linked to 20 chapters: supports 10 · contradicts 6 · mechanism 9 · same claim 2 · pairing 4

Machine Psychology, 2023 — the argument in five moves Behaviour,not weights The test setbroke Whattransfers How torun it Thin andthick words Read inputs andoutputs, not theweights. Anyonecan do it. Web-scale dataleaves nothingheld out, and theskills emerge. Biases, theory ofmind, language,in-contextlearning. Rewrite borroweditems, vary theprompt, controlthe test. Mental words addexplanation thecircuit diagramcannot give. What psychology lends is its experimental discipline, not its findings.
Figure 52.0 — The paper in five moves: a change of level from weights to behaviour, the collapse of the held-out test set, the paradigms that transfer, the rules that must travel with them, and the argument for using mental words at all.

A language model is trained to do one thing, predict the next token, and then does a great many things nobody trained it to do. There are two ways to find out what those things are: open it and read the weights, or leave it closed and put questions to it. Hagendorff argues for the second, and for taking the questions from experimental psychology, which has spent a century designing stimuli that separate one internal explanation from another in a subject that cannot be opened. The proposal is to treat the model as a participant: run it through the paradigms built for people — reasoning problems, false-belief stories, priming, repeated games — and read the construct off the behaviour. What psychology lends is not its findings but its experimental discipline: controls, hypotheses fixed before the data, batteries of items rather than one, and a long-earned caution about inferring a capacity from a performance. What must not travel with it is the assumption buried in the vocabulary, that the thing answering has a mind arranged like ours.

Behaviour rather than weights

Most attempts to explain a neural network look inside it. Mechanistic interpretability reads weights and activation patterns and tries to show how they implement the behaviour on the surface; to make its causal interventions possible it simplifies the data, the model, or both. Machine psychology works at the other end, studying the relation between what goes in and what comes out, in static models and in models manipulated during and after training, and inferring about internal mechanisms without opening them.

Four things recommend that level. The behaviour at the interface is where people meet the model, and it is usually too complex to predict from what is understood about weights and activations. Many behaviours worth studying appear only in models with billions of parameters, and a method that takes behaviour itself as the measured variable scales to them unchanged. And it can be applied to closed commercial systems by anyone with access to the interface. The programme is not benchmarking. A benchmark exists so that systems can be compared and improved, and returns a score; machine psychology is not primarily interested in how well a model performs but in what the performance reveals about the construct producing it, because that is what supports predictions — how the model will generalise, what different training data would do to it, where it will fail.

The model as a participant Construct Stimulus Battery Behaviour if X, then Y written fresh many wordings Y, Z or neither Control condition the other route to Y, ruled out What may be concluded the construct X, and nothing wider Without the control, the same behaviour supports several explanations at once.
Figure 52.1 — The experimental loop the paper proposes. The hypothesis is fixed before the run, and the control condition is what licenses reading a construct off a behaviour.

What the held-out test set no longer does

Machine learning's evaluation rested on a split: train on one set, test on data the model has not seen. Two things broke it. When training data is scraped at the scale of the internet, a test set is hard to keep genuinely held out. And a foundation model is trained only to predict the next token while displaying abilities nobody encoded — learning a new task from a few examples in the prompt is the standard case — so the behaviour of interest is not the trained task at all. Behaviour that emerges is hard to study through the components that produced it: as the paper puts it, "the most interesting evaluations are not 'held-out' exemplars of the training task".

The response has been test-only datasets, too small to train on and meant purely as instruments: BIG-bench with more than two hundred tests, the Abstraction and Reasoning Challenge, and others since. Several resemble the instruments of the behavioural sciences — personality inventories, intelligence tests, implicit association tests — which never obeyed a train-test split either. They fall into two kinds: some return a scalar, and others are built not to be scored but to expose a strategy, so that the answer reveals which of two constructs the subject is using. The paper is about the second kind.

The easiest version of the second kind is also the worst. Faced with a black box that produces language, the obvious move is to ask it about itself. Self-report has a questionable record in psychology, and in models the properties it is used for — personality, morality, clinical traits — move so much with the wording that differently prompted models are now used to stand in for different human populations. Hagendorff does not discard it: it characterises behaviour under a default prompt, and varying that prompt measures how far the model can be steered. But such an answer is a property of one prompt on one system, not a fact about the model.

The tradition worth importing is the experimental one. There, behaviour remains the measured variable, but the stimulus is designed so that different behaviours map onto different internal representations: compositionality, theory of mind, causality, logic. Experiments are hypothesis-driven — if the subject has construct X, behaviour Y should follow, otherwise behaviour Z — and everything then turns on two questions. Does behaviour Y reliably imply construct X? Only if the design has ruled out the other routes to Y, which is what a control is for. Does the absence of Y imply the absence of X? Usually not. People make mistakes on easy arithmetic without lacking arithmetic, and that gap between what a subject does on the day and what it can do is the performance–competence distinction Chomsky drew in 1965; Firestone in 2020 and Lampinen in 2022 carried it to machines. A failed test is weak evidence of a missing capability, and a passed test without a control is weak evidence of a present one.

What a test result licenses the behaviour appears the behaviour is absent the capabilityis there the capabilityis missing The test worked Performance failure Memory or shortcut The test worked the capability is there, the prompt did not draw it out the item was memorised, or another route produced it Both off-diagonal cells are common, and each calls for a different repair.
Figure 52.2 — Why a behaviour is not a capability. Both off-diagonal cells are live, and each has its own remedy.

The many aspects of intelligent behaviour

Not every paradigm transfers. Those that need a body, motor learning and pupillometry among them, do not, though the space widens as models take images and act through tools. Hagendorff sorts the purely linguistic ones into four areas.

Four areas that transfer through language alone Heuristicsand biases Socialinteraction Psychologyof language Learning Framing effects andreasoning problems.Human-like biasesfaded in the newestmodels. False belief, theninduced belief, thengames betweenmodels. Results movewhen tasks change. Surprisal, priming,garden paths.Larger models matchhuman errorpatterns more nearly. In-context learningwas not designed,so the algorithm itruns is unknown. Paradigms that need a body do not transfer; the purely linguistic ones do.
Figure 52.3 — The four areas surveyed, with what each lends and what it has already found.

Heuristics and biases came first. Binz and Schulz were among the earliest to put a language model through the reasoning problems of Tversky and Kahneman, and found GPT-3 showing some of the same biases as people. Then a reversal: the human-like biases visible in the previous generation of models have largely vanished in the newest ones. Two explanations are available and the paper declines to choose. The stimuli were designed to be hard for people and may no longer be hard for a model that reasons better; or the items have leaked into the training data. Framing changes how people solve a problem, and Dasgupta and colleagues found in 2022 that models, like people, reason more accurately about believable, familiar material than about the same logic stated abstractly. And since human heuristics are adapted to the environments people live in, the analogue for a model is its training distribution: Chan and colleagues traced in-context learning back to properties of the data, items arriving in clusters and a long tail of rare classes.

Social interaction has drawn most of the attention. The false-belief tests Wimmer and Perner built in 1983 for four-year-olds were handed to GPT-3, which struggled, and then to later models, which infer unobserved mental states more reliably. Around that sit comparisons with children aged seven to ten, higher-order versions requiring recursive reasoning about several minds, and Ullman's demonstration that small distracting alterations to a task can undo the result. From detecting a false belief it is a short step to inducing one, which is the deception work, and to the trade-off a model makes between honesty and helpfulness. Past the single model lie repeated games, where models show persistent cooperative and competitive signatures.

The psychology of language contributes the oldest methods, since connectionist work was studying syntax in small predictive networks decades ago. Surprisal measures show that models learn filler-gap dependencies; priming reveals how they represent what they have just read; deconfounded stimuli separate real syntax from semantic shortcuts. On pragmatics, larger and instruction-tuned models come closer to human judgements and human error patterns, and garden-path sentences lead them down the same wrong reading they lead people down.

Learning looks like the area least in need of psychology, because the training algorithm was designed and is known. The exception is the one that matters. In-context learning was not designed; it appeared, and what algorithm it implements is unknown, which puts it in the position of a human learner's. Studies compare in-context behaviour with the outputs of hypothesised algorithms, as cognitive modelling has always done, and ask what psychology asks of any learner: what its inductive biases are, how it generalises, and how its behaviour depends on the data it was given.

Good behavioural experimentation

Computer science has not historically been an empirical science, and machine learning became one under conditions that no longer hold: a test set fixed for all practitioners and effectively infinite. Behavioural experiments are small and test-only, and a study that never checks whether its conclusion survives rewording risks reporting something spurious. Psychology has had its own reproducibility crises, and machine psychology "should not share the same fate".

The commonest failure is contamination. Studies lift vignettes verbatim from published psychology and give them to a model that has almost certainly read the paper they came from, so what looks like reasoning may be a familiar token pattern reproduced. Hagendorff's rule is to keep the structure and change everything else: new wording, new agents, new order of events, new actions. Tasks generated procedurally rather than drawn from a fixed set are less exposed by construction.

Sample size is the second failure. Because small changes in wording move outputs, a single phrasing measures nothing; what is needed is a battery of varied prompts, so that one can tell whether a behaviour recurs systematically. Models also carry biases in how they read a prompt at all: toward material near the end of it, toward tokens common in training, and in few-shot settings toward the label that appears most often among the examples. A study that does not design around them may be measuring a shortcut.

Two roads to the same right answer Published study Its vignette The model with its vignette used word for word answers correctly In the training data along with its answer It recalls a pattern the test measured nothing It solves the task the test measured that Keep the structure; change the wording, the agents and the order of events
Figure 52.4 — Why a verbatim vignette proves nothing. Two routes produce the same answer, and only a rewritten item tells them apart.

Elicitation cuts the other way: leaving out the prefixes and suffixes that improve reasoning underestimates what the model can do. Appending an instruction to think step by step raises performance across many tasks, and sampling several reasoning paths and taking the majority answer raises it further. Multiple-choice formats help in some cases and hurt in others, since a short-answer format can cut reasoning off before it happens; answer order should be shuffled across runs. Run across model sizes, since the largest is not always the most capable, and fix a temperature of zero or a seed when reproducibility is wanted, remembering that a chosen seed carries its own bias. Scoring is its own problem, because contemporary models produce long, variable answers; instructing the model to put its final answer after a marker string preserves the reasoning and leaves only the tail of the output to classify.

Thin and thick descriptions

The last section is about a word problem. Once models are treated as participants, mental vocabulary follows — reasoning, intuition, creativity, intelligence, personality, even mental illness. Critics urge caution on the ground that the mechanisms behind these words differ in people and machines, and that the words are normatively loaded, widening the gap between what specialists mean and what the public hears.

Hagendorff takes the objection and refuses the conclusion, borrowing a distinction from ethnography. A thin description explains a system only by its internals: the activation atlas, the circuit, the weight. A thick description adds a layer of explanation in mental terms, and adds explanatory power even where no neural correlate for those terms exists. That is already the human case: the words used to explain what a person did correspond to no identifiable set of neural activations either, and nobody concludes that they explain nothing. A thin reading assumes a model has syntax and no semantics; a thick one assumes there are regularities beyond syntax to be found, and machine psychology is the attempt to find them.

The outlook follows from the method. Multimodal and tool-using models widen the space of paradigms that transfer, since experiments requiring perception or action become runnable. And once a battery is fixed it can be given to each generation in turn, which yields what this field has never had: longitudinal data on a moving subject, and a basis for extrapolating where reasoning is going rather than only describing where it is.

Since 2023

The article grew with its subject: the version circulating by August 2024 carried seven co-authors from an industrial laboratory and three universities, and had been rebuilt around the four areas above. The name stuck, and behavioural batteries have been packaged as benchmarks, so that a set of psychological tasks and the metrics for scoring them can be run against many models at once.

The warnings have aged better than the results they were aimed at. Contamination and prompt sensitivity turned out to be the field's central measurement problems rather than side conditions, and the theory-of-mind literature became the case study: reported successes moved when the task was reworded, while controlled comparisons found models matching human participants on some tests and failing conspicuously on others. Survey-style instruments proved as unstable as the paper predicted, shifting with option order and formatting, which is why a model's measured personality is now reported as a property of a prompt.

Two things have moved past the programme as written. Binz and Schulz, whose work opens the survey, went on to train a model on a large archive of psychology experiments to predict what people do in them, which inverts the borrowing: psychology's data used to build a model of the person, rather than psychology's tasks used to test the machine. And systems trained to reason before answering have made the older stimuli harder to read still, since a battery designed to catch a shortcut in a fast answer says little about a system that deliberates. The thick description has meanwhile won by default: mental vocabulary is now standard in technical writing about these systems, applied by people who rarely pause over whether the word is earned, which is at once what the paper argued for and what it warned about.

Claims drawn from this chapter · 9

evidenceGood behavioural experimentation

A model's low test score may just mean it was asked badly

When a model scores low on a test, you cannot tell whether the model lacks the ability or the question was put badly. Telling it to think step by step raises scores on many tasks, and sampling several reasoning paths raises them further.

Becausethe elicitation results come with a set of procedures — multiple-choice formats cut reasoning off before it happens, answer order should be shuffled, model sizes should be run across since the largest is not always the most capable, and a marker string separates the answer from the reasoning for scoring.
LimitsThe elicitation gains are themselves scale-dependent and were measured on models where chain-of-thought was still a prompting technique rather than trained-in behaviour.
PracticeSpecify how you asked before reporting what a model could not do. Elicitation is part of the measurement, and a failure you have not tried to elicit past is not a finding.
After AIMost public claims that a model cannot do something are elicitation results in disguise. The same is true in the other direction, which is why both need the procedure stated.
opinionWhat the held-out test set no longer does

Benchmark tests no longer measure what a language model can do

Holding back a set of test questions no longer measures anything. Training data scraped from the whole internet makes a test hard to keep unseen, and the behaviours worth studying were never what the model was trained to do.

Becausethe argument is that the most interesting evaluations are not held-out exemplars of the training task, with in-context learning as the standard case of a capability nobody encoded.
LimitsStated as a prescription rather than demonstrated, and held-out sets still work where the data is genuinely new — private corpora, freshly generated items, tasks invented after the training cut-off.
PracticeStop treating a benchmark score as a measurement and start treating it as a claim about what the model saw. Where you need to know whether a system can do your work, the test has to be one you built.
After AIEvery published benchmark is now in the training data of the next model. The measurement problem is structural and gets worse with each release, not better.
evidenceGood behavioural experimentation

Hagendorff says a model's self-description tells you about the prompt

Hagendorff argues that questionnaires given to a model measure the prompt, not the model. Scores on personality, morality and clinical tests shift so far with wording and the order of options that researchers use differently prompted copies to stand in for different human groups.

Becausethe survey-instrument results have held since the paper predicted them, so a measured personality is now reported as a property of a prompt, and self-report is kept only to characterise a default and to measure steerability.
LimitsInstability under rewording is well established; whether there is a stable underlying disposition that better instruments could reach is not settled, and steerability measurements assume there is a default to steer from.
PracticeLog every time you take a model's statement about itself as information — its confidence, its preferences, its account of its limits. None of it is a reading of anything durable.
After AIUsers ask assistants about themselves constantly, and vendors publish the answers as character. The circuit-level finding is the same from the other side: the account and the computation are different processes.
opinionThe many aspects of intelligent behaviour

Hagendorff says a test without a control proves little either way

Passing a test without a control condition is weak evidence that a model has an ability, and failing one is weak evidence that it lacks it. What a system does on the day and what it can do are different things.

BecauseChomsky's 1965 performance-competence distinction was carried to machines by Firestone in 2020 and Lampinen in 2022, and the everyday case is that people make mistakes on easy arithmetic without lacking arithmetic.
LimitsA borrowed distinction argued by analogy; it cuts both ways and can excuse any failure, which is why it needs the control it recommends rather than standing alone.
PracticeReview both directions of your evidence. A model that failed your task may fail your prompt, and one that passed may have seen it — and neither result means much without a control condition you designed.
After AIThe asymmetry matters for hiring a system as much as for hiring a person. Most deployment decisions rest on a handful of passed tests with no control at all.
opinionBehaviour rather than weights

Hagendorff says watching what a model does is a real explanation

Hagendorff argues that studying what a model does when you use it is a proper way to explain it, because its outputs cannot be predicted from what is known about its weights. For a closed commercial model it is the only way open to outsiders.

Becausefour arguments are offered for the level — where people meet the model, complexity beyond what weights explain, behaviours that appear only at scale, and access through an interface — set against mechanistic interpretability, which must simplify the data, the model or both to make its interventions possible.
LimitsAn argument for a method rather than a result; circuit-level work now explains particular behaviours the behavioural level can only describe, and the two levels have not been reconciled.
PracticeAudit the systems you use by what they do, in your domain, on your work. It is the level you have access to, and for a closed model it is the only evidence anyone outside the laboratory has.
After AIThe model is the first research subject that answers questions on demand and cannot be opened. That combination is why psychology's instruments were borrowed, and why its measurement problems were inherited with them.
contestedThin and thick descriptions

Hagendorff says words like believes and wants really do explain machines

Describing a model as believing or wanting something explains its behaviour better than a purely mechanical description, even though nothing inside the model corresponds to those words. Hagendorff notes that this is already true of the words we use about people.

Becausethe ethnographic distinction is borrowed and argued by symmetry rather than tested; the counter-case is that the mechanisms differ and the words are normatively loaded, widening the gap between what specialists mean and what the public hears.
LimitsContested and conceded in practice: the thick description won by default, adopted by writers who never pause over whether the word is earned, which is at once what the paper argued for and what it warned about.
PracticeLog which mental words you use about a system and what each one would have to mean to be earned. The vocabulary is useful and it is also the mechanism by which projection becomes permanent.
After AIThe words are now standard in technical writing, which means the argument was settled by usage rather than by evidence. That is worth noticing every time one of them does explanatory work in your own reasoning.
contestedSince 2023

Newer models stopped showing human biases and nobody knows why

The human-like reasoning errors found in one generation of models had largely gone in the next. Two explanations fit equally well, better reasoning or test items that leaked into the training data, and the evidence cannot tell them apart.

Becausethe reversal is reported against Binz and Schulz's own finding and the paper declines to choose between the explanations; Dasgupta 2022 sits on the other side, finding that models, like people, reason more accurately about believable familiar material than about the same logic stated abstractly.
LimitsContested because it is undecidable with the available evidence, not because the sides disagree about the facts; a decisive test would need a model whose training data could be inspected.
PracticeLog the interpretation you are inclined to give a model improvement, and note that the contamination explanation is always available and rarely tested. Holding both is the honest position.
After AIImprovement and exposure are confounded in every published capability trend. That is a permanent condition of evaluating systems trained on everything, and it applies to the ones you are buying.
evidenceGood behavioural experimentation

One phrasing of a question measures nothing about a model

Testing a model with a single phrasing measures nothing, because its answers shift with where things sit in the prompt, with how common the words were in training, and with the answer most often shown in the examples.

Becausethe prompt-sensitivity results collected there are why a battery of varied prompts rather than an item is the unit of measurement.
LimitsPrompt sensitivity has fallen with each generation without disappearing, and a battery is expensive, which is why single-prompt results remain the norm in practice.
PracticeRun anything you care about several ways before concluding the model can or cannot do it. One prompt is one sample, and the variance between phrasings is often larger than the difference between models.
After AIThe sensitivity traces back to a mechanism with no intrinsic sense of order, patched from outside. That is the clean line from the architecture to the measurement problem.
opinionGood behavioural experimentation

Testing a model with a published puzzle measures what it has read

Hagendorff argues that a test item copied word for word from a published experiment measures how much of that paper the model has seen as much as its ability. Only an item rebuilt with the same structure and new wording tests anything.

Becausethe rule is stated as a prescription — keep the structure and change everything else, wording, agents, order of events, actions — with procedurally generated tasks named as less exposed by construction; the warning has aged better than the results it was aimed at.
LimitsA prescription rather than a finding, and a hard one to satisfy: rebuilding a task around its structure may change what it measures, and there is no way to confirm that it has not.
PracticeSpecify your evaluations by structure rather than by item. Write down what the task is testing, then generate instances — it is the only form of test that survives the next model release.
After AIThis is the most durable advice in the unit, and it is the reason the paper outlived the results it was surveying.

All claims · Source map