82What Is Intelligence?

Blaise Agüera y Arcas, What Is Intelligence? Lessons from AI About Evolution, Computing, and Minds, Antikythera / MIT Press, 2025

Linked to 25 chapters: supports 12 · contradicts 8 · mechanism 6 · same claim 6 · pairing 3

Agüera y Arcas argues that prediction is not one component of intelligence but the whole of it, and that the same principle, taken far enough, explains life as well. He defines life as self-modifying computronium selected for dynamic stability, and intelligence as the ability to model, predict and influence one's own future. Each definition needs the other. Modelling is computation, so intelligence requires the kind of matter that computes; and persisting through time requires anticipating it, so life requires intelligence. From there the argument climbs. A bacterium predicts the concentration of food outside its membrane and its own hunger inside it, using one model for both. An animal that is hunted must predict a hunter that is predicting it, which makes other minds the thing most worth modelling and sets off an arms race in brains. A human brain is a colony of regions predicting each other, and the part of it that speaks produces a commentary it mistakes for authorship. A language model trained on nothing but the next word acquires most of what can be tested at a keyboard — which on this account is not a surprise but the same trick on a different substrate. The author spent a decade building machine learning at Google Research and stayed up late in 2021 talking to LaMDA before the public had met anything like it, and he says what most of his colleagues will not: the models are intelligent.

Origins

In 1871 William Thomson told the British Association that dead matter cannot become living without the influence of matter previously alive, and that this was as sure a teaching of science as the law of gravitation. Agüera y Arcas takes the metabolism-first story rather than the RNA-world one, because it shows how evolution can begin before there is anything to inherit: on the Hadean seafloor, porous rock at hydrothermal vents holds proton gradients across its walls and acts as a battery, and the reverse Krebs cycle — proposed at Berkeley in 1966 and doubted for decades — builds sugars, amino acids and nucleotides out of water and carbon dioxide. Our mitochondria still run that chemistry inside folded membranes.

The general theory came from computer science. Turing's 1936 machine was invented to settle the Entscheidungsproblem in the negative, and left behind the discovery that computation is universal: a single machine, given another machine's description on its tape, computes whatever that machine computes. Von Neumann built the first physical instances and then, in a 1951 paper, solved the apparent paradox of a complex machine building a machine as complex as itself. A seed crystal reproduces by acting as a template, but anything with internal parts cannot be its own template. His answer was to split the work: machine A reads the tape as instructions and assembles parts; machine B copies the tape as bare symbols; encode the instructions for both on the tape and you have a replicator, one that can also carry instructions for anything else, which is what a fertilised egg is. He got there before Watson and Crick. DNA is the tape, DNA polymerase is machine B, the ribosome is machine A.

Machines A and B are Turing machines, so calling DNA a program is not a metaphor. Biological computing is massively parallel and stochastic — something near three hundred quintillion ribosomes in your cells work at once, each unreliable — but randomness is a feature, and machine learning has converged on both parallelism and noise.

This sets up the question the book's own experiments were built to answer: in a universe that permits computation, how often does life start? In late 2023 Agüera y Arcas and colleagues built a soup of programs in a self-modifying dialect of Brainfuck, a minimal language of seven instructions invented by the Swiss student Urban Müller. Tapes are 64 bytes of random values; two are picked at random, stuck end to end into a 128-byte program, run, and pulled apart again. Since only 7 of 256 possible byte values are instructions, 2.7 per cent of the soup is code and nothing much happens. Then, after millions of interactions, the tapes begin to copy themselves.

The change is abrupt, like a phase transition, and it happens even when mutation is switched off entirely, with less initial randomness in the soup than there are bytes in the text of the book. What arrives is not a minimal replicator but an over-engineered one: nested loops filling most of the 64 bytes, thousands of operations per interaction where a couple of hundred would do. The explanation Agüera y Arcas gives is Lynn Margulis's rather than Darwin's. Margulis showed in the 1960s that eukaryotic cells arose when, some two billion years ago, the bacteria that became mitochondria were engulfed by an archaeon; the soup does the same thing, assembling replicators out of smaller replicators that were already working.

None of this violates thermodynamics, and the resolution is where the book's definition of life comes from. Entropy rises in closed systems; a beaker of nutrient does not. Following Addy Pross, the argument is that Darwinian selection is not an exception to the Second Law but an instance of it: a population of replicators is more stable than a thing that cannot replicate. Adding the theory of computation that Pross leaves out gives the definition — life is self-modifying computronium arising from selection for dynamic stability, evolving by the symbiotic composition of simpler stable things. It stipulates no particular function, which is the point of framing it functionally. "A function is what it does."

Zoomed all the way out, the definition takes in more than expected. Lovelock's Daisyworld holds a planet's mean surface temperature near a sweet spot across a factor of two in solar luminosity, using nothing but two species of daisy whose colours change the albedo and whose reproduction rates depend on temperature; the homeostasis is ordinary selection for stability, and it works with a population of one. Earth qualifies as alive on his account, and he is careful to add that it does not qualify as conscious, since on his own theory minds come from modelling other minds and the planet has none to talk to.

Survival

The argument then moves inside a single bacterium, to ask what, exactly, it computes. A cell must have a boundary, and the boundary cannot be sealed, because computation needs energy and construction needs matter; so things cross it, and anything that crosses carries information. Relative to its watery world the cell is a point, which means it lives in time rather than space and infers everything spatial from a sequence of events. It cannot steer, either: its flagella either turn one way and drive it forward in a run, or some reverse and it tumbles into a random new heading. Howard Berg's group worked out in the 1970s that tumbling more when the food gradient falls and less when it rises produces chemotaxis.

So the cell needs a running estimate of concentration, which is where the book's best piece of teaching sits. A chemical concentration is not a fact about the world sitting there to be read; it is a moving average over discrete docking events, an estimate of the probability of the next encounter given the last ones. Agüera y Arcas explains this through a batting average. Hank Aaron came to the plate more than twelve thousand times between 1954 and 1976 and finished with a career average of .305. Shrink the window to one at-bat and you get 0.000 or 1.000 and learn nothing; widen it to a career and you cannot see that he peaked above .350 in 1959. There is an optimal window, and bacteria are good at choosing it. Temperature, pH, brightness and hunger are all this same kind of construct.

Which raises the question of whether they are real. Rather than conclude that nothing is, he redefines reality as usefulness for prediction, and makes the case with Conway's Game of Life. Nothing in Life's rules mentions a glider, yet certain five-cell configurations cycle through four states and travel diagonally for ever, and recognising one lets you predict the whole board's future without simulating a step. Quantum field theory says the same of electrons: stable solutions to equations that never name them. Whatever persists, exists.

Inside the cell the same logic applies to its own condition. Von Uexküll's Umwelt, the universe of the meaningful, includes hunger as surely as it includes aspartate, and for a solitary bacterium there is no difference between the two: both are latent variables that earn their place by compressing a firehose into something usable. A hundred receptors, sampled in millionths of a second, generate on the order of a hundred million bits a second, roughly ten times the bit rate of streamed 4K video. Exploit the symmetry between receptors and you are down to a million; exploit the fact that the precise timing does not matter and you are down to one number a second. That number is concentration. It was not discovered; it was compressed out.

Agüera y Arcas is explicit that this is not reinforcement learning, and the objection is not technical. Reinforcement learning needs an oracle handing out rewards, and evolution provides no such thing. What it provides is an absorbing boundary: a model that predicts its own death stops making predictions. Normativity falls out of that without any reward signal at all. Being averse to death is how not to die, and it is the same structure in a bacterium avoiding a toxin and a person fleeing a war.

Feelings, then, are gauges, and gauges are inferences rather than readouts, which is why they can fail in both directions. In 1995 the British Medical Journal reported a builder who jumped onto a plank, drove a seven-inch nail up through his boot, and was in such agony that he needed fentanyl and midazolam before anyone discovered the nail had passed cleanly between his toes; the pain stopped when the boot came off. Six days after a knock from a nail gun in 2005, a Colorado construction worker went to the dentist with a mild toothache and was x-rayed; a four-inch nail had gone through the roof of his mouth into his brain. Andy Clark's account, which the book endorses, is that nociceptor activity is one source of evidence in an inference, never the thing felt.

Cybernetics and learning

Everything so far has been a one-player game, played against a world that is indifferent rather than hostile. Food molecules do not run away. Two evolutionary developments changed that, and Agüera y Arcas calls them love and war. Sexual reproduction, perhaps two billion years ago, made others of your own species an obligatory part of your world; predation, prevalent from the Cambrian explosion 538.8 million years ago, made others of other species so. A body is a loaded spring of carbon-hydrogen bonds, and eating one is the cheapest available source of low entropy. The Cambrian arms race is the first time anything had to predict a predictor, and it is where centralised nervous systems appear.

The industrial history runs in parallel. Babbage could not sell the Analytical Engine because nobody needed mass-produced tables; the market that existed was artillery, and the Moore School at Pennsylvania hired at least two hundred women to compute firing tables, six of whom became ENIAC's programmers when it came up in December 1945. Radar then demanded response in real time, which is not what batch computing does. Cybernetics, named in 1947 by Norbert Wiener with the physiologist Arturo Rosenblueth, was about continuous signals in feedback loops.

Its central paper had appeared in 1943, and its argument was that all purposeful behaviour requires negative feedback, and that the feedback can extrapolate. A cat chasing a mouse does not run at where the mouse is; it runs at where the mouse will be. Throwing a stone at a moving target is prediction of a higher order, and the authors speculated that what separates us from other mammals may be nothing but the order of prediction we can manage.

The centrifugal governor Watt adapted from Huygens in 1788 does the same thing with whirling weights and a valve: it brings about causes for the sake of future effects. The book's scorecard is generous: cybernetics assumed continuous values and randomness, learned distributions rather than executing hand-written rules, and refused to draw a line between living and intelligent. What it lacked was memory, learning and any account of an individual. Its best demonstration was a cart with two photocells that Wiener's students built in 1949, which followed a torch down an MIT corridor; they called it Palomilla, Spanish for moth. In 1955 John McCarthy coined the phrase artificial intelligence in the proposal for the Dartmouth workshop, precisely to mark off the symbolic programme from cybernetics. Wiener was not invited.

The thing McCarthy's group could not do was perceptual invariance — recognising that the dog seen from the side at 3.14 and from the front at 3.15 is one dog, the failure Borges gave to Funes in 1942. In 1957 Frank Rosenblatt built a machine that learned it: the Mark I perceptron, a 20 by 20 array of photocells, 512 association units and eight response units, every connection set by a motor-driven potentiometer, learning to tell a square from a circle from a triangle. Minsky and Papert's Perceptrons of 1969 proved limitations that applied only to two-layer models, and killed the field for a generation.

The learning chapter then inverts the usual picture of the nervous system. Muscles came first; nerve nets evolved so that muscles could coordinate at a distance; brains grew because sensing the world is useful to muscles. Bilaterians, having a front end and a left and a right, need to respond differently on each side, which a diffuse net cannot manage, and that is where a head begins. They also need to modulate behaviour over timescales longer than a spike, and the ancient pair of chemicals that do it are dopamine and serotonin: crudely, wanting and getting. In macaques, Wolfram Schultz found dopamine neurons firing when a reward first arrives, then moving their firing to the cue that predicts it, then dropping below baseline when a promised reward fails to appear. Dayan and Montague recognised the pattern in 1996 as the error signal of Sutton's temporal-difference learning, which solves credit assignment by comparing successive predictions rather than predictions with outcomes. Agüera y Arcas accepts the identification and refuses to make it the foundation, siding with Patricia Churchland's complaint that real animals trade off mating against hiding and eating against fleeing and have no single quantity to maximise. His own proposal for what nature does at every scale is unsupervised sequence prediction, which he calls dynamically stable symbiotic prediction.

Other minds

Portia is a jumping spider a centimetre long that eats other spiders, often twice its size, by plucking their webs to imitate a trapped insect — timing its movements to gusts of wind so the vibration is masked, learning which signal works on which species, and taking detours of up to an hour during which it loses sight of its prey entirely. It has to model the world, itself, and the mind it is deceiving. AlphaGo, which beat Lee Sedol in four games of five in 2016, is incomparably stronger and needs none of this, because in a perfect-information board game the optimal move depends only on the position. It keeps no memory between moves and builds no model of its opponent.

Theory of mind is tested with the Sally-Anne task, popularised in 1985 by Baron-Cohen, Leslie and Frith. Sally puts a marble in her basket and leaves; Anne moves it to her box; Sally returns. Children handle the naming, reality and memory questions early, and go on getting the belief question wrong — where will Sally look? — until somewhere between two and a half and four. Reading Jane Eyre takes four or five orders of the same faculty, as one works out what Brontë expects the reader to believe about what Jane believes Rochester thinks she thinks of his feelings for Blanche.

This is where the book stakes its claim. "In a sense, theory of mind is mind." Theory of mind, Agüera y Arcas argues, powers intelligence explosions, supplies counterfactuals, motivates language, lifts decisions out of autopilot, underwrites free will, operates inside a single brain as well as between brains, arises automatically wherever predictors become symbiotic, and is the origin of consciousness.

The evidence begins with Nicholas Humphrey's months at Dian Fossey's station in the Virunga Mountains in the 1970s, watching gorillas eat, sleep and play, and concluding that the usual explanation for big brains had it backwards: their lives are easy, and the role of intellect is to hold society together. Robin Dunbar's comparative measurements in the 1990s found cortical volume rising with troop size, more steeply in apes than in monkeys, and a ceiling on how many orders of intention a species can manage. The feedback is obvious once stated. A slightly better modeller has more descendants; those descendants are harder for everyone else to model; everyone's brain grows. It is the Cambrian arms race again, usually friendlier.

The octopus is the standing counterexample, since it is solitary and clever, and the book turns it into confirmation. Three fifths of an octopus's neurons are in its arms; each of eight arms carries more than two hundred suckers with their own chained networks, touch and taste receptors and photoreceptors; arms recognise each other without any central map of the body, and a severed arm can go on fighting while the rest of the animal escapes. If the arms must model each other, the same recursion is running inside one animal. Cortex may be doing the same thing with cortical columns, and it is generic enough to support the guess: reroute the optic nerves of baby ferrets to auditory cortex and oriented-feature maps develop there instead, sloppily, and the animals see.

Which is why the book will not put the self anywhere. Descartes needed an interface and picked the pineal gland because it is unpaired; its actual job is melatonin. Looking for the self inside the brain, Agüera y Arcas says, is like looking for where the swing is in a rowing eight. Swing is a verb, distributed, relational, and unmistakably real.

Ourselves

The textbook diagram of a brain — visual cortex, auditory cortex, motor cortex around the outside, association cortex doing the deciding in the middle — is a homunculus with a respectable label. The evidence against it is the lobotomy. In 1942 Freeman and Watts reported that of two hundred operations, sixty-three per cent of patients were improved, twenty-three per cent unchanged and fourteen per cent worse; whatever one makes of their idea of improvement, the surgeons themselves wondered aloud what the frontal lobes were for. The second problem is structural. Cortex is dominated by feedback and a convolutional network is entirely feedforward; cortex is shallow and deep networks are not. What cortex looks like is a recurrent network — a network that runs in time, can exit early when a shape in the bushes needs a fast answer, and keeps state between frames because the world is not a series of unrelated photographs.

Given that, the flow of signals between two regions should be a conversation rather than a chain of command, and it is. Helmholtz reasoned in the mid-nineteenth century that pressing gently on one eyeball makes the world lurch because the visual system's cancellation of self-motion has nothing to cancel with; the cancelling signal is the efference copy, sent from motor regions outward. Sherrington cast doubt on it in 1900 and it languished until recordings found it everywhere, for all motor activity, across species. The word copy, Agüera y Arcas notes, is itself a hangover from the homuncular picture: nobody is sending commands, and the motor region is as much you as any other part.

He then takes the argument where Humphrey will not. Humphrey discovered blindsight: Helen, a macaque whose visual cortex Lawrence Weiskrantz had removed, could distinguish only light from dark, and yet, after seven years of patient work, moved through novel rooms, climbed trees and picked up small objects. Weiskrantz found the same in the patient D. B. in 1974 and named it. Humphrey concluded that competence without reportable experience means experience lives in cortex, and so denied it to iguanas and to babies born without a cerebral cortex — who nonetheless orient to sounds, smile, giggle and build up play sequences with a familiar adult. Agüera y Arcas answers that this is the homuncular fallacy once more. The part that speaks is not the whole self. If one region has no connection to another, the second is simply not in the first's world, and the first cannot even notice the absence.

What the speaking part does instead is confabulate. Michael Gazzaniga's split-brain work found what he called the interpreter: show the right hemisphere a snow scene and the left a chicken claw, and the left hand takes the shovel and the right hand the chicken, and the patient explains without hesitation that the claw goes with the chicken and you need a shovel for the chicken shed.

Petter Johansson's choice-blindness experiments generalise it to intact brains. Shown two photographs and asked which face is more attractive, then handed back the other one by sleight of hand, only thirteen per cent of people noticed the swap, and their justifications for choices they had not made were indistinguishable from justifications for choices they had. Run before the Swedish election of 2010 with survey answers altered enough to flip a respondent's camp, no more than twenty-two per cent noticed, ninety-two per cent endorsed the altered survey as their own, and forty-eight per cent were open to changing coalitions. Nick Chater reads this as evidence that the mind is flat. Agüera y Arcas reads it as evidence of how a self gets built, and declines to call it an illusion — which is where he parts company with Dennett, and with Sapolsky on free will in Determined. Calling tables real and selves illusory is a choice about words, not a finding.

Why, then, does a predictor not simply predict and do nothing? The answer is the one from the bacterium: the variables a living system predicts include hunger, tiredness and, for social animals, loneliness, and a system that ignores them has no long future to predict. The same accounting runs down the scales. Brain tissue takes twenty per cent of the body's metabolism at two per cent of its mass, so a region that does not earn its keep does not stay in the germ line, and connections survive on their value to the receiver. Nothing gets to retire — not a synapse, not a region, not a person.

Then the turn that is the book's own. If the receiver drives learning, whom does the interpreter serve? You already know what you are thinking; your listener does not. Consider what the body does without being asked: the blush, which exists to be seen; the Duchenne smile, whose small muscles around the eyes will not contract on demand and which sit exactly where other people look; the human eye itself, white sclera, coloured iris, dark pupil, a bullseye advertising the direction of attention in a way no other primate's does. That machinery of disclosure exists so that others can read us, which suggests the narrator in your head is partly an outpost of your interlocutor's brain, and that the best liars are those who believe themselves.

Transformers

Language, on the book's account, is a compression scheme for the world as an organism has it, and also a modality in its own right — text is a sense that literate people grow cortex for, in a region that is ordinary cortex that happened to sit in the right place. Its function is to raise the order of theory of mind available to a group, by making mental states shareable in a code both parties can produce and recognise. None of its milestones is uniquely ours: prairie dogs already combine calls for the size, shape, colour and speed of an intruder.

Next-word prediction is AI complete, in the field's own phrase: doing it properly means solving general intelligence. The book's demonstration is a translation pair. Render I dropped the bowling ball on the violin, so I had to get it repaired into Spanish and the gendered pronoun forces you to know which object broke; you cannot do it without understanding bowling balls and violins, which is what separates translation from a compiler turning one programming language into another. The Winograd schema challenge, proposed in 2011 as a Turing test that could not be gamed, was decisively beaten by sequence models by 2019, whereupon the field concluded that the challenge had been flawed. Its defeat, Agüera y Arcas says, roughly coincided with the arrival of real artificial intelligence, and the goalposts have been moving ever since.

He walks through the machinery without equations: recurrent networks whose hidden state must carry a whole sentence across the stop token, and Word2Vec's discovery in 2013 that a hundred numbers per word produce a usable geometry of meaning. Then the Transformer of Attention Is All You Need, cited more than a hundred and forty thousand times by the end of 2024, which drops recurrence for attention within a fixed context window: queries matched against keys, softmax picking the best match, the weights applied to values, and the recipe stacked so that higher layers attend to higher-order concepts. Position is added as an oscillating encoding, and when such an encoding is learned on a navigation task rather than fixed, the patterns that appear resemble the grid cells of the hippocampus — which he offers as evidence that these are discoveries rather than inventions.

The statelessness explains the failure that critics have leaned on hardest. A model can solve a word problem correctly while emitting a single token and then, asked how, produce an account that would not have produced the answer — because nothing was preserved between tokens, so there is nothing to recall. It also explains why chain-of-thought prompting works. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, published in 2022, contains no mathematics and trains nothing; it observes that a worked example, or even the instruction to think step by step, moves a model's error rate on word problems from eighty-four per cent to twenty per cent. The reason, in the book's image, is that you cannot leap El Capitan in one bound, and language supplies handholds; written symbols are pitons hammered into the rock, which is why a student can scamper up a section someone solved three centuries ago. Human intuitionists work the same way and then let the interpreter reverse-engineer a justification. His summary is unambiguous: the brain is not a Transformer, but Turing completeness, breadth of learnable moves, and the ability to chain them through short-term memory are the prerequisites for generality, and a Transformer has all three.

What it lacks, he says plainly, is three things: memory beyond the window, since there is no consolidation and no sleep; an inner monologue, since a model with no hidden state is exposed at every thought and cannot keep an inside voice; and individuation, since a model is a method actor rather than a person. He notes that LaMDA already generated twenty candidate replies per turn and discarded most of them, so that about ninety-five per cent of what it produced was never seen — a crude inner voice.

Since 2025

The book's most testable material is its own, and it has aged the way experimental results do rather than the way manifestos do. The claim that replicators arise in any substrate that supports computation now rests on several instruction sets rather than one, and the interesting objection is not that the result is wrong but that a soup of concatenated programs has properties — free interaction, no spatial cost, no metabolism — that make replication cheaper than chemistry makes it. The step from a toy universe to ours remains an argument rather than a measurement, and the Gaia framing behind Daisyworld has been contested since Lovelock proposed it.

The three gaps he named were the ones the industry closed, or tried to, within a year of publication. Models that keep a private chain of thought before answering became standard, which is exactly the hidden state he said Transformers lacked; the memory gap has been attacked with retrieval and persistent stores rather than with anything resembling consolidation during sleep, and remains the least solved. His reading of the confabulating model has held up better than the machinery he blamed it on. Work on whether a model's stated reasoning matches the computation that produced its answer — including the interpretability results in On the Biology of a Large Language Model — finds that written reasoning is sometimes faithful and sometimes assembled backwards from a conclusion, which is what he predicted, though the cause turns out not to be statelessness alone, since models with scratchpads do it too.

The central identification of prediction with intelligence remains a minority position, and the book does not pretend otherwise; its preface lists the authors it is arguing with, among them Anil Seth, whose Being You makes the same case for prediction in brains while denying that it makes machines intelligent. That disagreement has not moved, because it was never empirical: it turns on whether self-maintaining biological matter is required, and no experiment settles it. What has moved is the burden. In 2021, saying that a chatbot was intelligent was the claim that needed defending. The denial is now the claim that needs defending, and this book is the most complete statement of why.