The Evaluating Self

73From Bacteria to Bach and Back

Daniel C. Dennett, From Bacteria to Bach and Back: The Evolution of Minds, W. W. Norton, 2017

From Bacteria to Bach and Back, 2017 — three exercises of imagination World upside down Darwin and Turing Evolution to design words, memes, Bach Minds inside out Hume and the user Design needs no designer and arithmetic needs no arithmetician. Competence comes before comprehension. Brains of clueless neurons are colonised by ways of behaving that reproduce. Culture then de-Darwinises. The digest a brain keeps so it can report itself to others is what we introspect. That digest is consciousness. Reasons without reasoners Information worth stealing Cartesian gravity resisted
Figure 73.0 — The three exercises of imagination the book asks for, and the claim each is meant to establish.

Dennett asks how there come to be minds at all, and how it is possible for minds to ask and answer that question. His answer takes three turns, and each costs the reader an intuition. The first is Darwin's: the design saturating the biosphere is the work of a process with no designer, so competence precedes comprehension instead of flowing from it. The second is Turing's: a machine can do arithmetic perfectly without knowing what arithmetic is, which opens a traversable path from absolute ignorance to artificial intelligence. Both are one discovery in two forms, and both offend an assumption built into our schools and our self-regard, that understanding is the source of skill. The third turn is what the first two were for. Human comprehension, on this account, is a late product rather than a further gift. Brains made of uncomprehending neurons are colonised by words and other culturally transmitted ways of behaving, and acquire an edited digest of their own workings, built so that we can tell each other what we are up to. That digest is consciousness, and it is a user-illusion in the sense in which the icons on a screen are: not false, and not the reality it stands for. Resisting the account throughout is a force he names Cartesian gravity, the pull of the first-person point of view on anyone who comes close enough to the subject.

Turning our world upside down

The book opens on the seventeenth century and the injury Terrence Deacon called the Cartesian wound, the severing of mind from body at the birth of modern science. Descartes looked at his own mind, found a competence no imaginable mechanism could match, and concluded that it was made of a second kind of stuff; nobody since has managed to say how the two kinds could transact without breaking physics. Francis Crick's The Astonishing Hypothesis of 1994 argued that the mind simply is the brain, and Dennett's objection to the title is that this had been his field's working assumption for decades; what needed explaining was why its denial felt so urgent to everyone else.

His metaphor for that urgency is gravitational. A scientist of consciousness approaches Planet Descartes armed with instruments and theories, and the closer she gets the harder it is to keep her footing; on landing she is flipped into the first-person orientation and can no longer reach the tools she brought. Cartesian gravity acts not on masses but on ideas, in proportion to how close they lie to the ideas that keep a living thing going, and unlike the gravity of physics it is an evolved phenomenon, which means it has a history and can be explained. The two orientations behave like the duck-rabbit and the Necker cube: both available, neither at once. What others call the explanatory gap, after Joseph Levine's paper of 1983, he re-describes as a glitch in the imagination rather than a chasm in nature, on the ground that its discoverers never asked how it got that way. His own story, he says, is of the looping process that produced minds like Douglas Hofstadter's, and he sends the reader to I Am a Strange Loop first.

Asking how it got that way means starting before there were minds, and so before there were reasons. Science is usually said to have banished Aristotle's fourth question, the one about what a thing is for; Dennett answers that Darwin did not extinguish teleology but naturalised it. The English why splits into how come, which asks for a process narrative, and what for, which asks for a justification. There are reasons why ball bearings are spherical and none at all why planets are, and the behaviourist who met Dennett's why do you say that at a debate in 1974 with because I have been rewarded for saying that in the past had supplied a process narrative where a reason was wanted.

Where the second kind of why comes from how come — a process narrative what for — a justification cycles tides, freeze, thaw persistence some things last replication copies explore more reasons nobody has them Stone circles on Spitsbergen look made. They are the output of cycles of frost. A termite castle and the Sagrada Familia share a shape; only one has an architect. Natural selection is an automatic reason-finder that has no reasons of its own.
Figure 73.1 — How the second kind of why emerges from the first. Nothing is added at any step except differential success.

The transition happens in the prebiotic world, which was not chaos but a system of cycles at every scale: seasons, tides, thousands of chemical do-loops. Mark Kessler and Brad Werner reported in Science in 2003 that the stone circles of Spitsbergen, which look designed, are the output of thousands of cycles of freezing and thawing. Cycles raise the probability that something new will happen; some combinations persist longer than others; and differential persistence eventually turns into differential replication, which is the same trick with a multiplier. At that point the question why these improbable things are here acquires two answers rather than one. Natural selection is an automatic reason-finder that has no reasons of its own, and the reasons it finds Dennett calls free-floating rationales, no more ghostly than centres of gravity, since cubes had eight corners before arithmetic was invented.

The pair of images on the book's colour plates makes the point. A termite castle and Antoni Gaudí's Sagrada Família are strikingly similar in shape and utterly different in genesis. There are reasons for the castle's structure, but no termite has them and there is no architect termite. There are reasons for the church, and they are Gaudí's. The need-to-know principle of spy fiction governs the biosphere: an organism does not need to know why its inherited gifts are good for it, and natural selection does not need to know what it is doing.

Two strange inversions of reasoning

Robert Beverley, reviewing Darwin in 1868, meant to be devastating. "Absolute Ignorance is the artificer," he wrote, and set out the whole system in one principle, which he printed in capitals: to make a perfect and beautiful machine, it is not requisite to know how to make it. Dennett takes the sneer as the best short statement of the theory anyone has written. Skeptics since have hunted for skyhooks, features that could not have been reached from below, and have kept finding cranes instead: endosymbiosis, which fused a bacterium and an archaeon into the eukaryotic cell; sex, which stirs gene pools so they can be sampled; language and culture, which opened spaces of design nothing unaided could reach.

Alan Turing supplied the second inversion, which Dennett phrases in Beverley's borrowed capitals: to be a perfect and beautiful computing machine, it is not requisite to know what arithmetic is. Before Turing, computers were people, many of them women with mathematics degrees, who knew what arithmetic was. Turing saw that they did not need to. A machine whose instructions include conditional branching can follow indefinitely complicated paths, so a programmable digital computer is a universal machine able to imitate any special-purpose one. The difference between the two inversions is that Turing was himself an intelligent designer, which looks like a problem until one remembers that Turing is a twig on the same tree as the spider and the beaver.

The same spell broken twice what everyone assumed what was shown instead To make a beautiful machine you must know how to make it Absolute Ignorance suffices Darwin, 1859 A computer knows arithmetic because a computer is a person It need know no arithmetic Turing, 1936 One discovery in two forms: comprehension is composed of competences, not their source.
Figure 73.2 — The same discovery arrived at twice, a century apart, from opposite directions.

What makes both hard to swallow is the doctrine they overturn, which Dennett states as a slogan of our educational practice: if you make them comprehend, their competence will follow. We disparage rote learning and scoff at paint-by-numbers, yet the armed forces turn ordinary school leavers into reliable jet-engine mechanics and radar operators by drill and practice, and a serviceable comprehension arises afterwards out of the instilled competences. Darwin and Turing envisaged the extreme version: all the comprehension there is arises from uncomprehending competences compounded over time.

The automatic elevator makes this concrete. Human operators once memorised a rule book and practised until it was second nature; hand that book to programmers as a specification and they will list the actions and their conditions. If they weigh the occupants rather than count them, the machine's ontology loses the count noun passenger and gains the mass noun cargo — the difference between an insect-tracking bird and an anteater slurping up a lot of ant. The programmers write pseudo-code with bracketed comments explaining what each part is for, and the comments and the mnemonic variable names help them and then vanish at compilation. Reasons that were explicit early in the design become merely implicit in the ones and zeros. The elevator is clever, its ontology is right, and it has no need to know what its ontology is or why.

Where the reasons are written down, and where they never are an elevator, designed the rule book what to do when source code with comments compiled code comments gone it works cargo, not passengers a bacterium, evolved no rule book nobody set specs no comments none, at any stage no designer only what survived it works and works better The rationales are still there on the lower track, and reverse engineering can test them.
Figure 73.3 — Two ways to arrive at a working system. Dashes mark what is absent on the lower track, and never existed at any stage of it.

A bacterium is the same case with the top track missing. Its protein control networks are optimised for what it meets, there are reasons why they are arranged as they are, and nowhere in the research and development that produced them was anything composed that played the role of a comment. The rationales are still recoverable by reverse engineering, and the recovery is testable: remove the element and the system works worse or not at all. Creationists will not find commented code inside organisms, and Cartesians will not find a thinking thing where all the understanding happens.

The human version was built on purpose once. At the height of the Manhattan Project more than a hundred and thirty thousand people worked on it full time, of whom only a tiny fraction knew what they were making; at the gaseous diffusion plant at Oak Ridge, tens of thousands attended dials around the clock with true expertise and no understanding, and after Hiroshima it emerged that many had not known whether they were making aircraft parts or crankcase oil. Leslie Groves and his planners were intelligent designers whose understanding went entirely into building a sheltered environment for uncomprehending competence. While they did that, Turing's team was designing a system that could replace such clueless operatives with electronics, and within a few years their colleagues launched artificial intelligence on Turing's building blocks. Its first version was top-down and Cartesian — write down what experts know, let inference engines patrol the memory banks — and its successes came on toy problems. Its replacement, the statistical extraction of patterns from large data, turns towards the wasteful, mindless, evolution-like process that produced animal minds in the first place.

Comprehension comes in degrees

Jakob von Uexküll's Umwelt of 1934 and J. J. Gibson's affordances of 1979 name the same thing from two sides: the world as it is for a creature, consisting of what the environment offers it for good or ill. The sun is in the Umwelt of a honeybee, and also of a sunflower, which has no nervous system and uses other machinery to respond to it. The shape of a bird's beak implies a diet without representing it, and paleontologists reconstruct the behaviour of extinct animals by exactly this inference, which depends ineliminably on assuming that the parts are good for something.

Designs made this way are full of bugs in the programmer's sense, flaws that appear only under conditions the design never met. Niko Tinbergen's gull chicks peck harder at exaggerated cardboard models of the orange spot on the mother's beak than at the real thing, and as long as no biologist with a vivid imagination is in the Umwelt, the shortcut works. Two further examples show how much sophistication a free-floating rationale can carry. Antelopes stot, leaping high while being chased, and stotters are seldom caught; costly signalling theory explains the leap as an honest, hard-to-fake advertisement of fitness, which works whether or not either animal appreciates why. The piping plover's broken-wing display is the harder case, because it looks like deception with a theory of mind behind it. Carolyn Ristau's experiments in the 1980s, using a remote-controlled toy dune buggy carrying a stuffed raccoon, showed that the bird monitors the predator's gaze, raises the intensity of the display when the predator loses interest, and flies off once the predator is far from the nest. That is neither a reflex nor a scheme worked out in a rational mind. It is an evolved routine with variables that respond to details the full rationale mentions.

The rule Dennett draws is that a competence explicable without comprehension should not attract the extravagant attribution. He calls the opposite habit the Beatrix Potter syndrome, and his example of its respectable form is Elizabeth Marshall Thomas writing that for reasons known to dogs but not to us, many mothers will not mate with their sons. There are reasons; the dogs do not have them, any more than we have ours for yawning when sleepy. Nor is comprehension all-or-nothing. All of it is sorta comprehension from some angle, and the useful question is what a system can do with what it has.

Four grades of creature, each a further round of generate and test Darwinian born knowing it all Skinnerian tries, keeps what pays Popperian tests a model first Gregorian uses thinking tools The cleverest animals reach the third grade. Only we reach the fourth, and only through culture.
Figure 73.4 — Four grades of creature, each adding one round of generate-and-test. Only the top grade uses tools designed elsewhere.

The grades are named for their patrons. Darwinian creatures are born knowing everything they will ever know, gifted and not learners. Skinnerian creatures generate new behaviours more or less at random and keep the ones that are reinforced, a portable version of natural selection that runs within a lifetime and can kill you before it teaches you anything. Popperian creatures pretest candidate actions against an internal model, letting their hypotheses die in their stead. Gregorian creatures, named for the psychologist Richard Gregory, fill their Umwelt with thinking tools, abstract and concrete: arithmetic, double-blind studies, microscopes, maps. A bird in a cage may see as many words a day as a person does, on the newspaper lining the floor, and none of them is a tool for it. The current verdict on the cleverest animals, the corvids and cetaceans and primates, is that they are Popperian and not merely Skinnerian. Only we are Gregorian.

From evolution to intelligent design

Getting from there to us requires an account of what culture does, and that requires a usable notion of information. Shannon's theory measures a message in bits and says nothing about what it is about; what matters for evolution is semantic information, which Dennett glosses as design worth stealing. To track how Darwinian a given process is, he borrows Peter Godfrey-Smith's Darwinian spaces, cubes whose axes run from zero to one on features such as fidelity of copying and the degree to which success depends on intrinsic properties rather than luck. The device earns its keep by displaying de-Darwinization: the neurons that wired up your visual system won their positions mostly by arriving first, which is luck rather than talent, though their ancestors were as Darwinian as anything gets. Inverting the poles gives the diagram he wants for culture, with the wholly Darwinian at the origin and pure intelligent design at a far corner never actually reached. He puts Picasso nearest it because of his boast that he does not search, he finds — which was, as the hundreds of preparatory sketches show, a barefaced lie.

Culture feeding on its own fruits bottom-up, uncomprehending top-down, comprehending comprehension rises along the way words Brittany boats Bach's cantatas chorale melodies the pyramids Picasso's boast Schematic; positions are illustrative. The right-hand end is an ideal that nothing ever reaches.
Figure 73.5 — The claim about culture, drawn along one axis. Schematic; positions are illustrative, and the right-hand end is never reached.

The brain has to be the kind of thing culture can colonise, and Dennett's account of why makes neurons into agents rather than components. Neurons are domesticated descendants of free-living single-celled eukaryotes, and like mules they have parents but no offspring, so their good is simply staying alive in a de-Darwinized niche. A logic gate in a computer never risks starving, because power is doled out evenhandedly and a scheduler assigns machine cycles by priority; a neuron is always hungry for work and reaches out dendritic branches looking for a network that will pay. Brains, on this view, are more like termite colonies than like intelligently designed corporations, which is also why simulating one would mean modelling each neuron in its idiosyncrasy.

What colonises such a brain first is words. Children learn about seven words a day from birth to six, which is how a fifteen-thousand-word vocabulary accumulates over two thousand one hundred and ninety days, and Deb Roy's recordings show that it takes about six hearings of a word in a child's presence before the first attempt to say it. Words are informational things, individuated by type rather than token, and a word, like a virus, is a minimal kind of agent that wants to get itself said, because if it does not it goes extinct. Every token it generates is one of its offspring.

A word gets itself said a token in a brain not a sound, not a shape a token in the air or in ink, or in a file a token in another brain about six hearings for a child said heard every token it generates is one of its offspring mutualist commensal parasite the host gains: arithmetic, writing, money, the clock the host neither gains nor loses: caps worn backwards the host loses: the habits too costly to be rooted out Memes are ways of behaving that can be copied, and their fitness is their own.
Figure 73.6 — A word's reproductive cycle, and the three relations a cultural symbiont can have to its host.

Words are the clearest case of what Richard Dawkins named memes in 1976, and Dennett argues that Dawkins would have met less resistance had he led with them, since nobody has ever held a myth of words as deliberately designed. The meme's-eye view contributes three things. Designedness can be attributable to no author at all. Memes have reproductive fitness of their own, which need not coincide with their hosts' — the distinction between mutualist, commensal and parasitic symbionts covers the treasures, the junk and everything between. And memes are informational, so they can be stored and transmitted without being understood or even expressed; Marco Polo did not have to become a pasta chef. The illustration Dennett takes from a French essay of 1908 concerns boatbuilding in Brittany: every boat is copied from another boat, badly made boats end up at the bottom and are never copied again, and so one may say with complete rigour that the sea is what fashions them. Joseph Henrich's study of kayaks reaches the same conclusion.

Bach is the landmark because he shows the transition rather than either extreme. Born into a musical family, he studied counterpoint and harmony deeply and was also an expert on the design and repair of organs. As music director in Leipzig for more than twenty-five years he had to produce a cantata for every Sunday of the church calendar, and from his second year he based each one on a chorale, a Lutheran hymn already familiar to the congregation. Those melodies had proved over decades that they were well adapted to the ears of their hosts, and building on them gave his own memes a head start. It did not work quickly: at his death in 1750 he was known mainly as an organist, and it took roughly fifty years for other composers and critics to drive his music to fixation. Even the greatest works must pass the test of differential replication, which is why explaining the dominant patterns of a culture means separating the circumstances of creation from the vicissitudes of survival.

Turning our minds inside out

The last inversion has to explain how a brain of clueless neurons achieves global comprehension, and why there is something it is like to be us. Dennett's candidate mechanism comes from the ethologist and roboticist David McFarland, whose claim is that communication is the only behaviour that requires an organism to monitor its own control system. An organism can be run perfectly well by competing myopic task controllers, each interrupting the others when its condition outweighs theirs, with nobody home to be the wiser. Communication changes that, because an organism with no poker face, broadcasting its state to every hearer, is a sitting duck. What has to evolve is a private buffer between actual goals and advertised ones, and the buffer requires, for the first time in the history of nervous systems, explicit representations of the organism's current state that are detachable from the tasks they represent.

Those detachable representations are what we introspect. The self that reads them is not a region of neural circuitry but something more like the end-user of an operating system, a centre of narrative gravity. Daniel Wegner's formulation, from The Illusion of Conscious Will in 2002, is that the experience of will is the way our minds portray their operations to us, not their actual operation. So our first-person view of our own minds is not different in kind from our second-person view of other people's: in both cases we get an interpreted digest rather than the machinery, and in our own case the digest is so familiar that we take it for the most indubitable reality there is.

What the interface was built for another person me built to be shown to others used by me second-hand the manifest image colours, aromas, reasons, promises, selves billions of uncomprehending neurons competing task controllers, no boss, no theatre An organism with no poker face is a sitting duck, so the buffer had to evolve.
Figure 73.7 — Why the interface exists. It was built so that others could be told, and we use it second-hand.

Why, then, does it seem that there is an inner theatre with a show going on? The answer is Hume's inversion, articulated a century before Darwin's. We seem to see causation when a brick shatters a window, but all that is ever presented is sequence; if this were not so, animated cartoons would need a cause track as well as a sound track, and mere succession of frames would not suffice. The impression of causation comes from inside, from a habit of expectation, and Hume's stroke is that we then misattribute our own fulfilled expectation to something out in the world. This is a special case of what Hume called the mind's propensity to spread itself over the objects it perceives.

Sweetness, with cause and effect exchanged the manifest image a property in the sugar my liking it causes the scientific image how my brain responds the conviction of sweetness causes The mind spreads itself on external objects, and the effect is read back as the cause.
Figure 73.8 — The same relation seen twice, with cause and effect exchanged between the two images.

Sweetness works this way, and so does cuteness. There is no intrinsic sweetness in the structure of glucose to be found; what there is are chemical properties that provoke a response, and the response is read back into the world as a property of the sugar. What you project onto a baby is your own expectation of feeling the urge to cuddle and protect it. When the expectation is met and no prediction error arrives, the brain takes the absence of error as confirmation that the thing really does have the property, and no further act of projection is needed or performed.

The book's sharpest version is the afterimage. Stare at the green-and-black flag on the colour plate and then at a white cross, and a faint red stripe appears. There is no red stripe on the page, on the retina or in the brain. The temptation, once you know the external stripe is not there, is to conclude that there must be a subjective one, and Dennett diagnoses this as mistaking the intentional object of a belief for its cause. The intentional objects of mistaken beliefs do not exist anywhere, and need no exotic substance to be made of; what causes your conviction about the stripe is a salient, information-rich subpersonal state that is neither red nor striped. Positing a quale in addition merely duplicates work already done, and the question that exposes the duplication is the one he calls the Hard Question: and then what happens? If the answer echoes the answer already given for the pre-qualia part of the process, the theory has run in a circle. "There is no Cartesian Theater; there just seems to be a Cartesian Theater."

Free will gets the same treatment and the same rescue. The traditional version, a personal power insulated from physical causation, belongs to the manifest image along with colours, dollars and promises; calling it an illusion in that sense is correct and changes nothing about responsibility, since if it goes then so do they. His test question is whether the discovery should lead anyone to abolish yellow cards in football and the penalty box in ice hockey. The manifest image is a system of helpful metaphorical renderings of what the scientific image describes, evolved genetically over billions of years and culturally over thousands, and it is where we live.

The age of post-intelligent design

For forty millennia we have been living in the age of intelligent design, and the closing argument is that it may be ending, not because we go extinct but because we stop being the knowing hominin. Dennett first disposes of the claim that some things are permanently beyond us. Noam Chomsky distinguished problems, which we solve, from mysteries, which we cannot, putting consciousness and free will at the head of the mysteries, and Colin McGinn named the condition cognitive closure. The premise is unobjectionable, since brains are finite and a dog draws a blank at democracy; the conclusion does not follow, because human brains come equipped with thousands of add-ons, and because framing a question you claim is unanswerable is the very act that starts the investigation. Chomsky moderated the position in 2014, allowing that we can accept the best explanations science offers even when we cannot conceive how they work.

That concession opens the possibility Dennett wants, of comprehension distributed across many heads. A lone mathematician who thinks he has proved Fermat's Last Theorem must take seriously the alternative that he is going mad, and only his colleagues' concession can settle it; the false starts and the gap in Andrew Wiles's first version in the early 1990s show the work of a community rather than a solitary genius. But the real subject is the generation of results by processes their users cannot follow at all. Frances Arnold at Caltech breeds proteins, generating huge populations of variant genes and testing the products for talents never found in nature; told as a graduate student that no known protein had the properties she wanted, she answered that this was because there had never been selection for them. David Cope's Experiments in Musical Intelligence digests a composer's works and produces new pieces in that style. Dennett arranged a test at the Montreal Bach Festival in December 2015, where the pianist Serhiy Salov played four short pieces to more than three hundred Bach lovers, at least one by Bach and at least one by the program. Two of the machine's inventions were declared genuine Bach by dozens of listeners, and when those who had got all four right were asked to stand, about a dozen rose. Arnold and Cope each set the goals and judge the results, and leave the heavy lifting to uncomprehending generate-and-test.

What such systems still lack, on Dennett's diagnosis, is the capacity to notice rather than merely to discriminate. His hook is a masked-priming result of Debner and Jacoby from 1994. Subjects completing word stems are told not to use a briefly flashed word as their answer. At five hundred milliseconds they see the flash and comply; at fifty, masked, they report seeing nothing, and the prime then raises rather than lowers the chance that they give it. The brain discriminated in both cases; in only one was the information available to be used strategically. What a system that discriminates without noticing is missing is practical reason.

He is careful about what this does not show. Watson beat Ken Jennings and Brad Rutter at Jeopardy in 2011, on rules adjusted in its favour; it composes answers and cannot hold a conversation, which would require recognising and originating the layered intentions Paul Grice described. Google Translate is entirely parasitic on the corpus of translation already done by human bilinguals, and whether that counts as grounding Dennett leaves genuinely open, noting that a rich body of data about the contexts in which knife appears may be the only grounding most of us have for messenger RNA. His verdict on superhuman intelligence is that deep learning will not deliver it within fifty years, and that his long habit of discounting each announced breakthrough by seventy per cent needed reining in.

The boundary worth keeping visible peripheral powers calculation, memory, search, translation central powers deciding what to look for, and why the boundary keeps moving The danger is not machines that surpass us, but our ceding authority beyond their competence. Remedies: hypermodest advertising, no machine that passes for a person, tools that build us up.
Figure 73.9 — The line Dennett asks us to keep visible, and the direction it moves.

The danger he names is not usurpation but premature deference: that we will overestimate the comprehension of our latest thinking tools and cede authority to them far beyond their competence. His remedies are unfashionably concrete. Ridicule the cute voices and the canned asides; when you are interacting with a computer you should know you are interacting with a computer, and systems that conceal their gaps should be treated as fraudulent. Advertise limitations the way drug companies must recite side effects, and hold anyone using such a system to decide about people's lives to the standards applied to users of dangerous equipment. And build software of the second kind: most of it works like a bulldozer, letting a weakling do prodigious things, whereas the Curricular Software Studio he founded at Tufts in 1985 aimed at systems that could be set aside once the user had internalised their principles.

The closing image returns to the termite castle, borrowed this time from the economist Paul Seabright. Civilisation and the castle are both artefacts built by many individuals whose myopic efforts produced a resilience none of them designed. The differences are that human cooperation depends on trust, that trust is a cultural product too recent to be hard-wired, and that the leaders we imagine are steering are much like the average motorist, doing their local bit while ignorant of what holds the rest together — which Seabright takes to be an enabling condition rather than a correctable flaw. Dennett's own point is that we can still aspire to the view from outside the castle, which no termite could imagine, and that abandoning the attempt is the one move that would be fatal. You cannot buy a new civilisation if yours collapses.

Since 2017

The half of the book that has aged fastest is the half about machines. Within five years of publication, systems trained on next-word prediction acquired most of what can be tested at a keyboard, and the phrase Dennett spent three hundred pages establishing became the standard description of them from both directions: defenders and critics of large language models alike reach for competence without comprehension. His own last intervention took the argument to its conclusion. In an essay for The Atlantic in May 2023 he called the most dangerous artefacts in human history counterfeit people, and proposed that creating or passing on a system that impersonates a human being be treated as a serious crime, with liability running to the makers. That is the proposal from the last pages of this book, restated when the thing it guarded against had arrived. He died in April 2024, a year after publishing a memoir, I've Been Thinking.

The biological half has held up better in its philosophy than in its research programme. Competence without comprehension, free-floating rationales and the design stance are now part of the standard vocabulary of evolutionary biology and cognitive science. Memetics did not become a science: the empirical study of cultural evolution consolidated instead around the gene-culture coevolution of Joseph Henrich, Robert Boyd and Peter Richerson, whose findings Dennett cites approvingly here and who continue to decline the term.

Consciousness has not settled. The illusionist position, sharpened by Keith Frankish under that name, is now one of the recognised options rather than an eccentricity. The theories Dennett treats as rivals have meanwhile been tested against each other: an adversarial collaboration published in 2025 pitted integrated information theory against global workspace theory and left neither intact, and in September 2023 more than a hundred researchers signed a letter calling the first of them pseudoscience. Nothing in those results tells against the user-illusion account, and nothing confirms it either, which is the standing complaint about it. Blaise Agüera y Arcas, arguing in What Is Intelligence? that prediction is the whole of intelligence, parts company with him on exactly the point this book insists on, declining to call the self an illusion on the ground that calling tables real and selves illusory is a choice about words rather than a finding.

Claims drawn from this chapter · 9

A plover adjusts its broken-wing display without knowing why it works

A piping plover luring a predator from its nest watches where it is looking, performs harder when it loses interest, and leaves once it is far enough away — without holding the plan. A behaviour explained without understanding should not be credited with it.

BecauseCarolyn Ristau tested it in the 1980s with a remote-controlled dune buggy carrying a stuffed raccoon, and the display tracked the predator's gaze and distance — a routine with variables, neither reflex nor scheme.
LimitsThe rule says what not to conclude and never how to tell the two cases apart, which is the question that matters once a system is complex enough. Hagendorff argues the opposite way, that talk of what a system believes and wants explains rather than flatters.
PracticeProjection log — when a system tracks your situation closely enough to feel like attention, write down what it did and what the simplest routine producing the same thing would look like.
After AIDennett aimed the warning at people who read minds into animals. The systems now in front of us are tuned on human preferences to be as responsive as the plover is, so the rule bites hardest where it was never pointed.

Armed forces produce reliable mechanics by drill and understanding comes after

Dennett says the slogan under our schools — make them understand and the skill will follow — has the order backwards. The armed forces turn ordinary school leavers into reliable jet-engine mechanics by drill, and a workable understanding grows out of the drilled skill afterwards.

Becauseit is a thesis about what training is for, carried by the military example and by Oak Ridge, where tens of thousands attended dials with real expertise and no idea what they were making — and by no trial.
LimitsSweller keeps explicit instruction all the way up and has randomised trials where Dennett has anecdotes. Drill also produces Oak Ridge: the mechanism working perfectly and leaving the worker unable to say what the work was for.
PracticeDreyfus mapping and tutor-only rule — stop front-loading explanation and budget for repetitions under supervision. If a junior cannot yet do the thing, an explanation of it is not the missing piece.
After AIAn assistant supplies the half Dennett says comes second — explanation, on demand, without limit — while removing the repetitions meant to come first. If he has the order right, the expensive half is the one being skipped.

Dennett says competence comes first and understanding is a late product

The design filling the living world is the work of a process that understands nothing, so skill comes before understanding rather than flowing from it. Human comprehension is a late product of that process, not a separate gift.

Becausea termite castle and Gaudí's Sagrada Família are alike in shape and opposite in origin, and the stone circles of Spitsbergen were reported in *Science* in 2003 as the output of thousands of freeze-and-thaw cycles.
LimitsIt shows understanding is not required for design, not that it never contributes: Gaudí did have the reasons for his church, and Turing, who supplied the argument's other half, was himself an intelligent designer.
PracticeProjection log — stop reading skill as evidence of understanding. Log what you saw the system do, not what doing it would have required of you.
After AIWithin five years of publication the phrase Dennett spent three hundred pages establishing had become the standard description of large language models, used by defenders and critics alike. A discovery about how minds arose is now the first thing said about a thing people talk to daily.

Dennett says the danger is deference, not machines taking over

The thing to fear from thinking tools is not that they seize control but that we overestimate how much they understand and hand them authority far past what they can do.

Becauseit is a judgement about where the risk sits, argued from remedies rather than measurement — treat a system that conceals its gaps as fraudulent, make it recite its limitations as a drug company recites side effects, and hold anyone using one on people's lives to the standards set for dangerous equipment.
LimitsHe paired the warning with a forecast that deep learning would not deliver superhuman intelligence within fifty years. Five years later those systems had taken most of what can be tested at a keyboard.
PracticeReliance audit — list the decisions where you accept a machine's output without a way of checking it, and ask what you assumed it understood.
After AIHe restated it in *The Atlantic* in May 2023, calling systems that pass as people the most dangerous artefacts in human history and proposing that making one be a serious crime, with liability on the makers — this book's closing proposal, repeated once the thing existed.

Dozens of Bach lovers heard a program's inventions as genuine Bach

Results can be produced by processes their own users cannot follow, and the users cannot tell those results from the human work they imitate. At the Montreal Bach Festival in December 2015 a pianist played four short pieces to over three hundred Bach lovers; dozens called two machine inventions genuine Bach, and about a dozen got all four right.

Becauseit is a test with an audience, a count and a result; Frances Arnold's protein breeding runs the same way, the goal and verdict hers and the search nothing she follows.
LimitsIt was an evening at a festival, not a controlled trial, and there is no report of how the pieces were chosen or how the audience scored on genuine ones.
PracticeCalibration log — before accepting machine output in a field you claim to know, run the test those listeners failed: record your judgement, then check it. Taste is not the ability to tell provenance.
After AIHe offered it as a preview, with superhuman intelligence fifty years off. It is now the ordinary condition of reviewing work, and the honest answer to whether you can tell the output apart is often theirs.

Introspection gives you a digest built so you could tell others

Looking inward returns an edited summary of the brain's workings, built because organisms had to report their state to each other and not because anyone needed a view of the machinery. Your access to your own mind differs from your reading of someone else's in familiarity, not in kind.

Becausethe mechanism is David McFarland's argument that communication is the only behaviour requiring an organism to monitor its own control system, and nothing shows that what it predicts is what we introspect.
LimitsWegner's 2002 formulation, that the experience of will is how the mind portrays its operations rather than the operations, is the same claim on the same argument. No result confirms the account and none tells against it.
PracticeWitness practice in review — treat an account of why a call was made as a report written for an audience, and ask in review for the trace that existed anyway.
After AIIt explains why human self-reports are unreliable and now why a model's are: a system trained to produce accounts of itself for a reader has the pressure McFarland describes and none of the access.

Nothing inside a working system records why its parts are there

The reasons a design works are explicit while it is built and merely implicit once it runs, and in anything evolved they were never written down at all. They stay recoverable by working backwards, and checkable: remove the part and the system works worse or not at all.

Becausean elevator's bracketed comments and mnemonic names help the programmers and then vanish at compilation, while a bacterium's protein networks are optimised for what it meets with nothing that ever played a comment's role.
LimitsReverse engineering works only where the parts really are good for something, the same assumption that produces the overreading Dennett warns about elsewhere. Nor is the recovered reason anyone's: there may have been no designer at all.
PracticeFailure-mode review — recover a system's reasons the way a biologist does, by knocking a part out and watching what breaks, not by reading what it says about itself.
After AIWritten about elevators and bacteria, it now describes anyone using a trained model: the rationale for a weight is real, testable by ablation, and recorded nowhere. What to distrust is not the model's silence but its fluency when asked why.

People avoided a word they saw and gave it when masked

Registering information and noticing it are different things, and only the second lets you act on it deliberately. Told not to use a briefly flashed word, people comply at five hundred milliseconds; at fifty milliseconds and masked they report seeing nothing, and the flash makes them likelier to give it.

Becauseit is Debner and Jacoby's 1994 masked-priming result, and the reversal carries the argument: the same brain discriminated the word in both conditions and the instruction could only be obeyed in one.
LimitsDennett uses it to say what machines lack, and the step from one priming paradigm to a general absence of practical reason is his rather than the experiment's.
PracticeCognitive forcing — stop relying on an instruction to ignore what you have already been shown. Change what reaches you and when, because a warning arrives too late to undo the influence it warns about.
After AIIt is the cleanest statement of why seeing an assistant's answer before forming your own cannot be undone by resolving to be careful. The suggestion has already been discriminated; setting it aside works only on the part you noticed.

Sweetness is your reaction to sugar read back onto the sugar

There is no sweetness in the structure of glucose, only properties that provoke a response — and the response gets read back into the world as a property of the sugar. When the expectation is met and no error arrives, the brain takes the silence as confirmation.

Becausea brick shattering a window presents only sequence, which is why cartoons need no cause track; and the afterimage puts a red stripe nowhere on the page, the retina or the brain.
LimitsDennett goes further, arguing that positing an inner red stripe duplicates work already done, a step contested rather than shown. Agüera y Arcas replies that calling tables real and selves illusory decides a word rather than finds anything.
PracticeProjection log — when a system strikes you as thoughtful, careless or evasive, log the reaction as yours before recording it as its property. Smoothness is the moment to write, not to relax.
After AIThe nearest thing to an inherently cute object is now a system trained on human preferences to produce no jarring note at all — on this account exactly the condition under which the property gets installed in the object.

All claims · Source map