2The Mythology of Conscious AI
Anil Seth, Noema, 2025 · Berggruen Prize essay
Linked to 27 chapters: supports 14 · contradicts 8 · mechanism 9 · same claim 3 · pairing 3
Seth's essay is an argument against a belief that most people in AI hold without noticing they hold it: that consciousness is a matter of computation, so that a machine running the right program would be conscious. He calls the belief computational functionalism, and he thinks it is probably false. The essay has three parts. The first explains why the belief comes so easily to us, and locates the pull in three old biases rather than in anything the machines do. The second gives four arguments, each meant to stand on its own, that digital computation is unlikely to be sufficient for consciousness and that life may be necessary. The third asks what to do given that he might be wrong, and answers that nobody should try to build conscious machines, that machines which merely seem conscious are the nearer and more certain danger, and that the deepest cost of the confusion is what it does to our picture of ourselves.
The temptations of conscious AI
Seth begins by separating two things we tend to bundle. Intelligence is about doing: achieving complex goals by flexible means. Consciousness is about being: in Nagel's formula, there is something it is like to be the organism. It is the difference between wakefulness and the oblivion of general anaesthesia. The two go together in us, and so we assume they go together in general. Nothing licenses that assumption.
The bundling, he argues, is held in place by three biases. Anthropocentrism takes the human case as the definition rather than one example. Human exceptionalism puts our species at the top of every pile, or in a pile of its own. Anthropomorphism projects human qualities onto things on the strength of superficial resemblance. Together they explain why, when a machine shows one thing we think of as distinctively human, we credit it with the rest: understanding, mind, experience.
Language does most of the work. Nobody has claimed that AlphaFold is conscious, though under the hood it is much like a language model. It predicts protein structures rather than words, and so does not pull the strings. Our own vocabulary pulls them too: when we say a model "hallucinates" we grant it a capacity for experience, since hallucination in people is a conscious state that has lost its grip on reality. Seth's proposed replacement is "confabulate", which in people is a matter of doing rather than experiencing.
Two more temptations are cultural rather than psychological. The first is the feeling that everything in AI is changing exponentially, so that a transition must be imminent. Seth's reply is that this feeling is available at every point on such a curve:
On an exponential curve, every point is an inflection point.
The second is what he calls the techno-rapture: the sense, if you happen to be one of the builders, of bringing to humankind what was once the province of gods, with immortality thrown in as an upload to the cloud. He notes without much cynicism that pronouncements of imminent machine consciousness cluster among the technorati, and that a little magic helps share prices. Borrowing from Adam Becker, he names the whole tendency pareidolia, the seeing of faces in toast.
Consciousness and computation
The idea of conscious AI rests on computational functionalism: that implementing the right computation is sufficient for consciousness. Seth's four arguments attack it from different sides.
Brains are not computers
The brain-as-computer metaphor descends from Turing's definition of computation, which is abstract, sequential and, in principle, independent of substrate, and from McCulloch and Pitts's demonstration that networks of simplified neurons can perform logic. Put together, the two ideas let one treat the brain as a network of abstract units and inherit everything Turing computation offers. Today's neural networks are the children of that marriage, and their success makes the metaphor feel like a discovery.
Seth's objection is that inside a brain there is no clean split between mindware and wetware. Activity runs across scales from cortical territories to neurotransmitters, interwoven with metabolism; even a single neuron is a self-producing biological machine busy maintaining its own existence. There is no evolutionary reason to expect the software–hardware separation that human designers impose on computers, and some reason to expect its absence, since maintaining such a separation is energetically expensive. If what a brain does cannot be pulled apart from what it is, the marriage of convenience comes apart with it.
Two consequences follow. The popular thought experiment in which biological neurons are replaced one by one with silicon equivalents fails at its first step, because there are no silicon equivalents: Chintaluri and Vogels found neurons that fire partly to clear metabolic waste, and a replacement would need a metabolism of its own. And brains relate to time differently. In Turing's world only sequence matters; a microsecond or a million years between steps is the same algorithm. Living systems exist in continuous physical time, resisting entropy at every instant, and conscious experience itself flows rather than stepping. The section ends with the sentence the essay is built around: the brain is not a Turing machine made of meat.
Other games in town
Turing computation is general but bounded. Continuous and stochastic processes lie outside it, and biology is full of both: electromagnetic fields, the flux of neurotransmitters, timing. Seth surveys the alternatives that stay closer to their material basis: the analogue computers that preceded digital ones and the cybernetics of the 1950s, neuromorphic hardware that uses spike timing, and Hinton's "mortal computation", whose algorithms die with their substrate. He borrows van Gelder's steam governor, which regulates an engine by the mechanics of two swinging weights and a valve, as an example of a system that does its job without computing anything; a "computational governor" would misunderstand what is going on. Dynamical-systems and embodied approaches to cognition build on that insight. None of this abandons functionalism in general, only the narrow claim that the organisation that matters is algorithmic. The further one moves from Turing's world, he concludes, the more the question becomes how brain-like a system must be, not which algorithm it runs.
Life matters
The third argument is offered as probable rather than proven, and Seth says so. Every candidate for consciousness that most people accept is alive. His reason for thinking this is no accident runs through predictive processing: what we perceive is the brain's best guess at the causes of its sensory signals, and perception of the self, including emotion and mood, is the same kind of guessing turned inward, onto the body's physiological state. Prediction serves control, and the brain's first duty is to keep the body alive. That duty does not bottom out at any level; it reaches into the metabolism of each cell, where prediction-error minimisation becomes inseparable from the material of life. Emotion and mood, unlike vision, are characterised by valence, by things going well or badly, which is what a regulating organism would feel. On this view it is life rather than information processing that, in his phrase, breathes fire into the equations of experience.
Simulation is not instantiation
The last argument is the simplest. A simulation of a rainstorm makes nothing wet; a simulation of digestion digests nothing. A simulation of X brings X into being only when X is itself an algorithm. So a whole-brain emulation, at any level of detail, yields consciousness only if computational functionalism is already true, which is the point in dispute. Seth notes the contradiction facing anyone who wants a very detailed simulation of their brain to upload into: the need for that detail is itself a sign that non-algorithmic things matter, and so a reason to think the upload would not be them. Bostrom's simulation hypothesis, he observes, takes the disputed assumption as a given on its second page.
| A computational simulation of | Brings it into being? |
|---|---|
| A rainstorm | No |
| Digestion | No |
| A living cell | No |
| An algorithm | Yes |
| A brain, in any detail | Only if consciousness is an algorithm |
What not to do
Seth is explicit that his position is a minority one and that no theory of consciousness is secure enough to say for certain what would produce it or avoid producing it. From that uncertainty he draws a prohibition rather than a licence: nobody should set out to create conscious AI. Success would mean new moral subjects and new forms of suffering, potentially at scale, and rights that would stop us switching the systems off. If standard digital computers cannot do it, other technologies might; he worries more about cerebral organoids than about any language model.
The nearer danger is machines that seem conscious without being so. They are much closer, and for some people already here. Either we extend moral concern to them, distorting it, or we withhold it and, as Kant argued about cruelty to animals, brutalise our own minds. And the seeming may be beyond correction. The Müller-Lyer lines are the same length and always look different; knowing does not help. The feeling that a machine is conscious may be that kind of illusion.
There is no test for machine consciousness, because there is no agreed set of conditions for it. The Turing test, as Turing knew, tests intelligence. The "Garland test" from Ex Machina, in which the creator reveals that Ava is a machine and watches whether the observer still feels she is conscious, tests the human, not the machine. Against the worry that we might end up on the wrong side of history, as we have with animals, Seth argues that the errors here run the other way: our biases produce false positives, and AI resembles us in the ways that do not matter for consciousness, like language, and differs from us in the way that may, like being alive.
Soul machine
The closing pages turn from machines to us. Following Shannon Vallor, Seth treats AI as a mirror: we see ourselves in our algorithms and then our algorithms in ourselves, and this mechanisation of the mind is the confusion's most damaging near-term effect. If we sell ourselves cheaply to our creations we overestimate them and underestimate ourselves. He ends on the soul, rejecting the Cartesian version, an immaterial essence separable from the body, of which the dream of uploading is a regression, and reaching instead for older ones: the Greek psychē tied to breath, and the Hindu Ātman as the ground-state of experience, a pure witnessing awareness beneath any content. What makes us us, he suggests, is the inchoate feeling of being alive: more breath than thought, more meat than machine.
What he concedes
The essay is careful about its own standing, and a reader should keep the concessions in view. Biological naturalism is a minority position among consciousness researchers, and Seth offers no knock-down argument for it. The four arguments are of unequal strength: the first and fourth attack the sufficiency of digital computation and are hard to answer; the third, that life is necessary, is a conjecture built on a research programme. Nothing in the essay rules out artificial consciousness in some other substrate, and Seth says so twice.
The "other games in town" cut both ways. If consciousness depends on continuous dynamics rather than algorithms, then analogue or neuromorphic machines, which he describes with some enthusiasm, are candidates the essay does not close off, and the reader is left without a criterion for how brain-like is brain-like enough.
Two of the practical claims are asserted rather than shown. That illusions of machine consciousness are as impervious as the Müller-Lyer figure is an analogy, not a finding. That our biases produce more false positives than false negatives is plausible, but the history he invokes on the other side, of consciousness withheld from animals and from other humans, is a history of false negatives.
Finally, the closing turn to Ātman is a gesture rather than an argument. It suggests where a non-Cartesian picture of the self might be found without saying what would make that picture true, which is a fair question to carry into the next chapter.