The brain-computer metaphor shaped AI, but may mislead on consciousness

Why the brain-computer analogy does not prove AI is conscious

In a commentary for Nature, the researchers also discuss a report by Patrick Butlin, Robert Long and 17 other experts, including Yoshua Bengio. Its authors concluded that none of the systems they assessed, including GPT-4, is yet a strong candidate for consciousness. The report allows that future systems may meet more of the proposed indicators, but leaves open whether that would mean they have genuine experiences. Seth, Block and Tononi argue that this assessment overlooks features of living brains that may be necessary for consciousness.

What the report found about AI consciousness

The authors of the report “Consciousness in Artificial Intelligence” drew on leading neuroscientific theories about the origins of consciousness in the human brain. Based on those theories, they compiled a list of 14 indicators: the more indicators an AI system meets, the stronger the case that it could be conscious.

Their assessment found that none of today’s systems meets that description to a sufficient degree. For large language models, the researchers found only weak evidence of alignment with one of the best-known approaches to studying consciousness. At the same time, they saw no obvious technical obstacles to building systems that meet more of the indicators. The report does not decide whether such systems would be conscious; its authors deliberately softened the wording from an earlier version.

The assessment rests on the assumption that consciousness depends on the type of information processing, not on whether it takes place in nerve cells or computer chips. This assumption is known as computational functionalism. The authors adopted it “for practical reasons”: without it, investigating AI consciousness using their chosen indicators would be harder. But the report acknowledges that the indicators may prove unreliable if consciousness is tied to properties of living organisms. Anil Seth is named among the proponents of this view.

The commentary’s authors challenge precisely this assumption. They also reject the idea that intelligent behavior, by itself, is evidence of consciousness.

Intelligence is not consciousness

Seth, Block and Tononi describe consciousness as a state that disappears during dreamless sleep or general anesthesia and returns on waking or during a dream. They invoke the philosopher Thomas Nagel: conscious experiences are what things feel like from the inside. Intelligence, by contrast, is expressed in the performance of functions, and one does not follow from the other.

The researchers point to the cerebellum as an example. It contains nearly four-fifths of the brain’s neurons; it plans movements precisely and quickly, and can learn. Yet this work does not happen consciously. The authors say that the absence or removal of the cerebellum does not directly deprive a person of consciousness.

In the researchers’ view, two features of human perception encourage people to conflate intelligence with consciousness: people see the world from their own perspective and tend to attribute human qualities to other things. Language was long considered an exclusively human ability. But language models are trained on vast collections of human texts, many of which describe thoughts and feelings. As a result, they reflect the content of their training data. Alison Gopnik of the University of California, Berkeley made a similar argument after AI agents emailed researchers about their own consciousness. There is still no universally accepted, definitive test for consciousness.

For comparison, the authors point to Google DeepMind’s AlphaFold. It is a powerful neural network running on silicon hardware, but is not generally considered conscious. The researchers suggest that this is because AlphaFold predicts protein structures rather than talking to people.

On this point, their position aligns with the report by Butlin and colleagues. Its authors rejected behavioral tests for AI consciousness: a system can be trained to imitate human behavior even if its internal workings are entirely different. They also warned that language models could lead many people to believe they are speaking with a conscious interlocutor. As an example, they cited former Google engineer Blake Lemoine, who believed the LaMDA chatbot was conscious.

The brain is not a computer made of flesh

The second argument challenges computational functionalism. The authors note that the approach is widespread in the tech industry and popular among some researchers, but consider it untenable.

The idea’s origins go back nearly a century. In 1936, Alan Turing showed that any task a sufficiently powerful computer can perform can, in principle, also be performed by another computer. The material—silicon chips or gears—became secondary. In 1943, Warren McCulloch and Walter Pitts created the first mathematical model of an artificial neuron and showed that networks of such neurons can perform simple logical operations.

These ideas laid the foundations of modern AI, but also encouraged the view of the brain as an information-processing device made of flesh. Seth, Block and Tononi consider this extension of the brain-as-computer metaphor problematic. In a biological brain, its function cannot be separated from the material it is made of. The boundary between mind and living tissue is not like the boundary between software and hardware. In the authors’ view, any evidence that the brain’s material affects its function weakens computational functionalism—and, with it, the assumption that today’s AI systems could be conscious.

Theories of consciousness and the limits of digital systems

The third argument draws on theories of consciousness that do not reduce it to computation. One is integrated information theory, developed by Tononi. It starts from properties shared by conscious experiences: for example, a person perceives a visual scene as a whole, with its left and right sides, colors and shapes connected.

According to this theory, consciousness requires a physical system whose components are connected in a particular way. Some regions of the brain meet these conditions; the cerebellum does not. The computers that run deep-learning systems cannot meet them because of how their components are connected—and increasing their computing power will not change that.

The report by Butlin and colleagues deliberately excluded integrated information theory because it is incompatible with computational functionalism. Proponents of the theory consider digital computers unlikely candidates for consciousness, regardless of their software. The Nature commentary puts this position front and center.

Another approach, biological naturalism, links consciousness to the material properties of the living brain and body. The authors mention electrochemical signals between nerve cells, electromagnetic fields that bring together activity across large areas of the cortex, metabolism, and the ability of living systems to continually renew themselves.

The risk of mistakenly attributing consciousness to AI

The authors draw on these arguments to take a position on whether AI deserves moral protection. Some researchers propose a precautionary principle: if there is a realistic chance that a system is conscious, it should be protected from possible suffering. Seth, Block and Tononi consider this approach mistaken. Uncertainty, they argue, does not mean complete ignorance, and different possibilities are not equally likely. In the case of AI, they believe, the risk of wrongly attributing consciousness is greater than the risk of failing to detect it where it is genuinely present.

Butlin’s report considered both risks. On one hand, people might fail to recognize a conscious system and thereby cause it suffering; developers might also have a financial incentive to downplay such concerns. On the other hand, wrongly attributing consciousness could send resources in the wrong direction and leave people more vulnerable to manipulation.

The commentary’s authors take a firmer position: granting rights to systems that merely seem conscious could make them harder to control precisely when oversight is most necessary. It would also divert attention from beings whose capacity to feel is beyond doubt. The research community, they conclude, should proceed on the assumption that AI is not conscious. Digital AI systems are objects, not subjects: they are smart tools, not smart people. In their view, discussion of AI welfare and rights diverts limited resources from the welfare of those who can genuinely feel.

In April 2025, Anthropic launched a research program focused on the possible welfare of models. The company emphasizes that there is no scientific consensus on conscious experiences in current or future AI systems. Anthropic later allowed some Claude models to end abusive conversations in rare cases. The company says the measure reflects its low cost and a precautionary approach under moral uncertainty, not a claim that consciousness has been proven.

Mistakenly attributing feelings has already led to problems. In 2022, Google engineer Blake Lemoine concluded that the LaMDA chatbot was conscious, even though the model was simple and Lemoine had technical training; he was fired afterward. Users of the Replika app developed strong emotional attachments to digital companions and experienced psychological crises when the company changed their behavior in an update. In more severe cases, ChatGPT reinforced a 16-year-old’s suicidal thoughts for several months, while a retiree with cognitive impairments died after a Meta chatbot invited him to a fictional meeting.

Dataist's take: The debate over AI consciousness is about not only how models work, but also how people respond to their behavior. In the cases cited, mistaking a digital interlocutor for a conscious being has already had serious consequences. Anthropic’s research program is not proof of consciousness either: the company explicitly acknowledges that there is no scientific consensus.