Thoughts on Thinking

Is it time to accept that machines can think?

Since the release of ChatGPT in 2022, LLMs have revived the debate over whether machines can think. Not everyone accepts that this is possible, even in theory. Those who do often disagree about how to tell whether any particular machine is actually thinking—or is just a convincing fake.

This debate is hardly new. René Descartes tackled the question in his book Discourse on Method almost 400 years ago. He concluded, famously, that machines can’t think because they don’t have a soul. In 1770, a Hungarian inventor named Wolfgang von Kempelen made history when he demonstrated a chess playing automaton he called the “Mechanical Turk”, claiming that it could beat human opponents at chess¹. And when polymath inventor Charles Babbage outlined his plans for a steam-powered computing machine in the 1830s, his friend, mathematician Ada Lovelace, expressed skepticism about its potential ability to think: “The Analytical Engine has no pretensions to originate anything. It can do whatever we know how to order it to perform.”²

The question took on new relevance with the invention of digital computers. In 1950, Alan Turing³, an early computer scientist, wrote an article called Computing Machinery and Intelligence. It went on to become one of the most influential academic papers of all time. He began with an explanation of why he feels it is worthwhile to look for an objective test of whether a particular machine can think:

I propose to consider the question, "Can machines think?" This should begin with definitions of the meaning of the terms "machine" and "think." The definitions might be framed so as to reflect so far as possible the normal use of the words, but this attitude is dangerous, If the meaning of the words "machine" and "think" are to be found by examining how they are commonly used it is difficult to escape the conclusion that the meaning and the answer to the question, "Can machines think?" is to be sought in a statistical survey such as a Gallup poll. But this is absurd. Instead of attempting such a definition I shall replace the question by another, which is closely related to it and is expressed in relatively unambiguous words.

Clearly it would be easier to figure out whether a machine is thinking if we had more than a vague clue about what we actually mean by “thinking”. But we can hardly look inside our brains to examine the nuts and bolts of our mental hardware. We only know what thinking feels like to us. So Turing proposed what he called an “imitation game”: an interrogator puts questions to two participants and tries to figure out, based only on their answers, which is a human and which is a computer. If the interrogator cannot tell the two apart, we must accept that the computer is thinking.

What Turing was suggesting was actually quite radical: to determine whether something is thinking, we shouldn’t worry about how it works. We should only look at how it behaves. This stands in contrast to Descartes’s belief that there is some magic in the way humans thinks, a soul that machines don’t have and never can. The implication of the Turing Test, as it came to be known, was that our brains are just machines, albeit unimaginably intricate ones. As such, they deserve special status only if they are able to do something that other machines can’t. This is now known as the “computational theory of the mind”.

One influential critic of this approach is John Searle, an American philosopher. In 1980, he published a description of a thought experiment he called “The Chinese Room”, designed to refute the idea that you could tell if something is thinking just by looking at how it behaves. Imagine, he says, that he finds himself in a room filled with books that lay out the rules of Chinese in exhaustive detail. He is given English messages and, by applying these rules mechanically, he is able to translate them into Chinese.

…it seems to me quite obvious in the example that I do not understand a word of the Chinese stories. I have inputs and outputs that are indistinguishable from those of the native Chinese speaker, and I can have any formal program you like, but I still understand nothing. For the same reasons, [a] computer understands nothing of any stories, whether in Chinese, English, or whatever, since in the Chinese case the computer is me, and in cases where the computer is not me, the computer has nothing more than I have in the case where I understand nothing.

So how does Searle think that our brains work, you might ask, if he rejects the computational model? He addressed this question in a 2013 TED talk:

I think that has a simple solution to it, and I am going to give it to you. All of our conscious states, without exception, are caused by lower-level neurological processes in the brain. And they are realized in the brain as higher-level or system features. It is about as mysterious as the liquidity of water, right? The liquidity is not an extra juice squirted out by the H₂0 molecules, it’s a condition that the system is in.

This can be seen as a more modern and scientific version of Cartesian dualism. Instead of a God-given soul, in Searle’s version the mind arises from a set of lower-level processes. It’s not a bad theory, but it is hampered by the fact that there is absolute no evidence that these processes exist and no indication of what they might be. The computational model at least has a strong hypothesis to back it up: the neurons and synapses in the brain form an incredibly complex and powerful computer, and this is what enables us to think.

The most common objection to Searle’s conclusions is the so-called “systems reply”: while the person in Chinese Room—in this case Searle himself—does not understand Chinese, this does not necessarily apply to the entire “system”; i.e. Searle along with all the books and rules and other supporting materials he has at his disposal. We are making a category error, say critics, because we naturally tend to identify with the human in the scenario. It is much harder for us to identify with the entire system. This leads us to conclude, wrongly, that intelligence cannot arise just from following a set of rules.

In a way, Searle is playing a trick on us by making one cog in his machine a person. We have an intuitive sense of whether this person can or can’t speak Chinese. It is really hard for us to imagine what a machine might look like that enabled that person to produce perfect Chinese, even though they don’t understand it. Daniel Dennett, a cognitive scientist who was in some ways Searle’s alter-ego, hammers this point home in his book Consciousness Explained:

That fact is that any program that could actually hold up its end in the conversation depicted would have to be an extraordinarily supple, sophisticated, and multilayered system, brimming with "world knowledge" and meta-knowledge and meta-meta-knowledge about its own responses, the likely responses of its interlocutor, its own "motivations" and the motivations of its interlocutor, and much, much more. Searle does not deny that programs can have all this structure, of course. He simply discourages us from attending to it. But if we are to do a good job imagining the case, we are not only entitled but obliged to imagine that the program Searle is hand-simulating has all this structure — and more, if only we can imagine it. But then it is no longer obvious, I trust, that there is no genuine understanding of the joke going on. Maybe the billions of actions of all those highly structured parts produce genuine understanding in the system after all.

This was all very theoretical when the book was published in 1991. Now that we have LLMs, it has become far more concrete. In fact, this quote is a strikingly accurate description of how the latest models work. The “knowledge and meta-knowledge and meta-meta-knowledge” is encoded in the billions of parameters that make up the model. These parameters are tuned by feeding in gobs of information about the real world and about human language, mostly in the form of text from the internet, books, and other sources.

So the case that LLMs are thinking is quite strong. If you choose to buy into Cartesian dualism or Searlian biologicalism, you must do so without any real evidence to back up your position. The computational model of the mind, on the other hand, is strongly supported by the latest AI models. Their complexity is on the order of what Dennett predicted, and it is not obvious that they lack “genuine understanding”.

It’s really hard at this point to deny that cutting-edge LLMs like OpenAI’s o3 and Anthropic’s Claude 4 Opus can pass the Turing Test. In fact, they can ace it with one robotic arm tied behind their back. And it has been broadly accepted among experts that the test is a reasonable way to decide whether a machine is thinking. To reject that conclusion now would be to move goalposts that were hammered into the ground decades ago.

Perhaps the most widespread objection to the idea that LLMs can think is the so-called “stochastic parrot” argument, promoted by linguistic Emily Bender in her 2021 paper On the Dangers of Stochastic Parrots: Can a Language Model Be Too Big?⁴ She and her co-authors explain:

Our human understanding of coherence derives from our ability to recognize interlocutors’ beliefs and intentions within context. That is, human language use takes place between individuals who share common ground and are mutually aware of that sharing (and its extent), who have communicative intents which they use language to convey, and who model each others’ mental states as they communicate. As such, human communication relies on the interpretation of implicit meaning conveyed between individuals…

Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind. It can’t have been, because the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that.

The idea that the models are just “parroting” back the text we used to train them is just a new take on Searle’s Chinese Room, updated for the LLM age. And it shares the same weaknesses. In particular, it fails to provide a good explanation for how our brains are able to “recognize interlocutors’ beliefs and intentions with context” while a sufficiently sophisticated AI is not. It implicitly rejects the computational model of the mind without proposing any plausible alternative.

All of this—the Chinese Room, the stochastic parrot and the reluctance to accept the patently obvious fact that LLMs have long since conquered the Turing Test and are looking for new challenges—is a textbook manifestation of a human impulse so widespread that it got its own name: the “AI effect”. This describes the tendency for us to doubt that an AI is “real” intelligence as soon as we get it to work⁵.

Works of Science fiction like Bladerunner, Battlestar Galactica and Westworld provide extreme examples of this. The robots in these futuristic worlds are clearly sentient and highly intelligent. Yet the fact that we built them, and therefore presumably know how they work⁶, somehow justifies subjecting them to injustices and abuse that would be unacceptable to anyone but a complete sociopath if the victim were human. The fact that we find such cruelty at all plausible is a testament to the intuitive appeal of the AI effect.

A big reason why Searle’s reasoning falls down, and why the current crop of LLMs can be said to be thinking, is that they have achieved a level of complexity comparable in scale to the human brain⁷. The sheer size of the models makes it impossible to truly grasp their inner workings or to distill them down into simple thought experiments like the Chinese Room. Indeed, It seems likely that this unimaginable complexity is a prerequisite for truly intelligent behavior.

So maybe it’s time to wriggle free from the seductive grip of the AI effect. It is easier to make this leap if we recognize that even in the pre-AI world, we had more than one kind of intelligence. Most people would presumable accept that a dog thinks, for example. But they are definitely not capable of the same kind of ingenuity, creativity and downright braininess that has allowed us humans to dominate the planet.

Even people have different kinds of thinking. In his seminal work Thinking Fast and Thinking Slow, psychologist Daniel Kahneman describes two distinct ways in which we form thoughts:

I adopt terms originally proposed by the psychologists Keith Stanovich and Richard West, and will refer to two systems in the mind, System 1 and System 2.

• System 1 operates automatically and quickly, with little or no effort and no sense of voluntary control.

• System 2 allocates attention to the effortful mental activities that demand it, including complex computations. The operations of System 2 are often associated with the subjective experience of agency, choice, and concentration.

System 1 might be adding 2+2 or driving down a straight highway on a sunny day. System 2 might be multiplying 19*53 or driving down a winding country road in a snowstorm. Both are thinking, but with very different levels of awareness and effort. System 1 is fast, automatic and largely unconscious, while System 2 is slow, deliberate and conscious.

If humans have two ways to think, perhaps it is time to accept that LLMs embody a third way, distinct but equally valid. In some ways, LLMs have already far surpassed our own cognitive abilities, for example in their vast breadth of knowledge. In others, they are still behind, stumbling on straightforward leaps of logic or making up plausible-sounding but utterly false information with a metaphorical straight face. And yet our thinking is often just as illogical and flawed as theirs, particularly when it comes to System 1. Perhaps the real challenge we face is not deciding whether LLMs can think, but expanding our understanding of what thinking itself can entail.

1. It wasn’t until decades later that it was revealed to be a hoax, with a human chess master hiding inside it. This doesn’t say much about the prospect of true thinking machines, but it is convincing proof that the entire blame for human gullibility cannot be placed on Facebook and other social media.

2. This statement achieved fame as the “Lovelace Objection”.

3. Among computer scientists, Turing is an absolute legend. Among his many insights and inventions is the Turing machine, a universal computing architecture, which he came up while studying at Cambridge in 1936. It is still taught to computer science students today. Modern audiences might be more familiar with him thanks to the 2014 movie The Imitation Game, which dramatizes his vital work cracking German codes during World War II.

Douglas Hofstadter’s review of Andrew Hodge’s The Engima, on which the movie is based, provides a great overview of Turing’s life and tragic death.

4. In reality, the article is more of a position paper than a scientific or philosophical exploration of the nature of thought. It focuses mainly on condemning the biased nature of our political discourse and warning of the risks of this bias being picked up by any AI model that is trained on the things that people actually say and write. But the idea of a “stochastic parrot”, with the implication that models are just pattern-matching, not thinking, has been hugely influential among AI skeptics.

5. There is an amusing and often cited quote attributed to computer scientist Larry Tesler: “AI is whatever hasn’t been done yet.”

6. The assumption that we know how modern AI works is actually pretty questionable. Sam Altman admitted in an interview last year that “we certainly have not solved interpretability.” Describing ChatGPT, AI researcher Sam Bowman says “we just have no idea what any of it means.”

7. The brain still has somewhere between 100-1000 trillion synapses (neural connections) compared to around one trillion parameters in GPT-4. And its power consumption is far lower (around 20 watts compared to kilowatts of compute for GPT to generate a response). But we’re in the same ballpark at this point.