When OpenAI finally released GPT-5 last week, expectations were sky high. For years we have been told that LLMs are on an exponential growth trajectory. When ChatGPT was launched in 2022, the underlying model—GPT-3—was already two years old. So when GPT-4 was released a few months later, it was natural to perceive the jump as almost instantaneous1. When reasoning models like o1 appeared in 2024, their superior abilities cemented the view that LLM performance was improving at an accelerating pace.

The evidence, however, says otherwise. There haven’t really been any major advances in LLM performance since the first cohort of reasoning models. Nor is there an obvious vector for achieving exponential progress. Most of the recent innovations have been around LLMs as systems—automatic model selection, better web browsing capabilities, etc.—rather than improvements to the underlying models themselves.

For a while, it seemed like building bigger and bigger models trained on more and more data might be the way forward, but that didn’t work out so well:

Then something funny happened. In December, reports started to appear about how OpenAI’s efforts to create their next big model had stalled. When they finally released the new model in February 2025, they named it GPT-4.5, rather than GPT-5, presumably to play down the hype and expectations. Developers were given access to the model, but only as a “preview”. And in April, just two months after it was released, OpenAI announced they were going remove developer access “in the coming months”.

In other words, the effort to push forward quickly to AGI by cramming more parameters, data and compute into existing LLM architectures had failed.

Yet despite indications to the contrary, the idea that AI is about to achieve escape velocity has been hard to quash. Sam Altman is partly to blame, as he’s zigzagged between teasing the incredible capabilities of yet-to-be-released models and playing them down. Well-known pundits have promoted the idea that AGI is just a few years away. And there is a large and vocal community of AI proponents awaiting the arrival of superintelligence with almost religious fervor.

With expectations this inflated, it was inevitable that the GPT-5 would be a letdown. And it was. According to an op-ed in the Washington Post:

OpenAI’s latest chatbot model, GPT-5, is an improved artificial-intelligence tool: faster, more capable, more accurate. But it’s not the technomagic wand some AI optimists hoped for. The leap to “superintelligence,” the prize behind $400 billion in Big Tech investment this year, now looks later rather than sooner, if even possible.

Some of the criticism boils down to people not liking stuff they’re not used to2. But there is also a palpable sense of disappointment among those who were hoping that GPT-5 would represent a giant step towards AGI:

All of this was predictable. Any hopes pinned on the impending singularity were driven more by motivated reasoning than any solid technical reasons.

It’s true that benchmark scores have improved steadily with each successive model release. At the same time, this may have more to do with AI vendors tuning their models specifically to perform well on popular tests, rather than improvement in the models’ ability to perform real-world tasks. Dean Valentine, CEO of AI cybersecurity startup ZeroPath, described how their experiences using newer models differed from what the benchmarks might suggest:

I have read the studies. I have seen the numbers. Maybe LLMs are becoming more fun to talk to, maybe they're performing better on controlled exams. But I would nevertheless like to submit, based off of internal benchmarks, and my own and colleagues' perceptions using these models, that whatever gains these companies are reporting to the public, they are not reflective of economic usefulness or generality. They are not reflective of my Lived Experience or the Lived Experience of my customers. In terms of being able to perform entirely new tasks, or larger proportions of users' intellectual labor, I don't think they have improved much since August.

So there is ample cause for skepticism about claims that models are achieving “exponential progress”, whether these are based on the results of public benchmarks or just wishful thinking.

While some may have been crestfallen to discover that GPT-5 was not the quantum leap they had been hoping for, from a practical perspective this may not be such a bad thing. After all, we should really be focused right now on producing better AI-powered software solutions that provide real value to real people. From the ZeroPath piece3:

Depending on your perspective, this is good news! Both for me personally, as someone trying to make money leveraging LLM capabilities while they're too stupid to solve the whole problem, and for people worried that a quick transition to an AI-controlled economy would present moral hazards.

As a fellow application developer leveraging AI to reimagine an existing software category4, I entirely share these sentiments. Since the initial release of ChatGPT, I have been convinced that we have rare opportunity to apply a new technology in a way that solves UX and functional challenges that would previously have been intractable. At the same time, I’ve been terrified of being caught in the vice-like grip of rapidly improving AI: on the one hand, forced to use technology that might not yet be up to the task; on the other hand, concerned that future versions might be so smart that they render all our efforts redundant.

Seen in this light, the lack of anything revolutionary in OpenAI’s latest release is a good thing. If we aren’t going to achieve AGI any time soon, we can shift our focus to the task of creating great AI-powered apps using current technology. New LLM versions with new capabilities are to be welcomed, even if they offer only incremental improvement. Some may lament that we are no closer to superintelligence than we were a week ago. To the extent that this serves as a wake-up call for the software industry, however, that might not be such a bad outcome.

Over the past three decades, the World Wide Web has transformed the digital landscape. Oldsters like me still remember the days when software was stored on 3.5” floppy disks and packaged in white cardboard boxes. It usually came with a hefty user manual, without which the user interface was impenetrable. It was designed to run on one specific operating system, and developers were forced to create multiple versions that ran on PC, Mac and—if they were truly gluttons for punishment—Linux.

Then came the web. Suddenly, software was something that appeared magically in your browser when you loaded a website. Languages like HTML and JavaScript were standardized across platforms, so web apps could be written once and used everywhere. If you needed help, you could turn to online forums like Reddit, Quora or Stack Exchange.

The web also transformed business models. Gone were the days when Bill Gates would complain about “hobbyists” stealing his software. The industry has shifted gradually to a pay-as-you-go model, also known as software as a service. E-commerce revenues have skyrocketed, as an increasing share of consumer purchases has moved online. And—as Google and Facebook have discovered—online advertising offers a seemingly inexhaustible source of high-margin revenue:

We’ve seen two major battles between web browser vendors since the release of Netscape Navigator in 1994. Both were initiated by established companies who saw the web as an existential threat to their existing business models.

The First Browser Wars

During the first browser wars, Microsoft feared that Netscape’s rise would encroach on its Windows revenues. At the time, Windows completely dominated the desktop operating system market with over 80% market share. Microsoft had even figured out a clever way to limit the impact of software piracy: instead of selling directly to end users, it charged PC manufacturers to preinstall Windows on their computers1. By the time Netscape was released, Microsoft was essentially levying a hefty tax on every PC sold.

But what if the web took off? Desktop operating systems might become irrelevant if the only thing that mattered was which browser you use. Maybe Microsoft would be forced face its competition on a level playing field, a scary proposition for a company used to charging monopoly rents.

In May 1995, Gates sent out his famous “Internet Tidal Wave” memo, extolling the importance of the internet2:

The Internet is at the forefront of all of this and developments on the Internet over the next several years will set the course of our industry for a long time to come. Perhaps you have already seen memos from me or others here about the importance of the Internet. I have gone through several stages of increasing my views of its importance. Now I assign the Internet the highest level of importance. In this memo I want to make clear that our focus on the Internet is crucial to every part of our business.

He singled out Netscape as a particularly formidable competitor. So Microsoft grabbed some of the best talent from its deep pool of developers and licensed technology from Spyglass Mosaic—at that point an also-ran in the browser wars. With Gates himself decrying the internet as a potentially existential threat3, the team worked literally day and night to develop the first version of Internet Explorer. Eric Sink of Spyglass described the intense work environment in a blog4 post a few years later:

I was asked to be the primary technical contact for Microsoft and their effort to integrate our browser into Windows 95. I went to Redmond and worked there for a couple of weeks as part of the "Chicago" team. It was fun, but weird. They gave me my own office. At dinner time, everyone went to the cafeteria for food and then went back to work. On my first night, I went back to my hotel at 11:30pm. I was one of the first to leave.

Over the next few years, Microsoft quickly caught up with, then obliterated Netscape:

It achieved this mainly by bundling IE with Windows5, taking advantage of its massive installed base. Over time, it worked in proprietary features that forced websites to decide between supporting other browsers or having access to the latest, greatest IE functionality. Many opted for the latter, and by 2001 IE had achieved absolute dominance.

Microsoft wasn’t making or even trying to make money from Internet Explorer. Its efforts to take over the browser space were aimed only at minimizing the threat Netscape represented to its cash cow Windows business. This strategy was stupendously successful. Netscape was sold to AOL in a face-saving deal that was valued at around $10 billion in stock when it closed in 1999. After a few years of benign neglect, the browser was quietly retired.

The Second Browser Wars

Early versions of IE were basically cobbled together from bubble gum, baling wire and Spyglass Mosaic source code. IE4 was arguably the first truly usable version. IE5 was fairly solid, and Microsoft began to use the “embrace, extend and extinguish” strategy that had cemented its dominance in other areas.

New features like XMLHTTP and ActiveX controls6 were added to the browser. At its peak, IE had 80-90% market share, and it was awfully tempting for web developers to use these features even if they broke compatibility with other browsers. This gave users even more reason to adopt IE, and developers even more reason to support its proprietary features. The ensuing cycle was either virtuous or vicious, depending on your perspective.

IE6’s reign of terror had begun. Many websites declared that they were “best viewed in IE6”. Corporate intranets, which often ran only on some version of Internet Explorer, were forcing some users to stick with the beast well into 2010s.

During this period, Mozilla served as the Rebel Alliance to IE’s Galactic Empire. After essentially giving up on the browser market in 1998, Netscape open-sourced its browser, creating the Mozilla project. The assumption was that only the combined efforts of open-source developers around the world could compete with the IE juggernaut.

At first, the effort faltered. Mozilla made what was, in retrospect, a strategic mistake by trying to market a massive all-in-one suite that encompassed a browser, email client, chat and much more. The product felt bloated, and it struggled to gain adoption.

Eventually, a small group inside Mozilla became frustrated with their lack of success in the market. They set out to create a lean, fast, user-friendly standalone browser that would be more appealing than the unwieldy Mozilla Suite. The result was Firefox. Released in 2004, it stripped away the bloat and introduced appealing new features like tabbed browsing and popup blocking.

By then the market was yearning for an alternative to IE, and Firefox was a smash hit. At its peak in 2009 it achieved 32% market share, forcing Microsoft to start innovating again and take standards support more seriously.

This period also coincided with the rise of Google. Founded in 1998, its search engine was a technical marvel that quickly grew a large user base. After implementing the pay-per-click model for AdWords—its search advertising product—in 2002, it was also throwing off huge wads of cash. Those revenues were closely tied to search engine usage; the more people who used Google for search, the more money the company made.

According to a fascinating contemporaneous account7 by Steven Levy in Wired magazine, Larry Page and Sergey Brin8 had been itching to make their own browser from the earliest days of the company:

"The browser matters," CEO Eric Schmidt says. He should know, because he was CTO of Sun Microsystems during the great browser wars of the 1990s. Google cofounders Larry Page and Sergey Brin know it, too. "When I joined Google in 2001, Larry and Sergey immediately said, 'We should build our own browser,'" Schmidt says. "And I said no."

At the time, Schmidt didn’t feel like they were ready for a head-to-head battle with Microsoft. Instead, Google contributed developers like Darin Fisher to Mozilla’s open source development efforts, where they played a key role. Over time, however, they became frustrated with difficulty of trying to improve a browser while maintaining compatibility with its large existing ecosystem9:

The problem with revamping existing browsers to accommodate this concept is that they have developed an ecology of add-on extensions (toolbars, RSS readers, etc.) that would be hopelessly disrupted by a radical upgrade. "As a Firefox developer, you love to innovate, but you're always worried that it means in the next version all the extensions will be broken," Fisher says. "And indeed, that's what happens."

Microsoft’s decision to build a browser was driven by the perceived threat to Windows revenues that Netscape represented. Similarly, Google was motivated by the possibility that Microsoft might use its dominance in the browser market to control how people use the web. And indeed, that influence—albeit unintentional—was already palpable as Google began to create rich web apps like Gmail and Maps that pushed the capabilities of legacy browsers like IE6 to the limit.

If Firefox was a smash hit, Chrome was a triumph. Many of its key features required amazing feats of engineering, including multiprocess sandboxing and the V8 JavaScript engine, which was an order of magnitude faster than the competition. It also had a sleek minimalistic design aesthetic and an innovative “omnibar” that incorporated search directly into the address bar.

Chrome took on new importance with the release of Bing in 2009. Suddenly there was a credible challenger to the cash geyser that was Google’s search monopoly. Chrome moved to a 6-week release cycle, spinning off new versions much faster than other vendors. Instead of relying on organic growth, they paid for distribution by bundling with popular open-source software like Adobe Flash.

With IE development in the doldrums for the better part of the previous decade, Microsoft was unable to withstand the onslaught of a much faster, more attractive and downright better browser. By 2012, just four years after launch, Chrome had established itself as the world’s #1 browser:

The release of Bing also clarified Chrome’s contribution to Google’s bottom line. It paid for itself not by generating cash, but by saving it. Google was already paying Mozilla about $300 million/year to be the default search in Firefox. They had a similar deal with Apple for Safari, and those payments grew rapidly with the iPhone’s launch in 2007. In 2022, Google paid Apple a reported $20 billion just to be the default search provider on iOS10. For Chrome, with its massive market share, they of course paid nothing.

The AI Browser Wars

Chrome has trounced the competition so comprehensively that the browser wars are now a faint memory. Every other browser is basically niche. The only other even vaguely significant player is Safari, which is preinstalled on the Mac, iPhone and iPad. Moreover, even the nichy alternative browsers almost all use Google as their default search provider.

Yet all of a sudden this Pax Browsana is being shaken up. The reason? AI is upending our assumptions about how we access the web and how companies monetize it.

Before the release of ChatGPT, using then web mostly meant you entered a query into Google, got a bunch of blue links, and clicked one of them. Sometimes the blue link was a search ad, in which case Google made money. Occasionally someone used another search engine, and a different company took a cut. But not very often.

This has been the case for the past 25 years or so. But is this really how people want to use the web, or is it just the best we could do at the time with the technology at hand? After all, in the early days of the web, directories like Yahoo reigned supreme, with search engines just an afterthought11. It took a revolutionary innovation like Google search algorithm to upend the status quo.

Could chat apps offer another such innovation? They have already started to replace the browser for some of the things people do online. The days when ChatGPT would warn you about its limited “knowledge cut-off window” are long gone. Instead of relying on the instantly out-of-date information that was used to train the model, it now goes off and browses the web for you, retrieving the freshest data available12.

But why stop there? In addition to answering your questions, surely it makes sense for the AI to perform actions for you as well. Sure enough, last month OpenAI announced “ChatGPT agent”, a new feature that does exactly this. The video below shows it searching for a flight from Prague to LAX. Instead of offering full-fledged browsing experience, ChatGPT paints a kind of virtual browser window inside the chat. It’s slow, clunky, and doesn’t feel quite ready for primetime. But the potential is clear.

Other AI vendors are taking different approaches. Anthropic has a “computer use model” that lets the AI take over your computer13 and control any application, including the web browser. More recently, Microsoft added a chat sidebar to its Edge browser and dubbed it “Copilot Mode”, while Perplexity released an entirely new browser with integrated AI capabilities. Opera is preparing something similar, as is OpenAI14.

All of this suggests that AI is enabling the creation of newer, better browsing experiences. The chat interface popularized by OpenAI, Google, Anthropic and others is just a stepping stone. The logical home for these new AI technologies is a wholesale reimagining of the web browser. There are some use cases where it makes sense to view a traditional webpage15. There are probably more, however, where a chat interface—or some combination of the two—makes more sense.

As with previous browser wars, the current shift is motivated to some degree by a desire to adapt the browser UI to accommodate new technologies. Microsoft built IE in part to make the web feel more like the PC desktop. Google needed a fast, flexible platform to run its new generation of dynamic web apps. Now AI vendors want to offer a modern chat interface for tasks where that makes sense, without forcing you to jump back and force between their app and the browser.

Then there’s the money. Much has been made of the fact that Google’s finances are likely to be heavily impacted as search traffic moves into the chat client. There’s clearly something to this. The majority of revenues for Alphabet—Google’s parent company—come from search. And while the evidence is still anecdotal, there are early indications that AI is already having an effect on user’s search habits:

A new study conducted by Future Publishing with 510 participants from the US and 518 participants from the UK reveals that more than a quarter of Americans (27%) now use AI chatbots like ChatGPT instead of traditional search engines.

Some players are exploring how to generate revenue from inside chat clients. ChatGPT already displays results in a decidedly browser-like way when users ask for product recommendations. They are reportedly experimenting with various schemes for monetizing these results, presumably through affiliate links.

With broad adoption of the Model Context Protocol (MCP)16, which provides a standard interface for chat clients to work with third-party tools, it is easy to imagine users turning to AI to book a holiday rental or flight—with the chat vendor taking a cut of every transaction.

Google is well-placed to fight off its assailants in these new browser wars. It is a both a top contender in the AI space and the undisputed leader in the browser market. At its I/O conference in March, it announced that it was integrating Gemini, its AI technology, into Chrome. The result—rolled out so far to a select group of users in the U.S.—is similar in its approach to Perplexity’s Comet browser.

Google would be well-advised to take the danger seriously, however. As previous browser wars have shown, new technologies open the door to new challengers. After 15 years of unchallenged dominance—and hundreds of billions in high margin search revenues—it will have to start innovating again if it wants to stay ahead.

Since the release of ChatGPT in 2022, LLMs have revived the debate over whether machines can think. Not everyone accepts that this is possible, even in theory. Those who do often disagree about how to tell whether any particular machine is actually thinking—or is just a convincing fake.

This debate is hardly new. René Descartes tackled the question in his book Discourse on Method almost 400 years ago. He concluded, famously, that machines can’t think because they don’t have a soul. In 1770, a Hungarian inventor named Wolfgang von Kempelen made history when he demonstrated a chess playing automaton he called the “Mechanical Turk”, claiming that it could beat human opponents at chess1. And when polymath inventor Charles Babbage outlined his plans for a steam-powered computing machine in the 1830s, his friend, mathematician Ada Lovelace, expressed skepticism about its potential ability to think: “The Analytical Engine has no pretensions to originate anything. It can do whatever we know how to order it to perform.”2

The question took on new relevance with the invention of digital computers. In 1950, Alan Turing3, an early computer scientist, wrote an article called Computing Machinery and Intelligence. It went on to become one of the most influential academic papers of all time. He began with an explanation of why he feels it is worthwhile to look for an objective test of whether a particular machine can think:

I propose to consider the question, "Can machines think?" This should begin with definitions of the meaning of the terms "machine" and "think." The definitions might be framed so as to reflect so far as possible the normal use of the words, but this attitude is dangerous, If the meaning of the words "machine" and "think" are to be found by examining how they are commonly used it is difficult to escape the conclusion that the meaning and the answer to the question, "Can machines think?" is to be sought in a statistical survey such as a Gallup poll. But this is absurd. Instead of attempting such a definition I shall replace the question by another, which is closely related to it and is expressed in relatively unambiguous words.

Clearly it would be easier to figure out whether a machine is thinking if we had more than a vague clue about what we actually mean by “thinking”. But we can hardly look inside our brains to examine the nuts and bolts of our mental hardware. We only know what thinking feels like to us. So Turing proposed what he called an “imitation game”: an interrogator puts questions to two participants and tries to figure out, based only on their answers, which is a human and which is a computer. If the interrogator cannot tell the two apart, we must accept that the computer is thinking.

What Turing was suggesting was actually quite radical: to determine whether something is thinking, we shouldn’t worry about how it works. We should only look at how it behaves. This stands in contrast to Descartes’s belief that there is some magic in the way humans thinks, a soul that machines don’t have and never can. The implication of the Turing Test, as it came to be known, was that our brains are just machines, albeit unimaginably intricate ones. As such, they deserve special status only if they are able to do something that other machines can’t. This is now known as the “computational theory of the mind”.

One influential critic of this approach is John Searle, an American philosopher. In 1980, he published a description of a thought experiment he called “The Chinese Room”, designed to refute the idea that you could tell if something is thinking just by looking at how it behaves. Imagine, he says, that he finds himself in a room filled with books that lay out the rules of Chinese in exhaustive detail. He is given English messages and, by applying these rules mechanically, he is able to translate them into Chinese.

…it seems to me quite obvious in the example that I do not understand a word of the Chinese stories. I have inputs and outputs that are indistinguishable from those of the native Chinese speaker, and I can have any formal program you like, but I still understand nothing. For the same reasons, [a] computer understands nothing of any stories, whether in Chinese, English, or whatever, since in the Chinese case the computer is me, and in cases where the computer is not me, the computer has nothing more than I have in the case where I understand nothing.

So how does Searle think that our brains work, you might ask, if he rejects the computational model? He addressed this question in a 2013 TED talk:

I think that has a simple solution to it, and I am going to give it to you. All of our conscious states, without exception, are caused by lower-level neurological processes in the brain. And they are realized in the brain as higher-level or system features. It is about as mysterious as the liquidity of water, right? The liquidity is not an extra juice squirted out by the H₂0 molecules, it’s a condition that the system is in.

This can be seen as a more modern and scientific version of Cartesian dualism. Instead of a God-given soul, in Searle’s version the mind arises from a set of lower-level processes. It’s not a bad theory, but it is hampered by the fact that there is absolute no evidence that these processes exist and no indication of what they might be. The computational model at least has a strong hypothesis to back it up: the neurons and synapses in the brain form an incredibly complex and powerful computer, and this is what enables us to think.

The most common objection to Searle’s conclusions is the so-called “systems reply”: while the person in Chinese Room—in this case Searle himself—does not understand Chinese, this does not necessarily apply to the entire “system”; i.e. Searle along with all the books and rules and other supporting materials he has at his disposal. We are making a category error, say critics, because we naturally tend to identify with the human in the scenario. It is much harder for us to identify with the entire system. This leads us to conclude, wrongly, that intelligence cannot arise just from following a set of rules.

In a way, Searle is playing a trick on us by making one cog in his machine a person. We have an intuitive sense of whether this person can or can’t speak Chinese. It is really hard for us to imagine what a machine might look like that enabled that person to produce perfect Chinese, even though they don’t understand it. Daniel Dennett, a cognitive scientist who was in some ways Searle’s alter-ego, hammers this point home in his book Consciousness Explained:

That fact is that any program that could actually hold up its end in the conversation depicted would have to be an extraordinarily supple, sophisticated, and multilayered system, brimming with "world knowledge" and meta-knowledge and meta-meta-knowledge about its own responses, the likely responses of its interlocutor, its own "motivations" and the motivations of its interlocutor, and much, much more. Searle does not deny that programs can have all this structure, of course. He simply discourages us from attending to it. But if we are to do a good job imagining the case, we are not only entitled but obliged to imagine that the program Searle is hand-simulating has all this structure — and more, if only we can imagine it. But then it is no longer obvious, I trust, that there is no genuine understanding of the joke going on. Maybe the billions of actions of all those highly structured parts produce genuine understanding in the system after all.

This was all very theoretical when the book was published in 1991. Now that we have LLMs, it has become far more concrete. In fact, this quote is a strikingly accurate description of how the latest models work. The “knowledge and meta-knowledge and meta-meta-knowledge” is encoded in the billions of parameters that make up the model. These parameters are tuned by feeding in gobs of information about the real world and about human language, mostly in the form of text from the internet, books, and other sources.

So the case that LLMs are thinking is quite strong. If you choose to buy into Cartesian dualism or Searlian biologicalism, you must do so without any real evidence to back up your position. The computational model of the mind, on the other hand, is strongly supported by the latest AI models. Their complexity is on the order of what Dennett predicted, and it is not obvious that they lack “genuine understanding”.

It’s really hard at this point to deny that cutting-edge LLMs like OpenAI’s o3 and Anthropic’s Claude 4 Opus can pass the Turing Test. In fact, they can ace it with one robotic arm tied behind their back. And it has been broadly accepted among experts that the test is a reasonable way to decide whether a machine is thinking. To reject that conclusion now would be to move goalposts that were hammered into the ground decades ago.

Perhaps the most widespread objection to the idea that LLMs can think is the so-called “stochastic parrot” argument, promoted by linguistic Emily Bender in her 2021 paper On the Dangers of Stochastic Parrots: Can a Language Model Be Too Big?4 She and her co-authors explain:

Our human understanding of coherence derives from our ability to recognize interlocutors’ beliefs and intentions within context. That is, human language use takes place between individuals who share common ground and are mutually aware of that sharing (and its extent), who have communicative intents which they use language to convey, and who model each others’ mental states as they communicate. As such, human communication relies on the interpretation of implicit meaning conveyed between individuals…

Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind. It can’t have been, because the training data never included sharing thoughts with a listener, nor does the machine have the ability to do that.

The idea that the models are just “parroting” back the text we used to train them is just a new take on Searle’s Chinese Room, updated for the LLM age. And it shares the same weaknesses. In particular, it fails to provide a good explanation for how our brains are able to “recognize interlocutors’ beliefs and intentions with context” while a sufficiently sophisticated AI is not. It implicitly rejects the computational model of the mind without proposing any plausible alternative.

All of this—the Chinese Room, the stochastic parrot and the reluctance to accept the patently obvious fact that LLMs have long since conquered the Turing Test and are looking for new challenges—is a textbook manifestation of a human impulse so widespread that it got its own name: the “AI effect”. This describes the tendency for us to doubt that an AI is “real” intelligence as soon as we get it to work5.

Works of Science fiction like Bladerunner, Battlestar Galactica and Westworld provide extreme examples of this. The robots in these futuristic worlds are clearly sentient and highly intelligent. Yet the fact that we built them, and therefore presumably know how they work6, somehow justifies subjecting them to injustices and abuse that would be unacceptable to anyone but a complete sociopath if the victim were human. The fact that we find such cruelty at all plausible is a testament to the intuitive appeal of the AI effect.

A big reason why Searle’s reasoning falls down, and why the current crop of LLMs can be said to be thinking, is that they have achieved a level of complexity comparable in scale to the human brain7. The sheer size of the models makes it impossible to truly grasp their inner workings or to distill them down into simple thought experiments like the Chinese Room. Indeed, It seems likely that this unimaginable complexity is a prerequisite for truly intelligent behavior.

So maybe it’s time to wriggle free from the seductive grip of the AI effect. It is easier to make this leap if we recognize that even in the pre-AI world, we had more than one kind of intelligence. Most people would presumable accept that a dog thinks, for example. But they are definitely not capable of the same kind of ingenuity, creativity and downright braininess that has allowed us humans to dominate the planet.

Even people have different kinds of thinking. In his seminal work Thinking Fast and Thinking Slow, psychologist Daniel Kahneman describes two distinct ways in which we form thoughts:

I adopt terms originally proposed by the psychologists Keith Stanovich and Richard West, and will refer to two systems in the mind, System 1 and System 2.

• System 1 operates automatically and quickly, with little or no effort and no sense of voluntary control.

• System 2 allocates attention to the effortful mental activities that demand it, including complex computations. The operations of System 2 are often associated with the subjective experience of agency, choice, and concentration.

System 1 might be adding 2+2 or driving down a straight highway on a sunny day. System 2 might be multiplying 19*53 or driving down a winding country road in a snowstorm. Both are thinking, but with very different levels of awareness and effort. System 1 is fast, automatic and largely unconscious, while System 2 is slow, deliberate and conscious.

If humans have two ways to think, perhaps it is time to accept that LLMs embody a third way, distinct but equally valid. In some ways, LLMs have already far surpassed our own cognitive abilities, for example in their vast breadth of knowledge. In others, they are still behind, stumbling on straightforward leaps of logic or making up plausible-sounding but utterly false information with a metaphorical straight face. And yet our thinking is often just as illogical and flawed as theirs, particularly when it comes to System 1. Perhaps the real challenge we face is not deciding whether LLMs can think, but expanding our understanding of what thinking itself can entail.

When ChatGPT vaulted into the public consciousness at the end of 2022, it was obvious to many observers that computers would soon become superintelligent. Seemingly overnight, AI had gone from laughably bad to frighteningly good. If we could make that much progress, that quickly, how long could it possibly take until machines became smarter than humans? The only question was whether they would remain our dutiful servants, or try to kill us before we have a chance to shut them off.

Silly as it may seem now, intellectual heavyweights from Yuval Noah Harari to Steve Wozniak rushed to sign a letter calling for a six-month pause in AI research. The implication was that potentially deadly computer intelligence was so close that it would be risky to wait even six months to see how things developed.

More recently, a group of futurologists led by Daniel Kokotajlo published an essay called AI 2027 laying out a detailed scenario for how superhuman AI could become a reality over the next 2-3 years. In 2021, Kokotajlo had made a bunch of mostly correct predictions about the shape of AI in 2026. So his updated forecast got a lot of press and attention. Spoiler: the AI gets superintelligent really fast, and decides to kill us before we have a chance to shut it off.

Unsurprisingly, people running AI companies are among the most optimistic about the advent of artificial general intelligence, or AGI. Cynics point out that they need to hype their wares to raise the huge sums required to train bigger and better models. A more generous take is that it’s human nature to “believe our own bullshit”. This is particularly true of entrepreneurs, who would never be willing to face the long odds of success if they weren’t exceptionally optimistic, even unrealistic, about their prospects.

And sure enough, Sam Altman, CEO of OpenAI, claimed in a 2024 blog post that “We are now confident we know how to build AGI as we have traditionally understood it.” Elon Musk even assigned it a timeline:

Meanwhile, calmer heads have started to argue that AGI might not be so easy. Maybe it wouldn’t appear for many years or even decades… if ever.

Meta’s Chief AI Scientist, Yann LeCun, is one of a number of prominent AI researchers who makes this point:

Another prominent skeptic is Gary Marcus, who has been adamant that large language models like ChatGPT are not the way to achieve AGI. Marcus has spent the past 25 years arguing that deep learning—which underpins pretty much all modern AI, including LLMs—will never lead to true machine intelligence. In a recent article, he points to research by Apple as proof that LLMs (and more recent “large reasoning models” aka LRMs) aren’t really thinking: they are just slicing and dicing the data they absorbed during training:

…even the latest of these new-fangled “reasoning models” still—even having scaled beyond o1—fail to reason beyond the distribution reliably, on a whole bunch of classic problems, like the Tower of Hanoi. For anyone hoping that “reasoning” or “inference time compute” would get LLMs back on track, and take away the pain of multiple failures at getting pure scaling to yield something worthy of the name GPT-5, this is bad news.

Meanwhile Rodney Brooks, another long-time AI pioneer, has been the most consistent voice pooh-poohing the idea that we’ll achieve AGI any time soon. Compared to him, Marcus is a relative newcomer to the debate; Brooks was already out there pushing back against the AI hype in 1988! Even then, he reminds us, inflated expectations about AI were decades old:

In his 1949 book GIANT BRAINS or Machines That Think, Edmund Berkeley reflected on the amazing capability of machines like ENIAC to perform 500 multiplications of two ten-digit numbers per second, and envisioned machines acting as automatic stenographers, translators, and psychiatrists.

Brooks has gone on to publish a lot of thoughtful analysis about why we always seem to believe that we are just about to achieve AGI, even though we aren’t. His 2017 blog post The Seven Deadly Sins of AI Predictions makes particularly worthwhile reading. (But if you do, read the edited version in the MIT Technology Review, if only for the whimsical illustrations by Joost Swarte.) He breaks down the cognitive and cultural traps that lead experts to constantly whiff on the AGI timeline—ranging from overgeneralizing narrow successes, to underestimating the complexity of the real world, to assuming straight-line progress.

So which is it? Will we get to AGI soon? Is it many years away? Or is it just a pipe dream?

Subscribe now

A Quick Refresher Course on LLMs (and LRMs)

To answer these questions, it is helpful to have a bit of background on LLMs and deep learning in general. I won’t belabor the point since there are already a zillion LLM primers on the web: Andrej Karpathy’s amazing deep dive is a good choice if you have a few hours to spare.

With that said, ChatGPT—and all the ensuing buzz around AGI—was made possible by four key innovations:

1. Deep learning - all modern AI is based on layered neural networks where one set of neurons1 applies weights to some inputs and then feeds the new values into another layer of neurons. These networks are great at tasks like speech recognition and image classification, but they are hard to train efficiently.

2. Transformers - efforts to apply deep learning to natural language were hampered by the fact that feeding in data sequentially is really slow and cumbersome. It also makes it hard for the network to prioritize more recent text while still taking older text into account. Transformers, introduced in a 2017 paper by a group of Google researchers, use clever mathematical tricks that let the network efficiently take in large blocks of text all at once, while giving more weight to recent information.

3. Lots of digitalized data - the amazing performance of LLMs is, to a large degree, down to the unbelievable amount of text now available to us in digital form. This includes websites, as well as vast libraries of digitized books, video and podcast transcripts, and online forums like Reddit and Stack Overflow.

4. Ever more powerful GPUs - A few days ago, Nvidia became the first company to reach $4 trillion in market cap. The reason is simple: they got on the GPU train early, initially as a way to efficiently display 3D graphics. Then, as luck would have it, the same math used for 3D turned out to be key to training neural networks. By scaling up the chips they had been working on since 1999, they almost single-handedly provided the vast computational resources needed to create the first modern LLMs.2

So that’s how LLMs work: unimaginable amounts of text are fed into huge, deep neural networks that use transformers to make sense out of all that sequential data in a reasonably efficient way. The daunting computational complexity is tamed by the use of extremely powerful GPUs that are tailor-made to handle these types of calculations.

Once the limitations of the original LLMs became apparent, vendors like OpenAI (with o1), DeepSeek (with R1) and Anthropic (with Claude 3 Opus) came up with LRMs. Instead of generating the output text directly, the LLM first writes a kind of step-by-step plan. It then uses this plan to guide the generation process. LRMs are still LLMs at heart, but have proven much more capable at certain kinds of tasks like solving math problems or reasoning through multi-step logic puzzles.

AG-Aye or AG-Nay?

The more optimistic predictions about AGI all assume (explicitly or implicitly) that it will be powered by LLMs. It has taken decades to get to where we are today; any significantly different approach would almost certainly take more than a couple of years to bear fruit.

Sure enough, the most prominent contestants in the race to AGI have been focused on supercharging LLMs, mainly by training them with more data and more compute. Take Kokotajlo’s AI 2027, which tries to lay out a fictional path for getting to AGI in the next couple of years. It is no coincidence that the essay contains a lot of verbiage describing the bigger and more powerful data centers that will be created by the major players:3

GPT-4 required 2⋅10²⁵ FLOP of compute to train. OpenBrain’s latest public model—Agent-0—was trained with 10²⁷ FLOP. Once the new datacenters are up and running, they’ll be able to train a model with 10²⁸ FLOP—a thousand times more than GPT-4. Other companies pour money into their own giant datacenters, hoping to keep pace.

In the real world, OpenAI has been following a similar path. When ChatGPT was first released, it used GPT-3 as its underlying model. A few months later, it moved onto GPT-4. While its exact parameter count—basically the size of the neural network—has not been disclosed, it is estimated to be about 10 times that of GPT-3. Among other things, this means that far more high quality data was needed to train it.

GPT-4 is way better than GPT-3, so naturally there was a lot of excitement about the potential of the next version: GPT-5. OpenAI didn’t do much to dampen expectations. In a speech at Stanford University in April 2024, Sam Altman made it clear he expected big things from future models:

GPT 5 is gonna be smarter than, a lot smarter than GPT 4, GPT 6 can be a lot smarter than GPT 5 and we are not near the top of this curve and we kind of know what to do and this is not like it's gonna get better in one area. This is not like we're gonna, you know, it's not that it's always gonna get better at this eval or this subject or this modality. It's just gonna be smarter in the general sense.

Last September, the President of OpenAI Japan predicted in a speech at a Japanese tech event that “GPT Next” would appear “soon” and would represent around a 100x performance improvement over GPT-4.

Then something funny happened. In December, reports started to appear about how OpenAI’s efforts to create their next big model had stalled. When they finally released the new model in February 2025, they named it GPT-4.5, rather than GPT-5, presumably to play down the hype and expectations. Developers were given access to the model, but only as a “preview”. And in April, just two months after it was released, OpenAI announced they were going remove developer access “in the coming months”.

In other words, the effort to push forward quickly to AGI by cramming more parameters, data and compute into existing LLM architectures had failed.

So does this mean that AGI skeptics like Gary Marcus and Rodney Brooks have been right all along? Yes and no.

Marcus and Brooks are both super smart guys who are experts in their fields. Nonetheless, their arguments can seem a bit dated and even downright curmudgeonly at times.

As deep learning has gone from strength to strength, culminating in the latest generation of LLMs and LRMs, Marcus’s arguments have taken on an increasingly shrill tone, as in this description of how the AI community reacted to his 2018 paper critical of deep learning:

The leaders of deep learning hated me for challenging their baby, and couldn’t tolerate any praise for the paper. When an influential economist Erik Brynjolfsson (then at MIT) complimented the article on Twitter, (“Thoughtful insights from @GaryMarcus on why deep learning won't get us all the way to artificial general intelligence”), Hinton’s long time associate Yann LeCun tried to contain the threat, immediately replying to Brynjolfsson publicly that the paper was “Thoughtful, perhaps. But mostly wrong nevertheless.”

Marcus is right that LLMs have begun to reach their limits, but seems unwilling to accept that deep learning has won because it has yielded incredible results, not because of some global conspiracy aimed at belittling his own (far less successful) efforts. And he regularly conflates the shortcomings of LLMs with those of deep learning in general when falsely accusing rivals like LeCun of secretly agreeing with (or even stealing) his ideas.

Brooks, meanwhile, has been banging the same downbeat drum since 2018 in his annual Predictions Scorecard. Overoptimistic prognoses about the future of AI have always been wrong in the past, he says, so they must be wrong now as well:

But this time it is different you say. This time it is really going to happen. You just don’t understand how powerful AI is now, you say. All the early predictions were clearly wrong and premature as the AI programs were clearly not as good as now and we had much less computation back then. This time it is all different and it is for sure now.

Yeah, well, I’ve got a Second Coming to sell you…

Brooks’s writing is intelligent and informed. It is worth reading, despite its occasionally smarmy and sarcastic tone. His skepticism about claims by AI proponents, informed by decades working in the field, is totally understandable.

But still, it is dangerous to assume that claims about some emerging technology must be wrong now since they were always wrong in the past. The same types of arguments were made about previous advances like electric lighting, airplanes and mobile phones. Consider the following statements, for example, made a few years—or even mere months—before the Wright Brothers’ first flight:

“Heavier-than-air flying machines are impossible.”4
- Lord Kelvin, President of the Royal Society, circa 1895

“Aerial flight is one of that class of problems with which man will never be able to cope.”
- Simon Newcomb, astronomer and mathematician, October 1903

“It might be assumed that the flying machine which will really fly might be evolved by the combined and continuous efforts of mathematicians and mechanicians in from one million to ten million years.”
- New York Times (editorial page), 1903

All new technologies are dismal failures… until they aren’t.

A more balanced view was expressed by Demis Hassabis, who leads AI research at Google, in a recent interview:

We’re seeing incredible gains with the existing techniques, pushing them to the limit. But we’re also inventing new things all the time as well. And I think to get all the way to something like AGI may require one or two more new breakthroughs.

It is becoming increasingly obvious to everyone involved that the initial hysteria around LLMs was overblown. They are incredibly useful tools and represent a huge step forward for AI, but they alone are not enough to achieve AGI. The addition of reasoning capabilities has made them a lot more useful, but recent research suggests that LRMs won’t get us to human-level intelligence either.

So we’re in search of “one or two more new breakthroughs”. Maybe they will arrive relatively quickly. But as LeCun said in the tweet quoted earlier, “the distribution has a long tail”. In plain English, this means that even top experts don’t really have a clue. Considering that we don’t even know what these breakthroughs might be, or how many will be needed, it is possible AGI won’t be reached for decades. Maybe never.

That is probably too pessimistic. But when you hear claims that superintelligence is just a year or two away, you can safely laugh them off.

Share