Beware: LLMs Do Not Reason

Meta description: How AlphaGo's 2016 Move 37 shows today's AI lacks real reasoning, as Thore Graepel leaves Google DeepMind to build auditable machine systems.

Tags: AlphaGo, Google DeepMind, machine reasoning, Chain of Thought, Lee Sedol

Featured image: abstract technology illustration of a Go board branching into an auditable reasoning tree; no text, no company logos, no people.

Background: From Deep Blue to Move 37

In March 2016, on an afternoon in Seoul, a program Thore Graepel helped build placed a stone on the fifth line of a Go board. Move 37 in game two of the five-game match appeared so absurd that some commentators thought it was a programming glitch. It was not. AlphaGo won that game and ultimately triumphed 4-1 over Lee Sedol. “I thought AlphaGo was based on probability calculation and that it was merely a machine,” Lee said afterwards . “But when I saw this move, I changed my mind. Surely, AlphaGo is creative.”

Go was more complex than chess. Deep Blue defeated then reigning world chess champion Garry Kasparov in 1997 by looking six to eight moves ahead per player and evaluating 200 million chess positions per second using rules hard-coded by humans. In Go, a stone’s value depends on how distant groups and territory unfold over dozens of moves. Computing even a fraction of possible outcomes would take a supercomputer billions of years. AlphaGo had to sense who was ahead at a glance and invent moves no human had thought to play.

Development: Search Made Move 37 Possible

Many accounts call Move 37 pure machine intuition, but that is a misunderstanding. AlphaGo had two systems. Its policy network was trained to guess what move a strong human would play. It regarded Move 37 as nothing special, with roughly a one in 10,000 chance of being played by an expert human. AlphaGo chose it because its search machinery looked beyond immediate plausibility and weighed future consequences. It explicitly constructed and searched a game tree with thousands of branches, each representing a different possible future.

This mirrors Daniel Kahneman’s split: System 1 is fast and effortless; System 2 is slow and deliberative. Networks supplied hunches, while search tested them against later moves and countermoves. Intuition alone would not have chosen Move 37, and brute-force search alone would have struggled to sieve through too many possibilities.

Current Gap: Chain of Thought Is Not Separate Reasoning

Today’s large language models pick the next token, or next text unit, over and over. That is System 1: fast, associative pattern completion across many subjects. After ChatGPT debuted, the field realized language fluency alone falls short of true usefulness. Models began generating intermediate steps that decompose problems, carry forward partial results, and influence subsequent reasoning, a process known as chain of thought. Gains proved real, especially in mathematics and coding. But that intermediate reasoning is still produced by the same next-token prediction process, iterated longer before an answer; it is not a separate mechanism like AlphaGo’s search.

Three shortcomings keep chatbots from qualifying as reasoning. First, they usually maintain no explicit, persistent, inspectable epistemic state: no open ledger of hypotheses, confidence, evidence, and unresolved questions revised as information arrives. Second, they lack clean separation between knowledge and reasoning; both are interwoven in neural-network weights. Third, research has demonstrated that bots often concoct chains of thought after the fact, reaching an answer by one route while reporting another.

In medicine, engineering, and scientific research, the path to a conclusion matters. If a diagnosis or treatment plan fails, researchers need to know whether reasoning was at fault, evidence was invalid, or assumptions were incorrect.

Latest Progress: Graepel’s Epistemic-State Proposal

Graepel recently left his position at Google DeepMind to pursue a fresh approach drawn from AlphaGo’s architecture. AlphaGo maintains a game tree: a record of what it knows about a position, with all considered variations annotated by its neural networks. It updates the tree and synthesizes information to decide a move.

A system should maintain an epistemic state: what is settled, doubtful, ruled out, and open. Reasoning is a sequence of moves that changes the state to advance knowledge and reduce uncertainty: deducing consequences, breaking problems apart, and deciding what question, calculation, or experiment comes next.

Open-world reasoning is harder than Go or chess: states are partially known, actions are large and variable, and consequences are stochastic or unknown. Yet LLMs and other neural models can suggest ways to tackle problems from known facts and resources, interact with tools via APIs or code, and help assess whether claims are supported by evidence. Graepel says an independent evaluator must judge each move by how much it resolves uncertainty, updating beliefs only when evidence supports the change. Enforced this way, a model could accumulate certified knowledge and improve its reasoning policy from past experiences—scientific method on steroids for knowledge that withstands scrutiny.

What This Means for AI's Next Stage

Trustworthy machine intelligence will not come from making System 1 bigger. Scale sharpens intuition but does not make it deliberative. Move 37 mattered because a machine held a position, weighed possible futures, and chose a move its artificial instincts would likely have rejected. Society needs such creative moves in drug discovery, materials, climate, and diagnosis, where nobody hands us the rules. The field is now trying to turn fluent pattern recognition into auditable sequences of evidence, inference, and belief revision, rather than after-the-fact stories.

Thore Graepel is chair of machine learning at University College London. He was a core member of the AlphaGo team at DeepMind and works to ensure that AI benefits human flourishing.