Claude Plays Pokémon: what the experiment actually showed
Anthropic's Claude Plays Pokémon experiment became an informal test of long-horizon agency. What happened, what it measured, and what it predicted about today's agent models.
Claude Plays Pokémon was an experiment in which a Claude model was given control of a Game Boy emulator running Pokémon Red and left to play it. It was never a formal benchmark. It became something more useful: the clearest public demonstration of what happens to a language model when a task is far longer than its context window.
Anthropic ran it publicly around the February 2025 release of Claude 3.7 Sonnet, the first Claude with extended thinking. Watching a model reason step by step through a children's video game explained agentic capability - and agentic failure - better than any benchmark table. This page is about what it measured and what it predicted, not about the model that ran it, which has since been retired.
#The experiment at a glance
- What it was
- A Claude model driving Pokémon Red through a Game Boy emulator, streamed publicly
- Era
- Began around the February 2025 launch of Claude 3.7 Sonnet
- What Claude received
- Game screen images, emulator state, and its own accumulated notes
- What Claude produced
- Reasoning, notes to itself, and controller button presses
- Status as a benchmark
- Informal - no official score, no published methodology
- What it actually tested
- Memory, planning and recovery across a task thousands of times longer than one context window
#What was Claude Plays Pokémon?
The setup was deliberately thin. A harness took screenshots and state from the emulator, passed them to Claude as a normal API request along with a scratchpad of notes the model had written earlier, and turned its chosen actions into button presses. Then it did that again, for as long as the run lasted.
Nothing in that loop is exotic - it is the same read-state, decide, act, record cycle every agent framework implements, and the same one an MCP tool integration gives a model over a real system. The value of a game was that every step was legible: when the model got confused, you could see exactly what it believed and why that belief was wrong.
#What the experiment actually measured
Not game skill. Pokémon Red is designed for children, and any competent player finishes it. What makes it hard for a language model is duration and statefulness: a full playthrough is a very large number of individual decisions, spread across a world you have to remember, with goals that only make sense in the light of things that happened hours earlier.
That put pressure on three things at once:
- Memory beyond the context window. The run was vastly longer than any context. The model had to compress its own history into notes, and then trust those notes - which meant every summarisation error became permanent.
- Planning across a hierarchy. "Win the game" decomposes into regions, then objectives, then routes, then individual button presses. Holding that stack together over hours is a different skill from solving a single hard problem.
- Recovery from being stuck. The interesting question was never whether the model would get stuck - it was whether it could notice, form a new hypothesis, and abandon a plan that was not working.
Those three are the axes today's long-horizon agent models are sold on. When Anthropic describes Claude Fable 5 as built for long-running agents, this is the capability it means - and the Pokémon run is the earliest public picture of what was missing.
#The failure modes, and what each one revealed
The experiment's real output was its failures, each with a direct counterpart in production agent systems.
| What went wrong in the game | What it revealed | Where it shows up in real agents |
|---|---|---|
| Looping - revisiting the same place, repeating a failed action | No reliable sense of whether progress was being made | Agents that re-run the same failing command until a budget stops them |
| Lossy self-summarisation - notes that dropped a crucial detail | Compression is where long-horizon agents actually fail | Compaction and memory tools discarding the one fact that mattered |
| Believing its own stale notes | No mechanism to distrust or expire its own memory | Acting on a cached assumption about a system that has since changed |
| Spatial confusion from tile-based screens | Reading pixels is not the same as understanding a map | Computer-use and browser agents misreading interfaces they can see |
| Plans that never got abandoned | Weak self-evaluation - persistence without a stopping rule | Long runs that burn hours and tokens on a doomed approach |
The pattern underneath all five is the same: the model was good at each individual decision and bad at knowing how the decisions were adding up. That gap - competent steps, incoherent trajectory - is still the defining problem of agentic AI, and it is why stopping rules and cost control matter more in production than raw capability. Those patterns are on the automation page.
#Where the experiment was a poor test
Two caveats, because it is often over-read.
First, a Game Boy game is a closed, deterministic world with no adversaries, no irreversible consequences and no untrusted input. Real agents work in systems where an action can delete production data, or where the input is trying to manipulate them. Nothing in Pokémon tests that. The launch measurements did include a relevant figure - prompt-injection resistance improving from 74% to 88% with extended thinking enabled - but that is a February 2025 number for a model retired on 19 February 2026, and it came from separate security evaluations, not the game.
Second, it was never controlled. Harness design, prompt wording and note-taking scaffolding all changed what the model could do, and none of it was standardised. Citing the experiment as evidence that one model is "better at agency" over-claims; it demonstrated a class of problem, not a ranking. The same caution applies to the other headline figure of that launch - a 96.5% GPQA physics subscore, again a February 2025 result for a retired model, and one reached with parallel test-time compute across many samples rather than by a single answer. How that machinery works is covered on the thinking and reasoning page.
#From curiosity to product category
In 2025, "can a model stay coherent for hours?" was a question you watched on a stream. By August 2026 it is a purchasing decision. The lineup is stratified by how long a task you intend to run: Fable 5 for long-running agents at $10 / $50 per million tokens, Opus 5 for multi-hour autonomous coding at $5 / $25, cheaper tiers for work measured in seconds. Those rates are on the pricing page.
The infrastructure changed to match. Context windows went from 200k to 1M tokens. Server-side compaction and memory tools took over note-keeping the Pokémon harness had to hand-roll. Task budgets gave agents the explicit ceiling the game run visibly lacked. And Anthropic's hosted agent harness bills a session-hour rate on top of tokens - pricing that only makes sense once "still running" is a normal state. To build on any of it, start with Claude for developers.
#What to take from it if you are building agents
Three things the experiment argues for, all still true:
- Design the memory, do not inherit it. Whatever compresses your agent's history is the component most likely to lose the thing that matters.
- Give the agent a way to know it is stuck. An external progress check - a test suite, a state diff, a budget - beats asking the model whether it is doing well.
- Cap the run. Long-horizon capability without a stopping rule is just a more expensive failure.
For the model that ran it, see the Claude 3.7 Sonnet archive; for what to build on now, the model index.
#Frequently asked questions
What was Claude Plays Pokémon?
An experiment in which a Claude model controlled a Game Boy emulator running Pokémon Red, streamed publicly around the February 2025 launch of Claude 3.7 Sonnet. It was an informal demonstration of long-horizon agency, not a scored benchmark, and Anthropic published no official methodology or result.
Did Claude finish Pokémon?
Runs varied enormously with the model, the harness and the scaffolding used, and no single official result was published. The interesting output was never the ending but the failure modes along the way: looping, lossy self-summarisation, spatial confusion and plans that were never abandoned.
What did the experiment actually test?
Three things a normal benchmark cannot reach: memory across a task far longer than the context window, hierarchical planning from a distant goal down to single button presses, and the ability to notice being stuck and change approach. Those are the same axes long-horizon agent models are sold on today.
Can I still run this with a current Claude model?
The idea, yes - an emulator plus a tool loop is straightforward to build, and computer use went generally available on the Claude API in August 2026. The original model cannot: Claude 3.7 Sonnet was retired on 19 February 2026 and its model ID now returns an error.
Is Claude Plays Pokémon a real benchmark?
No. There is no standardised harness, no controlled methodology and no published score, so results are not comparable between models or runs. Treat it as an illustration of a class of problem, not as evidence that one model is better at agency than another.
The February 2025 figures cited here come from Anthropic's Claude 3.7 Sonnet announcement at anthropic.com/news/claude-3-7-sonnet, and current agent tooling was checked against platform.claude.com/docs on 21 August 2026. Claude Plays Pokémon was an informal experiment: where a detail could not be verified from a primary source, it is described qualitatively here rather than given a number.