Mechanism Triggers & Recognition
Mark when a mechanism is triggered and the earliest evidence of correct understanding.
How Humans Learn Unfamiliar Worlds:
A Think-Aloud Dataset on ARC-AGI-3
1 Nanyang Technological University2 TU Darmstadt3 Basis Research Institute4 Harvard University5 Cornell University
Abstract
Learning through exploration and using newly acquired knowledge to achieve goals are central capabilities of general intelligence. Understanding these capabilities requires tracing how learners build, use, and revise world models during interaction. We construct the first dataset that records how humans discover and use the mechanisms of novel environments, using ARC-AGI-3 as a testbed: 144 trajectories from 64 participants across 25 public games, each pairing time-synchronized actions with think-aloud audio, plus 250 trajectories from large language model (LLM) agents with and without harnesses on the same games. We annotate mechanism triggers and recognition, action purposes, and errors and recovery, linking changes in understanding to the observations and actions that support them. We analyze characteristics of human cognition and compare them with those of LLM agents. Both groups anticipate mechanisms and use familiar analogies. Qualitative representations are more common in successful human trajectories than in successful LLM trajectories (50.0% versus 14.3%). Humans more often explore unfamiliar situations without a concrete hypothesis, while hypothesis testing is common in both groups. We also trace how learners recover through belief revision, replanning, and execution correction. As LLMs with well-designed harnesses reach saturated performance on public ARC-AGI-3 games, our cognitive analyses motivate next-generation benchmarks that challenge familiar priors, require accumulated evidence and abstraction, and test planning through subgoals, experimentation, and recovery.
Data Collection & Annotation
Humans
- Recruited through Prolific to think aloud while playing.
- Game actions synchronized with full audio recordings.
- 64 participants and 144 successful trajectories.
LLM
- Public action and reasoning records from the same 25 games.
- 100 trajectories without a harness.
- 150 trajectories with a harness.
Action Purposes
Distinguish goal pursuit, information seeking, and dual purposes, including exploration, hypothesis testing, and investigation of unexpected outcomes.
Errors & Recovery
Identify incorrect beliefs, planning failures, and execution deviations, linking each to subsequent corrections.
Dataset Explorer
Explore original records from all 25 games. Select a trajectory to play its audio and browse synchronized actions, transcripts, and annotations.
No matching trajectories. Try another game or clear the filters.
Environment & ActionsACTION 0
Loading trajectory…
⌘
Public Agent Record
Browse public reasoning and three annotation categories by action.
Open original replay ↗Think-Aloud Transcript
00:00 / 00:00
Findings
1. Humans Draw on Prior Knowledge
Of 1,425 human mechanism instances with both trigger and recognition annotations, 57 (4.0%) show evidence of correct understanding before the first trigger. Across the five LLM+harness configurations analyzed, this proportion is 7.3%–11.3%. Prior knowledge helps both groups anticipate the rules of unfamiliar environments.
For example, a participant in TU93 identifies the green endpoint as the goal and moves toward it before reaching it.
Analogies occur in 37/144 (25.7%) of human trajectories and 26/119 (21.8%) of successful LLM trajectories. For example, a human interprets an object in BP35 as a Mario-style pipe, then rejects that interpretation after feedback; LLMs draw on peg solitaire or Lights Out to interpret games.
2. Humans Use Qualitative Representations
Qualitative representations describe how a mechanism works without fully specifying its conditions, timing, or numerical details. For example, a participant in TU93 plans a route using the idea that the enemy follows their path, without specifying the movement delay.
LLM records also contain precise numerical descriptions: OPINE–Claude describes the TU93 patrol route using five row coordinates and identifies a round trip of eight valid moves.
3. Action Purposes
Goal and Information Value
An action can advance a goal, provide information about the rules, or do both.
| Action purpose | Humans | LLM |
|---|---|---|
| Goal pursuit | 47.2% | 57.1% |
| Information seeking | 9.1% | 11.9% |
| Dual purpose | 28.4% | 30.5% |
| Other / insufficient evidence | 15.3% | 0.5% |
For example, in TU93, a participant follows a plan to approach and eat an enemy, illustrating goal pursuit; in CN04, a participant attempts to connect shapes while trying to see what happens, combining goal pursuit with information seeking.
Purposes of Information Seeking
Among actions with an information-seeking purpose, humans more often explore unfamiliar situations without a specific hypothesis. Both groups frequently test hypotheses. These statistics also include actions that simultaneously pursue goals.
| Information-seeking purpose | Humans | LLM |
|---|---|---|
| Exploring unfamiliar situations | 40.6% | 10.7% |
| Hypothesis testing | 49.2% | 49.0% |
| Investigating unexpected outcomes | 9.8% | 4.8% |
| Mixed situations | 0.1% | 35.5% |
- LF52: Encountering unfamiliar purple objects, a participant clicks them and tries to make green objects jump over them.
- SC25: A participant hypothesizes that the yellow pattern enables teleportation, draws it, and checks the result.
- SC25: After arriving at an unexpected location, a participant reviews their actions, tries other patterns, and suggests that character size may affect the outcome.
4. Errors & Recovery
Recovery includes belief revision, replanning, and input correction. The table reports events with observed correction / error events.
| Error type | Humans | LLM |
|---|---|---|
| Incorrect mechanism beliefs | 11 / 15 | 451 / 593 |
| Planning failures | 916 / 1,002 | 728 / 822 |
| Execution deviations | 134 / 136 | 6 / 7 |
Humans / LLMs also explicitly express a lack of understanding of the rules in 79 / 29 events and an undetermined goal in 57 / 83 events. Even when the goal is undetermined, they may propose candidate goals (humans: 36/57; LLMs: 79/83). Public reasoning, code, and notes make explicit conclusions easier to inspect than think-aloud speech, so event counts should not be interpreted directly as error propensity.
- CN04: A participant revises the belief that resizing requires entering another shape, discovering that clicking a white endpoint is sufficient.
- LF52: After repeated ineffective clicks, a participant discovers the arrow-key controls and recognizes that multiple platforms move together.
- CN04: A participant moves from uncertainty about how red endpoints should connect to understanding how the shapes fit together.
- LP85: A participant plans to position the orange object before moving the yellow one, then replans after discovering that they move together.
- LP85: A click outside a control has no effect; the participant then adjusts the click location.
Implications for Benchmark Design
Counterfactual and Emergent Mechanisms
Introduce rules that violate familiar expectations, or define only local dynamics so that higher-level mechanisms emerge from interactions among components. Learners must discover the actual rules through experimentation and construct and test combinations that produce useful outcomes.
Induction through Accumulated Evidence and Abstraction
For instances with both trigger and recognition annotations, the mean action gap between the first trigger and first recognition is 90.8 for humans and 24.4 for LLM+harness agents. Future tasks could introduce hidden states, stochastic outcomes, and delayed effects to require evidence accumulated across interventions. More complex environments could make complete, precise modeling impractical, encouraging qualitative abstractions suited to the current goal.
Useful Subgoals and Testing Combined Outcomes
In levels meeting the analysis criteria, humans still require an average of 95.1 actions to finish after recognizing all relevant main mechanisms, compared with 49.2 for LLM+harness agents. Future tasks could require longer plans and deeper subgoal dependencies, such as building an automated sorting and transport system: decomposing functions, constructing components, testing how they work together, and revising plans after failures.