TL;DR
OpenAI says switching on two configuration settings on one of its models tripled its scores on ARC-AGI-3, an interactive reasoning benchmark. The specific settings and exact scores were not identifiable in material immediately available, and no independent party has verified the result. The claim highlights how sensitive AI benchmark numbers are to evaluation setup.
OpenAI says that enabling two settings on one of its models tripled its scores on ARC-AGI-3, an interactive benchmark designed to test whether AI systems can learn unfamiliar tasks from scratch. The claim comes from a company technical blog post titled ‘How enabling two settings tripled our scores on the ARC-AGI-3 benchmark.’ The two settings were not identified in the material immediately available, and the result has not been independently verified.
The post, published on OpenAI’s website, frames the finding as a configuration effect rather than a capability gain: the same underlying model produced roughly three times higher benchmark scores once the two settings were switched on. The company presented the work as a lesson in how sensitive benchmark results can be to evaluation setup. Because the full article body was not accessible at the time of writing, the specific settings, the baseline and final scores, and the exact model version could not be confirmed from the post itself.
What is confirmed: the post exists, it carries that headline, and it reports a threefold score increase tied to two settings. What is not confirmed: which settings were changed, whether the runs used the ARC Prize Foundation’s official evaluation harness, whether scores came from public or private tasks, and how much compute each run consumed.
ARC-AGI-3 is the third generation of the ARC benchmark family from the ARC Prize Foundation, the organization built around researcher François Chollet’s Abstraction and Reasoning Corpus. Unlike the earlier static puzzles, ARC-AGI-3 places agents in interactive, game-like environments where they must infer rules through trial and error, with no instructions — a design intended to resist memorization and measure skill acquisition.
Benchmark Claim Analysis · Late July 2026
Two Settings. One Model. Three Times the Score.
OpenAI says switching on two configuration settings on one of its models tripled its scores on ARC-AGI-3, an interactive reasoning benchmark. The company frames it as a configuration effect — not a capability gain. The settings, the scores, and the model version remain undisclosed, and no independent party has verified the result.
“A configuration effect rather than a capability gain.”
OpenAI’s own framingThe Evidence Ledger
What Is Confirmed — and What Is Not
The full article body was not accessible at the time of writing. Only the headline-level facts could be established; every methodological detail remains open.
- The post exists — published on OpenAI’s website as a technical blog post.
- The headline — “How enabling two settings tripled our scores on the ARC-AGI-3 benchmark.”
- A threefold increase is reported — the same underlying model scored roughly 3× higher once the settings were switched on.
- The framing — presented as a lesson in how sensitive benchmark results are to evaluation setup.
- Which settings changed — not identified in any material immediately available.
- Baseline and final scores — absolute numbers and the exact model version are unknown.
- The evaluation harness — whether runs used the ARC Prize Foundation’s official protocol.
- Task set and compute — public vs. private tasks, and the compute cost of each run.
Claim Audit
Line-by-Line Verification Status
Each element of the claim, checked against publicly available material as of late July 2026.
| Claim Element | Status | Detail |
|---|---|---|
| Blog post published with that headline | ✓ Confirmed | Live on OpenAI’s website; framed as a configuration finding. |
| Roughly threefold score increase reported | ✓ Confirmed | Confirmed as a claim made by OpenAI — not as a measured fact. |
| Identity of the two settings | ✗ Undisclosed | The post describes them only as “two settings”; specifics unavailable. |
| Before / after absolute scores | ✗ Undisclosed | Neither baseline nor final figures could be confirmed; model version unknown. |
| Official ARC Prize evaluation harness used | ~ Unknown | Public vs. private task split and protocol compliance unverified. |
| Independent replication | ✗ None yet | Neither the ARC Prize Foundation nor any third-party lab has confirmed the figures. |
The Benchmark Behind the Claim
How ARC Evolved Into ARC-AGI-3
Built by the ARC Prize Foundation around François Chollet’s Abstraction and Reasoning Corpus, the benchmark family moved from static puzzles to interactive environments designed to resist memorization.
ARC
François Chollet introduces the original benchmark: static grid puzzles testing abstraction and reasoning rather than pattern matching.
ARC-AGI-2
A harder second generation arrives. Frontier results — including OpenAI’s o3 with heavy test-time compute in late 2024 — spark debate on cost and methodology.
ARC-AGI-3
Agents enter interactive, game-like environments with no instructions — inferring rules through trial and error. Built to measure skill acquisition and fluid reasoning.
The Size of the Swing
Same Model. Different Setup. 3× the Number.
OpenAI’s own account attributes the entire jump to configuration — the underlying model did not change between runs. Indexed score illustration below (baseline = 1×); absolute scores were not disclosed.
The signal is only as clean as the setup.
If flipping two settings can multiply a score several times over, results reported under different configurations may not be comparable — even when they cite the same benchmark.
ARC-AGI-3 carries particular weight because it was built to measure fluid reasoning. A configuration-driven jump on it lands directly in the live debate over evaluation hygiene: how much of any reported gain is the model, and how much is the harness, prompts, tool access, and token budgets around it.
Reading the Gap
How a Score Can Triple Without the Model Changing
With the settings undisclosed, the plausible levers are the known pressure points of any evaluation harness.
Compute & Token Budgets
More test-time compute per task can lift scores dramatically — the same debate that surrounded o3’s headline ARC-AGI result in late 2024.
Interface & Tool Access
ARC-AGI-3 is interactive. How an agent perceives and acts inside the environment — retries, feedback channels, action formats — can reshape outcomes.
Attempt Aggregation
How individual attempts are pooled into a final score — best-of-N, pass rates, averaging — can move headline numbers by large margins on its own.
Verification Roadmap
What to Watch as Verification Catches Up
The claim now sits in the queue of every group that maintains or studies the benchmark.
Independent replication
Whether the ARC Prize Foundation or third-party researchers can reproduce a threefold jump under the official harness — and at what cost per task.
A formal leaderboard submission
An OpenAI entry with disclosed configuration and compute figures, measured against the Foundation’s public cost and efficiency constraints.
Competing labs’ results
Several rival labs are expected to report their own ARC-AGI-3 numbers — under configurations that may or may not match.
Pressure for standardized settings
If configuration effects of this size hold up, expect growing demands for industry-wide standardized evaluation protocols.
Key Questions
Frequently Asked Questions
Straight answers based only on what is currently known.
What two settings did OpenAI enable?
OpenAI has not identified them in the material immediately available. The post describes the change only as “two settings”. Until specifics are disclosed, any description would be speculation.
What is ARC-AGI-3?
The third generation of the Abstraction and Reasoning Corpus family, overseen by the ARC Prize Foundation. Agents must work out the rules of interactive, game-like environments on their own — measuring skill acquisition, not memorized patterns.
Has the tripled score been independently verified?
No. As of late July 2026, neither the ARC Prize Foundation nor any third-party lab has publicly confirmed the figures. Treat it as a vendor result pending replication under the official harness.
Does 3× the score mean 3× the capability?
No. OpenAI’s own framing attributes the jump to configuration — the same system produced both sets of scores. A settings-driven swing does not translate into a proportional gain in general capability.
Why do benchmark settings matter for comparing AI labs?
Evaluation choices — compute budgets, tool access, how attempts are scored — can move results by large margins. If two labs test under different configurations, their scores on the same benchmark may not be meaningfully comparable. Disclosed settings and standardized harnesses are what make leaderboard claims readable.
Traceability
From Blog Post to Pending Proof
Why Evaluation Settings Are Drawing Scrutiny
Benchmark scores have become the primary currency for comparing AI systems, shaping headlines, investor sentiment and customer decisions. If flipping two settings can multiply a score several times over, then results reported under different configurations may not be comparable — even when they cite the same benchmark. That has direct implications for how readers should treat leaderboard claims from any lab.
The finding also lands in an active debate about evaluation hygiene in AI: how much of a reported gain reflects the model, and how much reflects the harness, prompts, tool access and token budgets around it. ARC-AGI-3 carries particular weight because it was built to measure fluid reasoning rather than learned patterns, and it is closely watched by researchers who treat progress on it as one signal of movement toward more general intelligence. A configuration-driven jump on that benchmark is a reminder that the signal is only as clean as the setup.
As an affiliate, we earn on qualifying purchases.
How ARC-AGI-3 Measures Machine Reasoning
François Chollet introduced the original ARC benchmark in 2019 to test abstraction and reasoning rather than pattern matching. A harder second version arrived in 2025, and the foundation then developed ARC-AGI-3 as an interactive extension, moving from static grids to environments an agent must explore.
Frontier models’ performance on earlier ARC versions has been a recurring flashpoint. OpenAI’s o3 system posted headline results on the original ARC-AGI in late 2024 using large amounts of test-time compute, prompting debate about cost and methodology even as the achievement was acknowledged. The ARC Prize Foundation maintains public leaderboards with cost and efficiency constraints intended to keep submissions comparable — the same standard any new claimed result would be measured against.
Scaling AI: The AI Governance and Security Playbook for Executives
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Which Settings Were Used Remains Unconfirmed
The central unknown is which two settings OpenAI enabled and how each contributed to the gain. Also unconfirmed: the absolute scores before and after, the model version tested, whether the runs followed the official ARC-AGI-3 evaluation protocol, and whether results applied to public tasks, private tasks or an internal variant. No independent lab, and not the ARC Prize Foundation, has publicly confirmed the figures as of late July 2026. It is also not clear whether the improvement reflects better use of compute, better interaction with the environment’s interface, or changes in how individual attempts were aggregated into a final score.
As an affiliate, we earn on qualifying purchases.
What to Watch as Verification Catches Up
The immediate milestone is independent replication: whether the ARC Prize Foundation or third-party researchers can reproduce a threefold jump under the official harness, and at what cost per task. Readers should also watch for a formal leaderboard submission from OpenAI with disclosed configuration and compute figures, and for responses from competing labs, several of which are expected to report their own ARC-AGI-3 results. If configuration effects of this size hold up, pressure will grow for standardized evaluation settings across the industry.
Source: OpenAI
As an affiliate, we earn on qualifying purchases.
Key Questions
What two settings did OpenAI enable?
OpenAI has not identified them in the material immediately available. The company’s blog post describes the change only as ‘two settings,’ and the full article text could not be confirmed at the time of writing. Until OpenAI’s post or a follow-up discloses specifics, any description of the settings would be speculation.
What is ARC-AGI-3?
ARC-AGI-3 is the third generation of the Abstraction and Reasoning Corpus benchmark family overseen by the ARC Prize Foundation. It places AI agents in interactive, game-like environments where they must work out the rules on their own, aiming to measure skill acquisition and fluid reasoning rather than memorized patterns.
Has the tripled score been independently verified?
No. As of late July 2026, neither the ARC Prize Foundation nor any third-party lab has publicly confirmed OpenAI’s figures. The claim should be treated as a vendor result pending replication under the official evaluation harness.
Does a threefold score increase mean the model became three times more capable?
No. OpenAI’s own framing attributes the jump to configuration, not to a change in the underlying model — the same system produced both sets of scores. Benchmark results reflect performance on specific tasks under a specific setup, so a large score swing driven by settings does not translate directly into a proportional gain in general capability.
Why do benchmark settings matter for comparing AI labs?
Evaluation choices — such as compute budgets, tool access and how attempts are scored — can move results by large margins. If two labs test under different configurations, their scores on the same benchmark may not be meaningfully comparable, which is why disclosed settings and standardized harnesses matter for reading leaderboard claims.
Source: OpenAI
