I got API access to GPT-6 Astra this week. Before I ran a single prompt, I did what this publication always does: read the benchmark. And the benchmark everyone is quoting doesn’t say what they think it says.

The circulating comparison — you’ve seen it — runs like this. Claude Fable 5.1 scores 66 on the Artificial Analysis Intelligence Index; GPT-6 Astra scores 61. Five points is not a rounding error. But Fable burned 140 million output tokens to get there against Astra’s 42 million, at $3.69 per task versus $1.67. Conclusion: Fable wins the leaderboard, Astra wins the economics, and the next AI race is about intelligence per dollar.

It’s a tidy story. It’s built on numbers that were already stale when it was written, measuring a quantity that no longer means what it used to, aggregated in a way that hides the reversals that matter. Here’s what’s actually wrong — in ascending order of seriousness.

Problem one: the numbers moved while nobody was looking

Go to Artificial Analysis today. Fable 5.1 (Adaptive Reasoning, Max Effort) scores 57. GPT-6 Astra (max) scores 55. Astra (xhigh) scores 54.

Not 66 and 61. 57 and 55. The five-point gap that “is not a rounding error” is a two-point gap — which is, in fact, well inside anything you’d want to call a rounding error on an aggregate of ten evaluations.

What happened? The Index was revised. Version 4.1.1 became version 4.2 in the days around Astra’s launch. GPQA Diamond was dropped. AA-Briefcase and GDP.pdf were added. AA-LCR moved to v1.1. Every model was re-scored against a different basket, and every absolute number shifted. A third source, writing a day after launch, quotes Astra at 60 — a third figure, from a third snapshot.

None of this is Artificial Analysis doing anything wrong. Revising an index is how you keep it honest as the frontier moves. The error is downstream: writers grab a number, build a narrative on its magnitude, and the narrative outlives the number. The “five points” comparison was published against one version of the Index and is being read against another. It’s not that the benchmark is lying. It’s that the benchmark is a moving object being quoted as a fixed one.

Rule for the rest of this piece: quote the version, or don’t quote the number.

Problem two: AA’s own conclusion is the opposite of the story being told

The circulating narrative is that Astra “attacks the economics” of intelligence. Read what Artificial Analysis actually published about Astra, in its own benchmarking note:

GPT-6 Astra is 75% more expensive than GPT-5.6 Sol at max effort, and largely sits behind its predecessor on the Intelligence Index vs Cost-per-Task frontier. Pricing went up 2.5× — from $4/$20 to $10/$50 — and the token-efficiency gains only partially offset it. On the general Intelligence Index, AA’s verdict is that Astra is a worse intelligence-per-dollar deal than the model it replaced.

So the “efficiency” story has a specific, narrow home, and it isn’t the Intelligence Index. It’s the Coding Agent Index, where AA finds something real: Astra equals Fable 5 at less than half the cost, driven by a ~3× token reduction versus Sol. That is a genuine result. It’s also a coding result, and the circulating article stretched it across the entire intelligence comparison, where AA’s own data says the opposite.

Two things can be true. Astra is on the Pareto frontier for coding agents. Astra is off the frontier for general intelligence-per-dollar. Collapsing those into “Astra wins the economics” is how a true statement about one index becomes a false statement about another.

Building Robust AI Evals: Proven Strategies for Testing, Monitoring, and Improving LLM Performance (Engineered: Data, AI, and DevOps)

View Latest Price

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

Problem three: tokens stopped being a proxy for compute — and the Index doesn’t know

This is the one that matters, and it’s architectural.

GPT-6 Astra is widely reported — by Alan Thompson at LifeArchitect, by Sebastian Raschka, in a LessWrong analysis — to be a looped or recurrent-depth transformer. OpenAI’s system card documents the observable consequence without confirming the mechanism: Astra can complete a greatly expanded set of tasks without writing out any chain of thought. Processing layers run again on their own updated state. The model thinks in latent space. It reasons without emitting tokens.

Now look at what the Artificial Analysis Index measures. Cost per task is calculated from input, cache-hit, cache-write, reasoning, and answer token prices. “Verbosity” is output tokens. “Time per task” is output tokens divided by output speed. The entire efficiency apparatus of the Index is denominated in tokens.

For a model that reasons in tokens, that’s a fine proxy for compute. For a model that reasons in loops, it measures the receipt, not the work.

And the Index’s own data shows it. Look at the effort ladder for Astra on AA’s pages: Astra (non-reasoning) scores 55. Astra (max) scores 55. Medium scores 52, xhigh 54. The verbalized-reasoning dial — the thing the entire “reasoning tokens” column exists to track — barely moves the score, because the reasoning isn’t happening in the tokens. Non-reasoning Astra used 4.4 million tokens for the Index run; max used enough to cost $3,020. Same score. The Index is faithfully counting tokens that are no longer where the intelligence lives.

So when the circulating comparison says Fable used 140M tokens and Astra used 42M, it’s comparing two different things: Fable’s verbalized reasoning against Astra’s post-latent-loop output. That’s not a token-efficiency finding. It’s an artefact of one architecture externalizing its work and the other hiding it. What it costs OpenAI in GPU-seconds to run those latent loops isn’t visible in the token count, and it isn’t visible in the price either — because the $50/M output price is a list price that already bakes in whatever the loops cost, or doesn’t. We can’t tell. Nobody outside OpenAI can.

The honest statement is narrower than either camp wants: Astra is cheaper per task at API list price on some workloads, and nobody knows whether it is more compute-efficient, because the metric that used to tell us stopped working when the architecture changed.

Problem four: “max effort” is not the same experiment twice

Every Index comparison is run at each model’s maximum reasoning setting. That was a reasonable convention when maximum effort meant the same thing for every model: more tokens.

It doesn’t anymore. Fable at max effort means a larger token budget. Astra at max effort means — per the ladder above — approximately nothing, because the model is doing the same latent work regardless. So the headline comparison pits Fable with the throttle open against Astra with a throttle that isn’t connected to the engine.

There’s a further wrinkle. OpenAI’s API documentation states plainly that GPT-6 Astra does not support the none reasoning effort. Yet AA publishes a “GPT-6 Astra (Non-reasoning)” score. Either AA had access to a setting developers don’t get, or “non-reasoning” on the Index means something other than reasoning: none. Either way, the most efficient-looking Astra configuration on the leaderboard may not be one you can buy. Check before you build a cost model on it.

Problem five: the aggregate hides the reversals

The Index blends ten evaluations into one number. That number says Fable leads by two. OpenAI’s own published evaluations — vendor-run, self-reported, and to be treated with exactly that suspicion — show Astra ahead on six of seven head-to-head tasks: AutomationBench, BenchCAD, Terminal-Bench 4.0, DeepSWE, Terminal-Bench Science, FrontierMath Tier 4. Fable takes Humanity’s Last Exam with tools, 65.0 to 57.2.

You don’t have to believe OpenAI’s numbers. You do have to notice the shape: on a majority of individual, named tasks — several of them agentic and technical — the vendor claims a reversal that the aggregate erases. An index that reports “Fable +2” while the components split 6–1 the other way isn’t wrong. It’s just answering a question (“who is best on average across our chosen basket?”) that has almost nothing to do with the question you have (“which model completes my task?”).

And the basket is a choice. Drop GPQA Diamond, add a briefcase-of-documents eval and a PDF task, and the ranking moves. The weights are a choice. The effort setting is a choice. Ten choices deep, a two-point gap is noise wearing a number.

What actually changed — and it’s not on the leaderboard

Bury the horse race for a moment. The two most important Astra facts in AA’s data aren’t rankings.

Hallucination rate fell from 92% to 51% on AA-Omniscience at max effort, alongside a modest accuracy gain. That’s a forty-one point drop in the thing that makes models unusable in production. It matters more than any two points of Index.

And the cache economics are not “the same headline pricing.” Both models list $10/$50. Fable’s cache read is $0.25/M; Astra’s is $1.00/M — four times higher. Astra also carries a 25% cache-write premium. For an agentic workload that re-reads a large context on every step — which is every agentic workload — the cache line dominates the bill, and it’s the line the “same price” framing skips. Two models at “identical” list price can differ by multiples in practice depending on how much context they re-read.

The take

The benchmark isn’t broken. The way it’s being read is.

Three things happened at once. The Index was revised, so the numbers people are quoting are from a version that no longer exists — five points became two. The architecture changed, so the token-based efficiency metrics are measuring the output of the reasoning rather than the reasoning — “Astra is efficient” and “Astra is expensive” are both true, on different indices, and nobody knows what it costs to run. And the aggregate did what aggregates do: it turned a 6–1 task-level split into a two-point average and let the average become the story.

If you take one thing from this: a leaderboard position now tells you less than it ever has, and the more advanced the architecture, the less it tells you. Latent reasoning is only the first architecture to break the token proxy. Every one after it will break it further.

So here’s what to do with your Astra access, and mine. Ignore the Index number. Take your ten most important tasks. Run both models at the effort setting you’ll actually pay for — not max, not a “non-reasoning” mode that may not exist in the API. Measure the bill, including the cache line. Measure the failure rate, because the 41-point hallucination drop is the only number in this whole affair that I’d bet real money on.

Then decide. The benchmark can’t do it for you anymore. That was the point of this article, and it may be the point of this year.

Sources: Artificial Analysis Intelligence Index v4.2 and v4.1.1 pages for Claude Fable 5.1 and GPT-6 Astra at non-reasoning, low, medium, high, xhigh and max effort (scores, output tokens, total evaluation cost, cost-per-task methodology including cache-write and reasoning token prices), and AA’s “Benchmarking GPT-6 Astra” note (75% more expensive than Sol at max effort; behind Sol on the Intelligence Index cost frontier; equal to Fable 5 on the Coding Agent Index at under half the cost; ~3× coding-agent token reduction; hallucination rate 92%→51% on AA-Omniscience; 2.5× price increase from $4/$20 to $10/$50; 90% cache-read discount and 25% cache-write premium); OpenAI developer documentation for GPT-6 Astra (fewer-output-tokens claim; none reasoning effort unsupported; temperature/top_p/logprobs removed; cache-write billing; EU data-residency fast-mode restriction); Alan D. Thompson, The Memo, 4 September 2026 (recurrent-depth/looped-transformer characterization, system-card citation on capability without CoT, token-efficiency user reports; architecture explicitly unconfirmed); Sebastian Raschka and LessWrong analyses as cited there; OpenAI’s published Astra-vs-Fable evaluation table (self-reported, unreplicated); a DEV Community post quoting an Astra score of 60 and $0.96/task as a third snapshot; and the circulating comparison article quoting 66/61 and $3.69/$1.67, which appears to reflect a pre-v4.2 Index. Index scores are version-dependent and were changing at time of writing; all vendor figures are self-reported. Analysis and framing are the author’s; not investment advice.