TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Hugging Face researchers report that a ReAct agent using GPT-4.1 on AppWorld succeeds on 77.4% of runs on average but completes all five repeated runs for only 53.0% of tasks. A new Consistency Analyzer diagnoses flip-prone decisions, and consistency guidelines derived from it cut the gap from 24.4 to 12.0 points without reducing average accuracy.
Hugging Face researchers have introduced a new diagnostic and fix for a reliability problem in LLM agents: the same task can succeed one run and fail the next. In a post accompanying a technical report on arXiv, the team reported that a ReAct agent using GPT-4.1 on the AppWorld benchmark achieved a Mean@5 of 77.4% but a Pass^5 of just 53.0% — meaning it solved every one of five repeated runs for barely half the tasks. The post also introduced consistency guidelines, a new feature of the ALTK-Evolve system that the researchers said halved the consistency gap, from 24.4 points to 12.0 points, without costing average accuracy.
The core finding concerns what the researchers call the consistency gap: the difference between Mean@k, the average pass rate over k runs that leaderboards typically report, and Pass^k, the fraction of tasks an agent succeeds on in all k runs. In their setup, the agent ran at temperature 0.0, so the variance was not ordinary sampling randomness — yet nearly a quarter of the benchmark consisted of tasks the agent sometimes solved and sometimes did not, with nothing about the task changing between attempts. On hard tasks, the researchers said, the gap reaches about 30 points.
To find where trajectories diverge, the team built the Consistency Analyzer. Given a single recorded trajectory, it replays each decision step through controlled resampling — one additional model call per step, drawing k completions at once, with k=5 by default — against the already-recorded context. The approach requires no ground truth and no second end-to-end rollout. Each step receives a consistency score pinpointing decisions where the model’s next-token distribution is nearly tied between options.
The researchers explain the mechanism in terms of distribution shape: sharp distributions concentrate probability on one token and survive small perturbations such as floating-point non-associativity or request batching, while flat distributions spread mass across near-tied tokens that can reorder under slight shifts. Because a trajectory chains dozens of decisions, a small per-step flip chance compounds into a large chance that some run diverges. Consistency guidelines turn the analyzer’s scorecards into reusable rules injected back at inference time through the existing ALTK-Evolve pipeline, which distills guidelines from an agent’s own past trajectories. According to the post, this raised same-task Pass^5 by 16.0 points and similar-task performance by 13.0 points.
Your Agent Aced the Task. Will It Do It Again?
A ReAct agent with GPT-4.1 on AppWorld solves tasks 77.4% of the time on average — but completes all five repeated runs for barely half of them. A new Consistency Analyzer and inference-time guidelines aim to close the reliability gap.
Accuracy hides an unreliability problem. A 24.4-point consistency gap separates average success from repeatable success.
Hugging Face blog postGood on Average, Unreliable on Repeat
Leaderboards typically report Mean@k — the average pass rate over k runs. But real users ask the same question twice. Pass^k measures the fraction of tasks an agent succeeds on in every one of k runs. On AppWorld’s test_normal split, the same agent produces two very different stories depending on which metric you read.
Flat Distributions, Chained Decisions
The agent ran at temperature 0.0, so this isn’t sampling randomness. The researchers explain the flips through distribution shape: when a model’s next-token distribution is nearly tied, tiny perturbations — floating-point non-associativity, request batching, endpoint-side shifts — can reorder the tokens. And because a trajectory chains dozens of decisions, a small per-step flip chance compounds into a large chance that some run diverges.
Probability concentrated on one token
Sharp distributions survive small perturbations. The same token wins under floating-point shifts, batching changes, and endpoint variability — the step replays identically.
Mass spread across near-tied tokens
Flat distributions can reorder under the slightest shift. Each such decision step is a coin-flip in disguise — and dozens of chained flips make full-run repetition unlikely.
From Trajectories to Consistency Guidelines
The Consistency Analyzer replays each decision step of a recorded trajectory against the already-recorded context — one extra model call per step, drawing k=5 completions at once. No ground truth, no second end-to-end rollout. Each step gets a consistency score; flip-prone diagnoses then drive the guidelines that ALTK-Evolve injects back at inference time.
Record Trajectory
Run the agent once on a task and record the full decision trace — every prompt and every model call.
Consistency Analyzer
Replay each step via controlled resampling (k=5) against frozen context. Score every decision for flip-proneness.
Guidelines at Inference
ALTK-Evolve distills diagnoses into reusable rules, injected back at inference — same-task Pass^5 up 16.0 pts, similar-task up 13.0 pts.
Three Metrics, Three Different Questions
Pass@k asks whether at least one of k attempts succeeded — fine when outputs can be verified and retried. Pass^k demands every attempt succeed. Leaderboards often compute Mean@k with k=3 or even k=1, which does not answer what a real user cares about.
| Metric | Question It Answers | Use When | Fits Production? |
|---|---|---|---|
| Pass@k | Did at least one of k attempts succeed? | Outputs can be verified and safely retried | ~ Sometimes |
| Mean@k | What is the average pass rate over k runs? | Leaderboard-style average-case comparison | ✗ Hides variance |
| Pass^k | Did every one of k attempts succeed? | Mission-critical single-shot workflows | ✓ The right question |
What the Researchers Said
Three claims from the Hugging Face blog post that reframe how agent reliability should be measured and fixed.
“This isn’t a capability problem you fix with a bigger model. It’s an orthogonal axis: an agent can be capable and inconsistent at the same time.”
On model scale“It does not answer the question a real user cares about: will it still be good if I ask this exact question again?”
On Mean@k leaderboards“A workflow that succeeds once may fail the next time a user makes the same request — acceptable in a demo, but a showstopper for mission-critical work.”
On production deploymentWhat the Numbers Do Not Yet Show
One architecture (ReAct), one model (GPT-4.1), one benchmark (AppWorld). Whether the ~24-point gap generalizes to other frameworks and domains is untested.
The halving of the gap is the authors’ own evaluation; full methodology lives in the arXiv technical report, not peer-reviewed publication.
Whether offline resampling predicts live production flips — including real platform-side variability on hosted endpoints — is not established.
Why Repeatable Agents Matter for Production
The gap between average accuracy and repeatable success is a direct reliability problem for deployed agents. The researchers note that a workflow that succeeds once may fail the next time a user makes the same request — acceptable in a demo, but what they describe as a showstopper for mission-critical work such as reconciling a financial transaction or checking a contract for an obligation.
The finding also challenges a common assumption that reliability improves simply by using a bigger or better model. The researchers argue consistency is an orthogonal axis: an agent can be capable and inconsistent at the same time. It also explains why fixed decoding settings do not solve it — greedy decoding and a fixed seed govern how a distribution is turned into a token, not the shape of the distribution itself, and hosted endpoints shift probabilities slightly from run to run.
For anyone comparing agents on leaderboards, the practical takeaway is that Mean@k, often computed with k=3 or even k=1, does not answer the question a real user cares about: will the agent still succeed if asked the exact same question again.
Products Worth Considering
Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite
Anker SOLIX C1000 Gen 2 Portable Power Station
LG UltraGear 32GX850A-B 32-inch 4K UHD Glossy OLED Gaming Monitor
As an affiliate, we earn on qualifying purchases.
From ALTK-Evolve to Consistency Guidelines
ALTK-Evolve, introduced in an earlier Hugging Face post, is a system that turns an agent’s own past trajectories into reusable guidelines, distilled automatically and injected back at inference time. The researchers said it measurably improves task success, but those earlier evaluations asked only the average-case question, like most benchmarks.
The new consistency guidelines are built on top of that machinery as a two-stage pipeline: first the Consistency Analyzer detects flip-prone decision points in a recorded trace, then those diagnoses drive what guidelines get written. The evaluation was run on AppWorld’s test_normal split with a ReAct agent backed by GPT-4.1, five repetitions per task.
The post also draws a careful distinction between similarly named metrics: the familiar Pass@k asks whether at least one of k attempts succeeded — appropriate when outputs can be verified and retried — while Pass^k requires every attempt to succeed. The relationship Pass^k ≤ Mean@k ≤ Pass@k always holds, according to the researchers.
“Accuracy hides an unreliability problem. A ReAct agent (GPT-4.1 on AppWorld test_normal) that succeeds 77.4% of the time on average succeeds on all 5 repeated runs for only 53.0% of tasks — a 24.4-point consistency gap.”
— Hugging Face blog post
What the Numbers Do Not Yet Show
The headline results come from a single agent architecture (ReAct) with one model (GPT-4.1) on one benchmark (AppWorld). It is not yet clear how large the consistency gap is for other agent frameworks, other models, or other task domains. The roughly 30-point gap on hard tasks is reported by the researchers without a full breakdown in the post itself.
The claim that consistency guidelines halve the gap without costing average accuracy is the authors’ own evaluation; the full methodology and results are in the technical report on arXiv rather than, so far as the post indicates, peer-reviewed publication. The extent to which the Consistency Analyzer’s offline resampling faithfully predicts live production flips — including under real platform-side variability on hosted endpoints — is also not established in the announcement.
Wider Evaluation and Adoption of Pass^k
The natural next steps are replications across other agent architectures, models, and benchmarks to test whether the 24-point gap generalizes, and whether consistency guidelines deliver similar gains outside AppWorld. The researchers point readers to the arXiv technical report for full methodology and evaluations.
The post also implicitly pressures the field’s evaluation norms: if Pass^k gains traction, benchmark organizers and leaderboard maintainers may come under pressure to report consistency alongside average success — or to publish both Mean@k and Pass^k so buyers can judge whether an agent is merely good on average or reliably good on every request.
Where I land
I find the core measurement argument persuasive and overdue. The Pass^5 versus Mean@5 contrast — 53.0% versus 77.4% on the same tasks — is exactly the kind of number that deployment teams discover the hard way, and the explanation via flat versus sharp token distributions is mechanistically plausible. If you are buying or building agents for anything with real consequences, asking for Pass^k rather than Mean@k is, in my view, simply correct due diligence.
The strongest counterargument is generalization: one agent, one model, one benchmark. GPT-4.1 on AppWorld is a reasonable testbed, but agents with stronger verification loops, retry logic, or structured outputs may show much smaller gaps, which would weaken the case that a dedicated fix is needed. The claim that guidelines halve the gap at no accuracy cost also rests on the authors’ own evaluation, which I have not seen independently replicated.
What would change my assessment is straightforward: independent replications on other models and benchmarks showing comparable gaps and comparable gains from consistency guidelines — or the opposite, showing that newer models have largely closed the gap on their own. Either result would tell us whether this is a durable reliability problem or a snapshot of a specific model-generation weakness.
Key Questions
What is the consistency gap?
It is the difference between Mean@k (the average pass rate over k runs) and Pass^k (the share of tasks an agent succeeds on in all k runs). In the reported experiment the gap was 24.4 points: 77.4% average success but only 53.0% success on all five runs.
How is Pass^k different from Pass@k?
Pass@k asks whether at least one of k attempts succeeded — an optimistic measure suited to settings with verification and retries. Pass^k requires every attempt to succeed. The researchers note that Pass^k ≤ Mean@k ≤ Pass@k always holds.
Why does an agent fail a task it previously solved, even at temperature zero?
According to the researchers, some decisions have flat next-token distributions with several near-tied options. Small platform-side effects — floating-point non-associativity, request batching — can reorder near-ties, and because trajectories chain many decisions, small per-step flip chances compound into divergent runs.
What does the Consistency Analyzer need to run?
One recorded trajectory and no ground truth. It resamples each decision step against the already-recorded context with a single call requesting k completions (k=5 by default), rather than re-running the task end-to-end.
Did fixing consistency hurt average accuracy?
The researchers report it did not: consistency guidelines cut the gap from 24.4 to 12.0 points while raising same-task Pass^5 by 16.0 points and similar-task performance by 13.0 points, with no cost to average accuracy. These are the authors’ own evaluation results.
Source: Hugging Face
