TL;DR
Hugging Face reported that 1,221 participants used coding agents to examine 2,226 ICML 2026 papers during a 19-day reproduction challenge. The project verified thousands of claims but also recorded contested findings, failed tests and many cases where missing data or code prevented a firm verdict.
A Hugging Face-led community project used coding agents to test claims across 2,226 ICML 2026 papers, producing 6,816 public reproduction logbooks in 19 days. Organizers said experiments independently verified at least one claim in 1,103 papers, while 496 papers had at least one claim classified as falsified or contested, offering a large but still provisional view of how AI-assisted research holds up when examined at conference scale.
The ICML 2026 Open Reproductions challenge ran from July 15 through August 2 and involved 1,221 community members. Participants brought tools including Claude Code, Codex, Cursor and OpenResearch’s orx to read papers, write code, run experiments and document results. The effort covered about 34% of the conference, according to Hugging Face.
Each run generated a Trackio logbook containing its methods, code, outputs and, in some cases, the agent’s execution trace. An automated judge based on the open-weights GLM-5.2 model reviewed each submission and labeled claims as verified, falsified, supported only at toy scale or inconclusive. Hugging Face said the process produced verdicts for 35,908 claims, while participants launched 2,962 cloud jobs and published 274 agent-trace datasets.
Organizers reported that 3,978 individual claims were confirmed through experiments. They classified 266 papers as fully reproduced and another 632 as partially reproduced without a falsified claim. At the other end, 49 papers had every tested claim labeled falsified, while 242 papers drew conflicting verdicts from different teams. Another 502 had only toy-scale evidence, and 280 produced no firm result, often because required artifacts were unavailable.
What We Learned By Reproducing 2,200 Papers From ICML
1,221 participants used coding agents to examine 2,226 papers in 19 days. The challenge verified thousands of claims—but contested results, failed tests and missing artifacts show why reproducibility is a process, not a binary verdict.
Research scrutiny at conference scale
From July 15 through August 2, participants used Claude Code, Codex, Cursor and OpenResearch’s orx to read papers, build implementations, run experiments and publish auditable records.
Agents widened the audit surface
Parallel coding agents let a volunteer community inspect far more work than conventional manual reproduction could cover in the same 19-day window.
Every run left a trail
Trackio logbooks captured methods, code, outputs and sometimes execution traces, making each reproduction a starting point for inspection rather than a sealed verdict.
Automation assessed automation
An open-weights GLM-5.2 judge classified extracted claims as verified, falsified, toy-scale or inconclusive—without automatically accepting the logbook’s own conclusion.
Paper-level outcomes were mixed—and categories may overlap
Bar lengths compare reported counts with the 2,226 papers examined. They are not additive percentages because the published categories do not reconcile as mutually exclusive groups.
Four labels, four different evidentiary meanings
A reproduction label describes what happened in one submitted test. It does not erase differences in datasets, implementations, compute budgets or interpretation.
| Verdict | What the run showed | What it does not prove | Best next step |
|---|---|---|---|
| ✓ Verified | Experimental evidence supported an extracted claim. | That every claim, proof or reported setting was reproduced. | Check scale, data, evaluation procedure and coverage. |
| ✗ Falsified | A submitted implementation produced evidence against a claim. | That the paper is definitively wrong. | Compare implementations and ask authors to inspect the run. |
| ~ Toy scale | The idea worked—or failed—under simplified conditions. | That the published result was recreated at reported scale. | Restore original data, checkpoints and compute where possible. |
| ~ Inconclusive | The available run could not support a firm determination. | That the claim is either valid or invalid. | Resolve missing artifacts, ambiguity or execution failures. |
Key distinction: “falsified in a reproduction attempt” is evidence for investigation—not automatic proof of research misconduct or error.
The useful product is the evidence chain
Agent-assisted verification matters most when every conclusion remains connected to inspectable inputs, code, outputs and judging criteria.
Extract a specific empirical or theoretical claim.
Interpret the method and construct a test.
Run code with available data and compute.
Record methods, code, outputs and traces.
Resolve disputes and assess scientific meaning.
Why the verdict dataset is not the final word
The challenge demonstrated reach and traceability, but it did not establish the accuracy of every reproduction—or of the automated judge itself.
The judge was not independently calibrated
The supplied material does not quantify GLM-5.2’s verdict accuracy, error rate or agreement with expert human reviewers.
Original conditions were not always matched
It remains unclear how many runs used the papers’ full datasets, checkpoints, hardware, training budgets and evaluation procedures.
Missing artifacts changed the test
When resources were unavailable, teams could use synthetic data or reduced-scale experiments—useful probes that are not full reproductions.
Published totals need reconciliation
Reported accepted-paper totals differ—6,352 versus an index of 6,341—and paper-level categories appear to overlap if treated as exclusive.
“The auditing process itself had to be auditable.”
Hugging Face organizers
Scale is the breakthrough. Judgment remains human.
Agents can expand post-publication scrutiny
They make broad, parallel checking practical when submission growth outpaces conventional review capacity.
Independent attempts reveal fragility
Opposing verdicts expose sensitivity to implementation choices, datasets, compute limits and interpretation.
Artifacts determine what can be known
Missing code, data and checkpoints often convert a decisive experiment into toy evidence or an inconclusive result.
Transparent logs are more valuable than scores
Auditable records let authors and researchers locate disagreements, rerun tests and improve the evidence.
The challenge did not settle whether 2,226 papers were right or wrong. It showed that coding agents can create a large, inspectable map of where scientific claims look strong, fragile, disputed or impossible to test.
Agents Expand Research Scrutiny
The exercise suggests that coding agents can widen the amount of post-publication checking performed on machine-learning research. ICML received 23,918 submissions and accepted more than 6,300 papers in 2026, Hugging Face reported, creating a workload that volunteer reviewers cannot fully replicate or audit before publication. Running many experiments in parallel could help identify missing artifacts, fragile results and disputed claims sooner.
The conflicting outcomes also show why an agent-produced verdict is not a final judgment. Different implementations, compute budgets, datasets and interpretations can lead teams to opposite conclusions. For researchers and readers, the logbooks may be most useful as auditable starting points for human review rather than definitive rulings on whether a paper is correct.
Coding with AI For Dummies (For Dummies: Learning Made Easy)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Submission Growth Strains Peer Review
Reproducibility concerns predate the recent growth of generative AI, but Hugging Face tied the project to a sharp increase in research output. The organization said ICML 2026 accepted roughly twice as many papers as the previous year, while review capacity did not grow at the same rate. It argued that the technology helping researchers run and describe experiments can also support large-scale verification.
Participants were encouraged to test the same paper independently. When original datasets, checkpoints or other materials were unavailable, teams could conduct toy reproductions using synthetic data. Those tests could probe whether an idea worked under simplified conditions, but they did not establish that the published result could be recreated at its reported scale.
“The auditing process itself had to be auditable.”
— Hugging Face organizers
As an affiliate, we earn on qualifying purchases.
Verdict Reliability Remains Unsettled
The results have not been described in the supplied material as a peer-reviewed study, and the verdicts came from an automated judge whose own accuracy is not quantified. It is also unclear how many reproductions matched the original papers’ full datasets, hardware, training budgets and evaluation procedures. A falsified label may reflect a genuine error, but it could also result from implementation differences or incomplete materials.
Some reported totals do not reconcile cleanly. Hugging Face cited both 6,352 accepted papers and an index of 6,341 papers. The listed paper-level categories also overlap or exceed the stated total if treated as mutually exclusive. Further documentation is needed to explain the counting rules, category overlap and how conflicting reproductions affected aggregate percentages.
As an affiliate, we earn on qualifying purchases.
Disputed Claims Await Human Review
The immediate next step is for paper authors and independent researchers to inspect the public logbooks, reproduce disputed runs and identify whether disagreements stem from the original work, unavailable artifacts or errors in the new implementations. Papers with opposing verdicts and those labeled fully falsified are likely to receive the closest scrutiny.
The larger test will be whether conferences adopt agent-assisted reproduction as part of review or post-publication checks. Any such system would need transparent judging criteria, stronger validation of automated verdicts and a clear process for authors to respond. Hugging Face’s dataset offers material for that work, but it does not by itself settle the reliability of the papers or the agents that tested them.
As an affiliate, we earn on qualifying purchases.
Key Questions
How many ICML 2026 papers were examined?
Participants attempted reproductions of 2,226 papers, which Hugging Face described as about 34% of ICML 2026.
Does a falsified verdict prove that a paper is wrong?
No. The label means a submitted reproduction produced evidence against at least one extracted claim. Missing artifacts, implementation choices and limited compute can affect the outcome, so disputed findings require human and independent review.
How were the reproduction attempts judged?
An automated judge using GLM-5.2 reviewed each logbook and assigned claim-level labels: verified, falsified, toy or inconclusive. It was instructed not to trust a logbook’s own verdict automatically.
Were all reproductions run at the original scale?
No. When datasets, checkpoints or other resources were unavailable, participants could use synthetic data and reduced-scale tests. Those results were categorized as toy evidence rather than full verification.
Are the challenge results final?
The verdict dataset was frozen when the challenge closed, but the scientific questions remain open. Authors can examine the public evidence, and further reproductions may confirm, revise or reject the initial classifications.
Source: Hugging Face
