TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Ai2 researchers report that six large language models gave unstable judgments of scientific idea novelty across different evaluation setups. In one experiment, changing a few prompt words shifted accuracy by more than 50 percentage points; retrieval and extra reasoning did not consistently help. The authors released a benchmark, data, code and a pipeline, while noting limits in how some benchmark ideas were labeled.
Ai2 researchers released a study finding that six large language models can give sharply different novelty judgments about the same scientific ideas when evaluation prompts or other setup choices change. The result challenges the reliability of using LLMs to screen AI-generated research ideas, and the team has published a benchmark, data and code for further evaluation.
The study tested GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6 using both absolute novelty ratings and pairwise comparisons. In one pairwise experiment, changing a few words in the prompt shifted accuracy by more than 50 percentage points. The authors also report that adding literature retrieval or giving models more reasoning did not consistently improve their judgments.
To build one test set, the researchers mined ICLR 2026 peer reviews for explicit comments about originality. Their high-novelty group contains 154 ideas from accepted papers in the top-rated 10% of their research areas, selected where reviewers praised novelty and none disputed it. The lower-novelty group contains 145 ideas from rejected papers in the bottom 10%, selected where reviewers raised originality concerns and none praised novelty. An annotator agreed with the extracted novelty-signal direction in 47 of 50 checked cases; the authors say none of the three disagreements changed how a review was classified.
A second test setting pairs the 154 human-authored ideas with 154 ideas from a basic language-model generator. The researchers treated the generated ideas as less novel, but describe those labels as weak rather than individually verified. In a separate expert check of 30 pairs, selected heavily from cases the strongest judge got wrong, the expert preferred the human idea in every pair and judged none of the generated ideas novel. The authors also report that two purpose-built novelty systems performed worse than a simple prompted Claude Opus 4.6 baseline.
Research evaluation · Ai2 study
Can We Trust LLMs to Judge Whether a Research Idea Is Novel?
Six models produced unstable novelty judgments across evaluation setups. In one experiment, changing a few prompt words shifted pairwise accuracy by more than 50 percentage points.
6
LLMs evaluated>50
Point prompt shift299
Review-based ideas30
Expert-checked pairs01 / Why it matters
A score can reflect the setup
Research teams may use LLMs to filter proposals before literature review, experiments, or expert assessment. If small setup choices change the verdict, comparisons become difficult to interpret.
Reliability
Prompt sensitivity
A few wording changes produced a shift of more than 50 percentage points in pairwise accuracy in one experiment.
Method choices
No consistent boost
Adding literature retrieval or giving models more reasoning did not consistently improve their novelty judgments.
Scientific use
Screening needs evidence
A single chosen setup is not enough to show that an LLM can reliably assess whether an idea adds something new.
02 / Models and evaluation
Six judges, multiple ways to ask
The study compared absolute novelty ratings with pairwise judgments and varied evaluation choices such as prompts, reasoning budgets, and retrieval.
Find review signals
Mine ICLR 2026 peer reviews for explicit comments about originality.
Build test groups
Select higher- and lower-novelty human ideas, plus a generated-idea setting.
Vary the setup
Compare model judgments across formats, prompts, retrieval, and reasoning.
Check the evidence
Use annotator and expert reviews to examine parts of the benchmark labels.
03 / Benchmark construction
Useful signals, imperfect labels
The authors used reviewer comments and selection criteria as evidence of relative novelty. They caution that acceptance or rejection alone is not a sound novelty label.
Higher vs. lower novelty
Higher-novelty ideas came from accepted papers in the top-rated 10% of their research areas, where reviewers praised novelty and none disputed it. Lower-novelty ideas came from rejected papers in the bottom 10%, where reviewers raised originality concerns and none praised novelty.
Human ideas vs. model proposals
A second setting paired 154 human-authored ideas with 154 ideas from a basic language-model generator. The study treated generated ideas as less novel, while noting those labels were not individually verified.
An expert reviewed 30 pairs selected heavily from cases the strongest judge got wrong. The expert preferred the human idea in every pair and judged none of the generated ideas novel. This check does not validate every benchmark label.
04 / How to read the result
A caution, not a universal verdict
The study challenges confidence in one-shot novelty scores while leaving open where carefully designed LLM judges may still help.
What the study supports
Setup choices can change judgments
Reported instability suggests teams should test prompt and evaluation sensitivity before using scores to rank or filter research proposals.
What remains uncertain
Labels and transfer have limits
Peer reviews are not a definitive originality test. Results may differ across disciplines, idea generators, and systems with broader literature access.
Where I land
Do not treat an LLM novelty score as a stable fact about an idea. Confidence would grow if independent studies found consistent results across fields and those judgments held up against expert review and later literature evidence.
05 / Next steps
The benchmark makes comparison possible
Ai2 released the paper, benchmark data, code, and an automated pipeline intended to help researchers build more evaluation sets.
Replicate
Test across fields
Check whether the reported instability appears in disciplines beyond the benchmark’s source material.
Broaden
Vary idea sources
Compare human and generated proposals from different systems, with stronger checks of individual labels.
Improve
Seek consistency
Evaluate prompts, retrieval, and judge designs against expert review and evidence from later literature.
06 / Key questions
What to keep in view
The findings are most useful when read alongside the benchmark’s construction and stated limits.
What did the study find?
Six tested LLMs could judge the same research ideas differently when evaluation choices changed. One prompt change shifted pairwise accuracy by more than 50 percentage points.
Which models were tested?
GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5, and Claude Opus 4.6.
How were human ideas labeled?
Researchers used explicit originality comments in ICLR 2026 reviews for highly rated accepted papers and poorly rated rejected papers. These are benchmark signals, not universal measures of novelty.
Were generated ideas individually confirmed as less novel?
No. The authors describe those lower-novelty labels as weak assumptions that were not verified for every idea.
Why Unstable Novelty Scores Matter
Research teams may use LLM judges to filter candidate ideas before investing in literature review, experiments or expert assessment. If a verdict changes substantially with small prompt edits, a system’s score may reflect setup choices as much as the idea’s originality. That could make comparisons across projects or research groups difficult to interpret.
The findings also bear on efforts to automate scientific discovery. Tools that generate hypotheses or synthesize literature need a credible way to assess whether a proposal adds something new. The study does not establish that all LLM novelty evaluations fail, but it shows that accuracy on one chosen setup is not enough to demonstrate dependable judgment.
Products Worth Considering
Logitech MX Master 4 for Mac, Ergonomic Bluetooth Mouse with Haptic Feedback, Ultra-Fast Scrolling, USB-C Charging, for MacOS, iPadOS, Windo
Sony WH-1000XM6 Wireless Noise-Canceling Over-Ear Headphones, Blue, Bundle
Sonos Era 100 Wireless Smart Speaker, Black
As an affiliate, we earn on qualifying purchases.
How Ai2 Built the Benchmark
Ai2 describes its work as part of a broader effort to build systems that assist scientific discovery, including tools for exploring structured data, generating and testing hypotheses, and synthesizing research literature. In that setting, a “novelty judge” is a model asked to assess whether an idea is new relative to prior work, either by itself or against another idea.
The authors say there is no standard recipe for such judges: teams can choose different models, prompts, evaluation formats, reasoning budgets and literature-retrieval systems. They argue that judges are often tested on human-written papers even when their intended use is to assess AI-generated ideas. Their benchmark separates human ideas with reviewer evidence about originality from a second setting that includes basic model-generated proposals. The paper cautions that acceptance or rejection alone is not a sound novelty label, since papers can succeed or fail for reasons beyond originality.
““seemingly small changes to the evaluation setup produce dramatically different judgments of the same ideas.””
— Ai2 study authors
Limits of the Novelty Labels
The benchmark relies on reviewer comments and selection criteria as evidence of relative novelty; those signals do not amount to a definitive measure of originality. In the Human+Generated setting, the study assumes the basic model-generated proposals are less novel than selected human ideas, while explicitly acknowledging that the label was not verified for every idea. The expert check covered 30 pairs, not the entire set.
The supplied study description does not include the full results for every model, prompt, reasoning budget or evaluation format, nor does it specify the calendar date of release. It also does not establish how these findings would transfer to other disciplines, different idea generators, or evaluation systems with broader literature access. The authors say some conclusions come from disagreement between evaluation setups and do not depend on benchmark labels, but the extent of that result across settings is not detailed in the provided material.
Benchmark Enables Further Testing
Ai2 has released the paper, benchmark data, code and an automated pipeline intended to help researchers build additional evaluation sets. The next step is for other teams to test whether the reported instability holds across fields, idea sources and judge designs, and to assess methods that could make judgments more consistent.
The release description does not announce a specific follow-up study or timeline. For now, the benchmark offers researchers a way to compare evaluation choices, while the study leaves open what combination of prompts, retrieval and model design, if any, can support dependable novelty assessment.
Where I land
I read this as a warning against treating an LLM novelty score as a stable fact about a research idea. The reported prompt sensitivity is large enough to make evaluation choices part of the result, and that matters if labs use these scores to rank or filter proposals.
The strongest counterargument is that the benchmark’s labels are imperfect: peer reviews are not a definitive originality test, and the human-versus-generated comparison uses weak labels. A judge could also be useful in a narrower workflow even if it is not reliable as a stand-alone evaluator. I would change my assessment if independent studies across disciplines found that carefully specified protocols produce consistent judgments, and if those judgments held up against expert review and later literature evidence.
Key Questions
What did the Ai2 study find?
It found that six tested LLMs could judge the same research ideas differently when evaluation choices changed. In one experiment, a small prompt change shifted pairwise accuracy by more than 50 percentage points.
Which models were tested?
The study tested GPT-5.1, GPT-5.2, GPT-5.4, Claude Sonnet 4.5, Claude Opus 4.5 and Claude Opus 4.6.
How did the researchers identify higher- and lower-novelty ideas?
They used explicit originality comments in ICLR 2026 peer reviews to select ideas from highly rated accepted papers and poorly rated rejected papers. The authors caution that these signals are evidence for the benchmark, not a universal measure of novelty.
Were the AI-generated ideas individually confirmed to be less novel?
No. The authors describe the lower-novelty labels for generated ideas as weak assumptions, rather than verified judgments for every idea. An expert reviewed 30 selected pairs and preferred the human-authored idea in each, but that check did not cover the full set.
Source: Hugging Face
