TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
CWE-bench v2 makes 100 cybersecurity coding tasks, its grader and its evaluation harness publicly available. The tasks span 91 CWE categories; reported results show the leading evaluated model at 67.8% pass@1, while every model fails some tasks on every attempt.
Each task gives an agent a checkout of a real repository and a short instruction to find and fix a security weakness while preserving existing functionality. The benchmark authors say scoring is fully programmatic, with curated checks intended to make evaluations repeatable. Tasks are packaged as self-contained Harbor environments, and the dataset includes a registry file for running them from Hugging Face.
The release reports evaluations of 11 models. Gemini 4 Argon leads the reported results at 67.8% pass@1 and 81.0% pass@4, narrowly ahead of Claude Opus 5.5 at 67.3% and 78.0%, respectively. At least one evaluated model solved 95 of the 100 tasks; the results do not mean that any one model solved all 95.
The authors also compare reported cost and performance. DeepSeek V4.1 Flash (1017) scored 57.0% pass@1 at a reported $0.04 per rollout; Claude Opus 5.5 scored about ten percentage points higher at $1.83, and Gemini 4 Argon edged ahead at $3.92. These figures are the release’s reported comparison, and costs can depend on model pricing and evaluation setup.
CWE-bench v2 More coverage, quality, and transparency
Collinear AI has released a public benchmark for testing coding agents on security fixes in real open-source repositories. Its results enable shared comparisons, while showing that strong scores still leave important tasks unsolved.
What agents must do
Each task asks an agent to repair a weakness while keeping existing software behavior intact.
The agent receives a checkout and an instruction describing the security fix.
It must identify relevant code and make a repair.
Curated, programmatic verifiers check the fix and preserved functionality.
Reported leaders
Pass@1 measures success with one attempt; pass@4 measures success across four attempts.
| Model | Pass@1 | Pass@4 |
|---|---|---|
| Gemini 4 Argon | 67.8% | 81.0% |
| Claude Opus 5.5 | 67.3% | 78.0% |
A public, repeatable evaluation
The release makes the task set and evaluation tools available for researchers and developers to run and review.
Tasks, grader, and harness
The release provides a deterministic grader and evaluation harness for reproducible runs.
Results and feedback
Contributors can submit leaderboard results through pull requests and share task feedback.
Gated public dataset
Users must accept the dataset’s access conditions before running the public tasks with Harbor.
What the scores can tell readers
- Shared tasks support comparisons across coding agents.
- Public checks let others inspect and repeat part of the evaluation.
- The leading pass@1 score remains below 70%.
- Held-out tasks are private, and benchmark scores do not establish performance in production security work.
A Broader Test of Security Fixes
The benchmark tests whether a model can repair weaknesses across a range of categories while keeping software working. That makes it more demanding than a collection focused on one familiar vulnerability pattern: agents must interpret instructions, identify the relevant code and produce a fix that passes behavior checks.
The results also show why a high score should not be read as general readiness for security work. Even the top reported pass@1 result is below 70%, and the release says the leader fails 19% of the set on every attempt. A benchmark score can help compare systems under the same tasks, but it does not establish that an agent can safely handle every real-world repository or vulnerability.
Products Worth Considering
Logitech MX Keys S Wireless Keyboard, Graphite
DJI Osmo Mobile 7P Phone Gimbal with Mic Mini Transmitter, Infinity Black
Bose SoundLink Plus Portable Bluetooth Speaker, Wireless Outdoor Speaker, Up to 20 Hours Battery Life, Waterproof and Dustproof, Black
As an affiliate, we earn on qualifying purchases.
How the Benchmark Was Built
Version 2 keeps the prior task format and expands the publicly available set to 100 tasks. The authors say they reviewed each task by hand and checked that the instruction was realistic, that graded behavior could be inferred from it, and that the verifier assessed correct behavior rather than one specific code structure.
They also report checking that starter code fails, the reference fix passes, existing functionality remains intact, and verifiers reject memorized solutions such as known upstream patches. LLM judge panels helped shape programmatic checks for difficulty and fairness, according to the release. The benchmark authors maintain a separate held-out set to track progress on tasks not seen by evaluated models; access requires contacting Collinear AI.
Results are tracked through two Hugging Face channels: a dataset-page leaderboard for open-weight models and a leaderboard Space for broader comparisons, including API-only models. The release says contributors can submit results through pull requests. The dataset is gated, so users must accept its access conditions before running it.
“A defender has to anticipate the ways an attacker could get in, and get it right every single time.”
— Collinear AI, in its CWE-bench v2 announcement on Hugging Face
Limits of the Published Results
The announcement does not provide, in the supplied results, enough methodological detail to independently assess every comparison, including the exact model configurations and the conditions behind each cost estimate. Scores may also shift as new results are submitted. The held-out tasks are not public, so outside readers cannot directly inspect or reproduce evaluations on that set.
It is also unclear how well performance on these 100 tasks predicts outcomes in production security work. The release describes task review and verifier checks, but benchmark performance alone cannot establish how an agent behaves across the full variety of software, threat models and deployment conditions.
More Evaluations and Community Review
Researchers and developers can run models against the public set using Harbor, after accepting the dataset’s access conditions and setting up Hugging Face authentication. The release invites feedback and contributions to the tasks, and leaderboard results can be added through pull requests.
Collinear AI says interested teams can contact it to evaluate models on the held-out set. Further submissions and task feedback may change the public comparisons and help identify weaknesses in the benchmark itself.
Where I land
I see CWE-bench v2 as a useful step toward more transparent comparisons of coding agents on security repairs. Its public tasks and deterministic checks let others inspect and repeat at least part of the evaluation, while the reported results make clear that strong performance still leaves a meaningful share of tasks unsolved.
The strongest counterargument is that a curated benchmark can reward success on its own task design without showing how a model performs in live software environments. The held-out set may help track progress, but its results are not independently reproducible while those tasks remain private. I would put more weight on the benchmark as evidence of practical capability if independent teams reproduced the public scores, results remained stable across model configurations, and performance on held-out tasks tracked outcomes in real security reviews.
Source: Hugging Face
Key Questions
What is CWE-bench v2?
It is a benchmark of 100 tasks in which coding agents attempt to fix security weaknesses in real open-source repositories while preserving existing functionality.
Which model leads the reported results?
Gemini 4 Argon leads among the 11 models evaluated in the announcement, with 67.8% pass@1 and 81.0% pass@4. These are reported benchmark scores, not a guarantee of performance elsewhere.
What do pass@1 and pass@4 mean here?
The figures report success with one attempt and across four attempts, respectively. For a Harbor run configured with four attempts per task, the release says the score is the mean reward over 400 trials.
Can anyone run the benchmark?
The public set is available through Hugging Face, but it is gated: users must first accept its access conditions. The tasks can then be run with Harbor using the published repository and task registry.
Are there tasks that models have not seen?
Collinear AI says it maintains a held-out set for tracking progress on unseen tasks. Teams interested in evaluating against it are asked to contact the company; those tasks are not part of the public set.
Source: Hugging Face
