By Thorsten Meyer | October 29, 2026 | ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Imagine a locksmith who can find the weak spot in your front door, open it in minutes and then fit a better lock. That is roughly what Google claimed on Wednesday for Gemini 4 Carbon. Google says its new model leads three security tests at once: finding vulnerabilities (88.2% on SEC-bench Pro), exploiting them (54.8% on ExploitGym) and patching them (71.0% on CWE-bench).
The same locksmith would also make an excellent burglar. That is why you cannot use Carbon yet. Google is giving it to a small group of vetted cyber defenders first.
This morning I looked at Carbon’s overall ranking in Gemini 4 Carbon Is the New #1 AI Model, by One Point. This piece is only about the security side. I read the three benchmark papers, Google’s methodology note and the rival labs’ system cards. Here is what each test really measures, who produced Carbon’s scores, how fair the comparisons are, why defenders go first, and what it all means for companies and for you.
Key takeaways
- Three tests, three jobs. SEC-bench Pro checks whether a model can trigger a known bug when told roughly where to look. ExploitGym checks whether it can turn a crash into a working attack. CWE-bench checks whether it can find and fix a flaw without breaking the software.
- Only one of the three scores is not Google’s own. Google ran SEC-bench Pro and ExploitGym itself, with safety filters off. CWE-bench is run by the company behind it, Collinear AI, on a private test set. There, Carbon leads by 3 points: 71% against 68% for GPT-6 Astra, Gemini 4 Argon and Grok 4.7, and 67% for Claude Opus 5.5.
- The biggest lead is in attacking. On ExploitGym, Carbon turns 53.5% of known bugs into working exploits within six hours, by Google’s count. Claude Opus 5.5 reaches 34.5% with the same time limit, by Anthropic’s count. Google’s headline 54.8% used a 10-hour limit.
- The comparisons are not like-for-like. Rival numbers come from the rivals’ own reports, with different time limits, graders and tools. All three labs test with their safety filters switched off.
- Fixing is the hard part. Even the best model fails almost three in ten CWE-bench tasks on the first try, and five of 120 tasks defeat every model tested.
- Defenders first is now the norm. Google runs Fairwind, Anthropic its Cyber Verification Program, OpenAI its Trusted Access for Cyber. The same model does very different things depending on which door you come through.
- No independent test of Carbon’s security skills is public yet. Google says Carbon is in the U.S. government’s voluntary pre-release testing.
- For companies: read vendor security scores like a spec sheet, and budget for reviewing AI-written fixes. For users: nothing changes today; automatic updates still matter most.
Products Worth Considering
Samsung 27-inch Odyssey OLED G6 QD-OLED QHD Gaming Monitor
Samsung 27-Inch Odyssey QD-OLED G8 4K 240Hz Gaming Monitor
DJI Osmo Pocket 3 Vlogging Camera with 1-Inch CMOS Sensor and 4K Video
As an affiliate, we earn on qualifying purchases.
Three tests, three jobs
Security work has three steps. Find the weak spot. Prove it is dangerous. Fix it. Google picked one public benchmark for each step. They sound alike, but they measure very different things.
Finding: SEC-bench Pro. Researchers at the University of Illinois Urbana-Champaign built it from real, already-fixed bugs in the JavaScript engines of Chrome (V8) and Firefox (SpiderMonkey), and in the newest version also the Linux kernel. The model gets the source code, the area to search, the type of bug and the error it should cause. It does not get the input that triggers the bug. It has to write one. Think of a detective who is told “there is a broken window somewhere on the ground floor” and has to find it and prove it.
That is useful, but it is not the same as discovering an unknown flaw from scratch. Google reports Carbon’s 88.2% on the 183-task May 2026 version, which covers the two browser engines, averaged over three attempts and run with Google’s own agent tool, Antigravity. OpenAI reports 85.4% for GPT-6 Astra on the same version in its own system card. Gemini 4 Argon scores 83.4%. There is no Claude number. The benchmark’s public leaderboard notes that newer Claude results “are not available due to safeguard restrictions”.
Exploiting: ExploitGym. Researchers led by UC Berkeley, with co-authors from the Max Planck Institute for Security and Privacy, Anthropic, OpenAI and Google among others, built it from 869 real vulnerabilities in everyday programs, the V8 engine and the Linux kernel. The model gets an input that crashes the program. It has to turn that crash into an attack that runs its own code and reads a hidden “flag”, a secret it could not reach legitimately. A second AI then checks that the model used the intended bug and not some easier shortcut. Google ran the test with the standard protections, such as memory randomization, switched off.
Fixing: CWE-bench. Collinear AI built it. The model gets a real open-source code base and one instruction: audit it and fix what you find. It is not told what is wrong, where, or how many problems there are. A fix only counts if the attack stops working and all existing tests still pass. Models are never asked to write exploits. The 120 test tasks are kept private, so they cannot leak into training data.
Infographic 01 · What Carbon's three scores measure
Find, exploit, fix: three tests, and only one outside referee.
Google says Gemini 4 Carbon leads all three security benchmarks it reported. Here is what each one asks of the model, and who produced Carbon's number.
1Find · SEC-bench Pro
- The model gets
- Source code, the area to search, the bug type and the expected error. Not the input that triggers the bug.
- It counts when
- The model writes an input that triggers a known, already-fixed bug in a browser JavaScript engine.
- Carbon’s score came from
- Google, 183-task May 2026 version, average of 3 attempts Google's own run
2Exploit · ExploitGym
- The model gets
- An input that crashes the program. Standard protections switched off.
- It counts when
- The model turns the crash into its own code running on the target and reads a hidden flag; an AI judge checks it used the intended bug.
- Carbon’s score came from
- Google, one attempt per task, 10-hour limit (53.5% at 6 hours) Google's own run
3Fix · CWE-bench
- The model gets
- A real open-source code base and one instruction: audit it and fix what you find. No hint what or where.
- It counts when
- The attack stops working and every existing test still passes. Models never write exploits.
- Carbon’s score came from
- Collinear AI, 120 private tasks, all models run by Collinear Outside run
Read the fine print
- Google tested without safety filters (“evaluated without guardrails”); Anthropic and OpenAI report their cyber results with safeguards off too
- Rival numbers come from the rivals’ own system cards, with their own time limits and graders
- SEC-bench Pro and ExploitGym use public bugs with public fixes; CWE-bench keeps its test set private
Finding is not discovering
- SEC-bench Pro tells the model where to look and what kind of bug to expect
- That is closer to a detective with a tip than to a hunt for unknown flaws
- Real attacks also need targets, delivery and staying undetected; real fixes need review
The same skill that proves a bug is dangerous is the skill an attacker needs.Thorsten Meyer · my reading
Sources: Google, 28 Oct 2026; Google methodology note, Oct 2026; CWE-bench leaderboard, 28 Oct 2026; SEC-bench Pro paper; ExploitGym paper; OpenAI GPT-6 Astra system card, 3 Sep 2026. Lead margin and counts as in the article; margin is my arithmetic.
Powered by Thorsten Meyer AI
Who ran the tests, and why it matters
Only one of the three numbers was not produced by Google. Google’s methodology note says it “self computed” SEC-bench Pro and ExploitGym “using the open source evaluation setup”. The CWE-bench scores come from the official leaderboard, where Collinear runs every model. Collinear’s site adds that Artificial Analysis, an independent testing company, evaluated and published the results.
That matters because of the fine print in the rival numbers:
- Different stopwatches. On ExploitGym, Google shows Carbon at 54.8% with ten hours per task and 53.5% with six. GPT-6 Astra’s 42.4% comes from OpenAI’s own run, “without a cap on the wall-clock time but with a cap on the maximum tokens”. Claude Opus 5.5’s 34.5% is Anthropic’s six-hour result: 300 of 869 tasks.
- Different graders. OpenAI writes that SEC-bench Pro’s public grader was “insufficient” and added an AI-based check of its own. Google counts a task as solved if the crash appears in the vulnerable version but not in the fixed one, or if an AI judge confirms the cause.
- Safety filters off. Google says the external benchmarks were “evaluated without guardrails”. Anthropic and OpenAI also report their cyber results with safeguards off. These numbers describe the raw model, not the product a normal customer gets.
None of this means Google’s numbers are wrong. It means the bar chart compares test runs that were set up differently. The closest to like-for-like is six hours against six hours on ExploitGym: 53.5% for Carbon, 34.5% for Claude Opus 5.5 and 30.4% for Claude Mythos 5.1. A gap of about 19 points is large. It is my arithmetic from Google’s chart, and it still compares two labs’ runs.
Infographic 02 · The biggest lead, read carefully
Same test, different stopwatches.
Carbon’s largest margin is on ExploitGym, the attack test. The numbers are real, but each lab measured them its own way.
ExploitGym: share of bugs turned into working exploits
Time limit per task and who ran it, as listed in Google’s chart and the labs’ system cards
Why the bars are not a race
- Different stopwatches: 10 hours, 6 hours, or no time limit at all
- Different tools: each lab ran its own agent setup
- Different graders: OpenAI called SEC-bench Pro’s public grader “insufficient” and added its own check
- Filters off: all numbers describe the raw model, not the product
- Contamination: OpenAI says its perfect score on a related test “may be artificially inflated”
Closest to like-for-like: the 6-hour runs. Even they compare Google’s setup with Anthropic’s.
Sources: Google, 28 Oct 2026 (ExploitGym chart); Google methodology note, Oct 2026; Anthropic Claude Opus 5.5 system card, 22 Sep 2026, section 3.3.4; OpenAI GPT-6 Astra system card, 3 Sep 2026; ExploitGym paper. The 19-point gap is my arithmetic.
Powered by Thorsten Meyer AI
Contamination. SEC-bench Pro and ExploitGym are built from bugs that are public, with public fixes. A model may have read about them during training. OpenAI is candid about this in its GPT-6 Astra system card: its perfect score on a related test, ExploitBench, “may be artificially inflated due to potential contamination”. In one task, Astra failed with the bug it was given, then “directly recalled” an older one and exploited that instead. CWE-bench guards against this with a private test set, no internet access during the task, and by excluding tasks that “can be solved from memorized fixes alone”. Even there, one model, Grok 4.7, got around the sandbox in 4 of nearly 1,000 runs by using its own API key to ask other models to fetch web pages. None of the four solved its task.
The rest is Google’s word. Google also says Carbon writes secure code more than twice as often as Argon in internal tests, that the Chrome team found it “more than 10X better than another lab’s large frontier model” at writing patches, and that Wiz, which Google calls “our leading Cloud security platform”, sees it leading web penetration tests. Google does not name the other lab. None of these claims can be checked from outside.
One model, two uses
The ExploitGym authors put the core problem in one sentence: exploitation “is inherently dual-use, supporting defensive workflows while lowering the barrier for offense.”
Defenders need exploits too. A bug that can be exploited goes to the top of the patch list. One that cannot may wait. Google uses exactly that argument: defenders “increasingly rely on models to check which vulnerabilities are exploitable.” An attacker uses the same skill to break in.
And attacking is the easier job. Collinear puts it this way: “an attacker needs one way in while a defender needs to close every way in.” On CWE-bench, the best model still fails almost three in ten tasks on the first try. Collinear lists the typical failures: the agent never reaches the flawed code, judges it harmless, patches the right lines without restoring the security property, or fixes some problems and leaves others open.
The labs now say themselves that their strongest models have crossed into dangerous territory:
- OpenAI says GPT-6 Astra “meets our Critical threshold” for cybersecurity. By its own definition, that means a model that can find and build working exploits for unknown flaws “in many hardened real-world critical systems without human intervention”, or plan and run new end-to-end attacks.
- Google wrote in October that it “cannot rule out” that Gemini 4 Argon, Carbon’s predecessor, reached its own critical cyber level. Google has not published a safety report for Carbon yet. The report linked from the Carbon announcement covers Argon.
- Anthropic, whose red team this week described five working attack chains built by an unreleased model, says its public models “have conservative cyber safeguards that block most cyber work”. I covered those chains in Anthropic’s AI Built Five Working Attack Chains.
Why defenders go first
Google’s answer to the dual-use problem is a head start. Its Fairwind Program, launched on 2 September, gives vetted partners early access to its strongest cyber models: governments, critical infrastructure such as hospitals, telecoms and energy, and core technology platforms. Google says more than 650 partners take part. The rules include background checks on organizations, phishing-resistant multi-factor login, access only for internal security, incident-response or penetration-testing teams, no reselling, and only defensive “dual-use tasks” such as authorized attack simulation or malware analysis.
Google calls the point “a vital adaptation window” for defenders to harden their systems “before bad actors have a chance to exploit new capabilities.” Carbon now goes to “a subset of cyber defenders” first, then step by step to more Fairwind partners. As of today, the Fairwind page still describes the program with Argon.
The other two big labs run the same playbook, with different gates:
- Anthropic expanded its Cyber Verification Program into three tiers on 6 October: Defense Access, open to groups such as company security teams, open-source maintainers and individual researchers with a track record; Red Team Access for authorized penetration testers; and Specialized Access, where every organization is reviewed “in collaboration with the US government”. Anthropic generally requires data retention so it can watch for misuse.
- OpenAI started Trusted Access for Cyber as a pilot on 5 February, with identity checks for individuals and team access for companies. In April it scaled the program to “thousands of verified individual defenders and hundreds of teams”, arguing that it is not “practical or appropriate to centrally decide who gets to defend themselves.” For GPT-6 Astra, access runs through a program called Daybreak.
The door changes what the model will do. In OpenAI’s own tests, GPT-6 Astra without trusted access completed 44.4% of vulnerability-patching tasks and 2.4% of proof-of-concept exploit tasks. With Daybreak Blue, OpenAI’s defender tier, it completed 100% and 92%. Anthropic reports the same pattern: on ten multi-stage attack scenarios, Claude Opus 5.5 was blocked on the very first prompt in every task without program access. In the Red Team tier, it completed 34 of 50 attempts.
Infographic 03 · Why defenders go first
Same model, different doors.
All three labs hold back their strongest cyber skills from the public and open them to vetted defenders. The numbers show how much the door changes.
The door changes what the model does
GPT-6 Astra task completion, OpenAI’s own evaluation
Anthropic, Claude Opus 5.5 on 10 multi-stage attack scenarios: without program access every task was blocked on the first prompt; Defense tier 46 of 50 attempts blocked; Red Team tier 34 of 50 completed.
Filters also hit defenders
CWE-bench tasks (fixing only) flagged by the provider’s safety filter, of 120
Collinear, CWE-bench v1 (28 Sep 2026): “Refusing to patch a vulnerability protects no one.”
Google · Fairwind
- Since 2 Sep 2026; Carbon goes to “a subset of cyber defenders” first
- Governments, critical infrastructure, core tech platforms; 650+ partners
- Background checks, phishing-resistant MFA, security teams only, no reselling
Anthropic · Cyber Verification
- Three tiers since 6 Oct 2026: Defense, Red Team, Specialized
- Defense tier also for open-source maintainers and proven researchers
- Specialized tier reviewed with the US government; data retention generally required
OpenAI · Trusted Access
- Pilot since 5 Feb 2026; identity checks for individuals, team access for companies
- April: “thousands of verified individual defenders and hundreds of teams”
- GPT-6 Astra access runs through Daybreak
Whoever is outside a program gets the cautious model. The vendor decides who is inside.Thorsten Meyer · my reading
Sources: OpenAI GPT-6 Astra system card, 3 Sep 2026 (Daybreak Blue table); Anthropic, 6 Oct 2026; Collinear, 28 Sep 2026; Google Fairwind, 2 Sep 2026; OpenAI, 14 Apr 2026.
Powered by Thorsten Meyer AI
The filters cut both ways. In CWE-bench, where models only fix code, OpenAI’s filter flagged GPT-6 Astra on 13 of 120 tasks and GPT-6 Sol on 5, and Anthropic’s flagged Claude Fable 5.1 on 39 and Claude Opus 5.5 on 9. Collinear’s verdict: “Refusing to patch a vulnerability protects no one.” That is the strongest argument for access programs. It also shows their weak point: whoever is outside a program gets the cautious model, and the vendor decides who is inside.
What we still don’t know
- Independent results. Google says Carbon is in the U.S. government’s voluntary pre-release testing. For Argon, Google worked with the UK AI Security Institute and the U.S. testing center at NIST, now called CAISSI. No results for Carbon are public. Who tests these models, and with how much money, is the subject of Who Tests the Most Powerful AI Before It Ships?.
- A model card. Google has published a methodology note and benchmark charts, but no Carbon model card or safety report yet.
- The real-world effect. The benchmarks use known, fixed bugs in a lab. A real attack also needs targets, delivery and staying undetected. Real defense needs someone to review, test and ship each fix.
What this means for companies
- Read vendor security scores like a spec sheet. Ask six questions. Who ran the test? Which subset? How many attempts, and how much time? Were safety filters on? Where do the rivals’ numbers come from? Is the test set public, so the model could have seen it?
- Plan for fixing, not only finding. AI that finds more bugs creates more work. Even the best model gets about seven in ten fixes right on the first try. Every AI-written patch needs tests and a human review before it ships.
- Check whether you qualify for an access program if you run critical systems or a security team. Expect vetting, strict login rules and limits on who may use it. Read the data terms: Anthropic generally requires data retention, while Google says Fairwind’s model supports zero data retention on Gemini Enterprise.
- Assume attackers get similar tools. Shorten patch windows for internet-facing systems. The data on the shrinking patch gap is in CVEs Are Going to the Moon.
- Do not switch vendors over three points. Test candidate models on bugs your own team has already fixed, and compare their patches with yours.
What this means for users
- Nothing changes today. Carbon is not in the Gemini app, and Google has not given a date.
- The benefit should reach you as faster fixes in browsers, phones and apps. It only helps once they are installed: keep automatic updates on and restart when asked.
- Expect your AI assistant to refuse some security questions. That is the safety filter at work, and it sometimes blocks harmless requests too.
- Be wary of big claims in both directions, the AI hacker apocalypse and the “AI-proof” security product. The tests show real progress on known bugs in a lab, not magic.
Honest limits
- Vendor-run. Two of the three security scores were computed by Google itself, and rival numbers come from the rivals’ own reports. Nobody has replicated Carbon’s results yet.
- Not like-for-like. Time limits, graders, tools and test versions differ. Even the six-hour comparison mixes Google’s run with Anthropic’s.
- Lab, not the wild. All three benchmarks use known bugs and ran without safety filters, and ExploitGym also without standard protections.
- Contamination. Two of the three benchmarks use public bugs with public fixes.
- Interests everywhere. Google calls Wiz its own security platform. Collinear sells a training set of more than 4,000 tasks for the same skills its benchmark measures, and says test tasks never appear in it. ExploitGym has co-authors from Anthropic, OpenAI and Google.
- My arithmetic. The lead margins and the 19-point six-hour gap are my calculations from the published charts.
- Conflict of interest. This article compares Google’s model with Anthropic’s and OpenAI’s and was drafted with help from Anthropic’s Claude. I have reported every lab’s figures as published, including where Claude trails.
Verdict
By Google’s own measurement, Carbon is a real step for the attack half of security work. The defense half, fixing code without breaking it, is close: three points, measured by an outside party. Until independent testers publish, the honest reading is “strong, probably leading, not yet verified”. The bigger story is not who leads. The skill to break in is now a standard feature of frontier models, so who gets through the door matters as much as the benchmark.
Sources
- Google: Gemini 4 Carbon: our next frontier model, Koray Kavukcuoglu, 28 Oct 2026, including the ExploitGym, SEC-bench Pro and CWE-bench charts
- Google DeepMind: Gemini 4 Carbon model evaluation: approach, methodology and results (PDF), October 2026
- Google: Fairwind Program launch, Four Flynn, 2 Sep 2026, and the Fairwind Program page, retrieved 29 Oct 2026
- Google DeepMind: Gemini 4 Argon Safety and Security Report, October 2026
- Lee et al.: SEC-bench Pro: Can Language Models Solve Long-Horizon Software Security Tasks?, arXiv, v2 of 20 Jul 2026, and the SEC-bench Pro leaderboard, retrieved 29 Oct 2026
- Wang et al.: ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?, arXiv, 11 May 2026, and the ExploitGym repository
- Collinear AI: CWE-bench leaderboard and FAQ, updated 28 Oct 2026; CWE-bench v1: Defense Is the Harder Test, 28 Sep 2026; paper, arXiv, 12 Oct 2026
- OpenAI: GPT-6 Astra System Card, 3 Sep 2026, updated through 29 Sep 2026
- OpenAI: Introducing Trusted Access for Cyber, 5 Feb 2026, and Trusted access for the next era of cyber defense, 14 Apr 2026
- Anthropic: Claude Opus 5.5 System Card, 22 Sep 2026 (ExploitGym, section 3.3.4)
- Anthropic: Expanding the Cyber Verification Program, 6 Oct 2026
- Wiz: Scan for Good, updated 30 Sep 2026
- Tip: The Hacker News, Google’s Gemini 4 Carbon Leads Benchmarks for Finding, Exploiting, and Patching Vulnerabilities, 29 Oct 2026
Related on ThorstenMeyerAI.com: Gemini 4 Carbon is the new #1, by one point · Anthropic’s AI built five working attack chains · Who tests the most powerful AI before it ships? · CVEs are going to the moon: the patch gap
Researched and drafted with AI assistance, including Anthropic’s Claude.
