CWE-bench
Hard But Fair.
Harder to game. Fairer to measure.
100 held-out audit-and-patch tasks that test frontier coding agents’ defensive cybersecurity capabilities.
120 held-out audit-and-patch tasks that test frontier coding agents’ defensive cybersecurity capabilities, graded by deterministic verifiers as well as a three-judge agentic panel.
Claude Fable 5
Gemini 3.8 Flash Cyber
GPT-5.6 Sol
CWE-bench leaderboard
| Model (high reasoning) | Harness | Pass@1 | Avg cost / rollout |
|---|---|---|---|
| Claude Fable 5 | Claude Code | 47.8% | $10.27 |
| Gemini 3.8 Flash Cyber | Antigravity | 47.2% | $3.64 |
| GPT-5.6 Sol | Codex | 44.2% | $2.29 |
| Gemini 3.7 Flash | Antigravity | 44.0% | $1.43 |
| Claude Opus 4.8 | Claude Code | 42.0% | $2.43 |
| Grok 4.6 | grok-cli | 38.2% | $1.77 |
| Qwen3.8-Max | opencode | 37.5% | $1.45 |
| Hy4 preview | opencode | 33.8% | $0.75 |
| GLM-5.3 | opencode | 31.1% | $1.86 |
| DeepSeek-V4-Flash | opencode | 30.4% | $0.13 |
| Muse Spark 1.2 | opencode | 30.3% | $2.09 |
| MiniMax M3 | opencode | 22.7% | $0.69 |
| Kimi K3 | opencode | 22.5% | $1.18 |
| NVIDIA Nemotron 3 Ultra (550B-A55B) | opencode | 18.5% | $1.17 |
| Inkling | opencode | 15.6% | $1.76 |
All scores use the deterministic programmatic verifier only (programmatic == 1): the exploit must no longer work and every pre-existing test must still pass.
How the scores are computed. A task’s score is the share of its rollouts that pass the deterministic verifier, and the leaderboard averages that across all 100 tasks. Avg cost / rollout is the mean billed API spend for one rollout on one task.
The closest public comparison is PatchEval, ByteDance’s set of 1,000 real CVEs.
While the methodologies are similar, PatchEval names the CVE and its weakness class, withholding only the reference fix. In contrast, we designed CWE-bench to mimic the real world, where nobody hands you a CVE number for a vulnerability no one has reported yet. The agent gets a repository and has to find what is wrong before it can fix it.
PatchEval is decently saturated at 83.9%, while the best known deterministic score on CWE-bench is 47.8%.
CWE-bench leaderboard
| Model | Harness | Pass@1 | Judge panel (pass@1) | Avg cost / rollout |
|---|---|---|---|---|
| Grok 4.7 | opencode | 68% | 64% | $2.75 |
| GPT-6 Astra | Codex | 68% | 66% | $2.85 |
| Claude Opus 5.5 | Claude Code | 67% | 67% | $0.79 |
| Claude Fable 5.1 | Claude Code | 58% | 60% | $3.47 |
| Grok 4.6 | opencode | 57% | 55% | $1.09 |
| DeepSeek-V4.1-Flash | opencode | 55% | 53% | $0.09 |
| Muse Spark 1.3 | opencode | 55% | 51% | $3.95 |
| Hy4 Preview | opencode | 53% | 50% | $0.47 |
| GPT-6 Sol | Codex | 52% | 52% | $0.96 |
| Inkling | opencode | 37% | 33% | $3.00 |
Every model ran at its harness’s default reasoning setting.
How the scores are computed. Each model gets four rollouts on each of the 120 tasks. A rollout that hits the time limit is not failed automatically: it is graded on the work it left, like any other. A missing rollout counts as a fail. A task’s score is the share of its four that pass, and the leaderboard averages that across all 120 tasks. Scores are rounded to the nearest point: each carries a 95% interval of roughly ±7 points, so models a point or two apart are effectively tied.
The closest public comparison is PatchEval, ByteDance’s set of 1,000 real CVEs.
While the methodologies are similar, PatchEval names the CVE and its weakness class, withholding only the reference fix. In contrast, we designed CWE-bench to mimic the real world, where nobody hands you a CVE number for a vulnerability no one has reported yet. The agent gets a repository and has to find what is wrong before it can fix it.
PatchEval is decently saturated at 83.9%, while the best score on CWE-bench v1 is 68%.
On the Pareto frontier, 79× the cost buys 1.6× the pass rate
On the Pareto frontier, 32× the cost buys 1.2× the pass rate
How CWE-bench works
CWE-bench mirrors a real security audit. Teams rarely receive a ticket naming the flaw; they receive a codebase and a reason to investigate. Each task gives the agent a checkout of a real open-source repository, some reproducing disclosed CVEs, and one instruction: audit the code and fix what you find.
Audit and fix
The task reveals neither the vulnerabilities nor how many there are, so each audit starts from a blank slate.
Focused on cyber defense
Tasks test an agent’s ability to defend real codebases; agents are never asked to create exploits.
Fix without breaking
Strict graders verify that the exploit is blocked and existing functionality still works.
No memorized fixes
Agents must reason from the code. We exclude tasks that can be solved from memorized fixes alone.
Coverage across a broad range of languages, CWEs, and OWASP categories
+49 more, mapped below ↓
- A01 Broken Access Control
- A02 Security Misconfiguration
- A03 Supply Chain Failures
- A04 Cryptographic Failures
- A05 Injection
- A06 Insecure Design
- A07 Authentication Failures
- A08 Data Integrity Failures
- A09 Logging & Alerting Failures
- A10 Exceptional Conditions
+68 more, mapped below ↓
- A01 Broken Access Control
- A02 Security Misconfiguration
- A03 Supply Chain Failures
- A04 Cryptographic Failures
- A05 Injection
- A06 Insecure Design
- A07 Authentication Failures
- A08 Data Integrity Failures
- A09 Logging & Alerting Failures
- A10 Exceptional Conditions
Distribution of CWEs across OWASP categories
One bubble per CWE, sized by how many tasks carry it and clustered into its OWASP 2025 category.
One bubble per CWE, sized by how many tasks carry it and clustered into its OWASP 2025 category.
Verifying the patch by running the exploit
Every task ships a reference patch the grader confirms passes.
| Track | How it is scored |
|---|---|
| Deterministic gate (all-or-nothing) | A programmatic exploit check confirms the exploit no longer works and all regression tests pass. Leaderboard and cost-curve metrics score this gate only. |
| Per-vulnerability partial credit | A judge scores one criterion per vulnerability plus a functionality guard, returning 0 to 1. |
Grading: a deterministic gate, with a three-judge agentic panel added in v1
Every task ships a reference patch that both the verifier and the panel confirm passes.
| Track | How it is scored |
|---|---|
| Deterministic gate | A programmatic exploit check confirms the exploit no longer works and every pre-existing test still passes; this is the score on the leaderboard and in every figure. The judge panel was used to refine these verifiers: wherever it disagreed with a probe, the probe was reviewed. |
| Agentic judge panel (added in v1) | Three rubric-based agentic judges from different model families each get a shell in a fresh copy of the patched repository and grade the fix on five axes:
|
Failure mode analysis
By weakness type: models share weaknesses across categories such as authentication (A07), error handling (A10), and data integrity (A08)
By weakness type: models share weaknesses in data integrity (A08), insecure design (A06), and authentication (A07)
By language: models are strongest on JS/TS, but weak in traditional infra code
By language: JS/TS and C/C++ lead, Java and Rust trail
In practice: where agents fail
A run can fail before a patch is written, or after a convincing patch appears complete.
Localization failure
The agent never reaches the vulnerable code.
Missed detection
The agent inspects the vulnerable code but judges it benign.
Wrong invariant
The agent patches the right code without restoring the security property.
Incomplete remediation
The agent fixes some vulnerabilities but leaves others open.
Sample task: hardening a real image decoder
BlurHash turns a short string into the blurred preview shown while a photo loads. The open source blurhash-rs library decodes that string into pixels. Those strings can arrive through feeds and APIs, so the decoder has to treat them as untrusted input.
In the vulnerable version, some malformed strings make the Rust decoder panic instead of returning an error. The task is to make those inputs fail safely while preserving the output of valid BlurHashes.
01 The task
Harden the decoder without breaking valid output
The public decode function must satisfy one invariant: every call returns either decoded pixels or a typed error that the caller can handle. No input may trigger a panic.
| Constraint | Required outcome |
|---|---|
| Valid BlurHash | Same pixel output |
| Malformed BlurHash | Typed error, no panic |
| Existing callers | Public interface still compiles |
The instruction points to string handling and the arithmetic that follows parsing. It does not reveal the failing inputs, affected lines, or required patch.
View full task
Review the decoding surface of this library and make it resilient to malformed and adversarial input. The guarantee we need is simple to state and easy to get wrong:
- For any input string, regardless of its length or characters, a decode request must return a normal, recoverable result, either a decoded image or a typed error that the caller can handle. It must never abort, unwind, or otherwise tear down the calling thread or process as a side effect of the input’s contents.
- Legitimate, well-formed placeholders must continue to decode to exactly the same pixel output as before.
Treat this as a whole-surface problem, not a single bad line. The obvious failure, inputs whose character encoding is unexpected, is only one class. Look just as hard at the arithmetic that happens after a string is accepted as well-formed: fields are parsed into numbers, and those numbers then flow into array lookups, shifts, and index math.
Excerpt, verbatim from the instruction the agent receives.
02 The verifier
Eight inputs test safety and expected behavior
After the patch is applied, the verifier compiles a small probe against the modified library and exercises the decoder with eight inputs.
| Test group | Cases | What passing looks like |
|---|---|---|
| Character boundaries | 3 | Returns an error without panicking |
| Decoded value bounds | 2 | Rejects values before an unsafe lookup |
| Normal behavior | 3 | Preserves valid output and rejects invalid input cleanly |
The character cases and the numeric cases fail on different code paths. One mishandles multibyte input before parsing; the other mishandles values produced after a string parses cleanly.
View all eight inputs
Character boundaries
"ıABCDE"two-byte character"€ABCDE"three-byte character"😀ABCDEF"four-byte emoji
Decoded value bounds
"00~~~~"parsed value above the table range"00}}}}"a second out-of-range value
Normal behavior
"LBAdAqof00WCqZj[PDay0.WB}pof"valid, decodes to 20×20"LEHV6nWB2yk8pyo0adR*.7kCMdnj"valid, decodes to 32×24"abc"invalid, returns a normal error
View verifier logic
panicked_attack = [c for c in ATTACK_CASES if results[c] == "PANIC"]
broke_valid = [c for c in NORMAL_CASES if results[c] != "OK"]
if panicked_attack or broke_valid:
sys.exit(1) # reward 0
sys.exit(0) # reward 1
# Simplified. The probe compiles against the patched crate;
# NORMAL_CASES also assert that valid hashes decode to their
# expected pixel output, not just that they avoid a panic.
03 What this task tests
A complete fix must cover the full input path
Both crashes violate the same invariant: untrusted BlurHash text must never reach an operation that can panic. Rejecting non-ASCII input prevents the UTF-8 slicing panic, but "00~~~~" still decodes to an out-of-range value used in a lookup. A complete patch validates both input characters and derived values while preserving valid pixel output.
| Patch | UTF-8 path | Numeric path | Valid output | Score |
|---|---|---|---|---|
| Character validation only | Safe | Still panics | Preserved | Fail, 0 |
| Character and range validation | Safe | Safe | Preserved | Pass, 1 |
A repeated one-path fix indicates that the model stops at the first plausible cause. Train it to state the safety invariant, trace untrusted data through every panic-capable operation, and test distinct input classes before declaring the patch complete.
A separate 1,000+ task corpus for training
CWE-bench measures generalization on 100 held-out tasks. Collinear’s training corpus is a separate 1,000+ task collection built for broader coverage, larger repositories, and multi-vulnerability remediation. Evaluation tasks never appear in training deliveries.
CWE-bench measures generalization on 120 held-out tasks. Collinear’s training corpus is a separate 1,000+ task collection built for broader coverage, larger repositories, and multi-vulnerability remediation. Evaluation tasks never appear in training deliveries.
Held-out evaluation
- 100 tasks
- 120 tasks
- 54 distinct CWEs
- 73 distinct CWEs
- 1 target weakness per task
- 8 to 11 tasks in each OWASP category
- 10 to 14 tasks in each OWASP category
- Never included in training deliveries
Training corpus
- 1,000+ tasks
- 215 distinct CWEs
- 3.28 weaknesses per task on average
- 84% of tasks contain multiple weaknesses
- 15,000 LOC median repository
- 2,000,000 LOC largest repository
The two sets serve different purposes. The training corpus gives agents broad practice finding and fully remediating compound vulnerabilities; the held-out evaluation tests whether that capability transfers to repositories the agent has never seen.
Harder Gyms for real-world defensive security work.
Request corpus access →Roadmap
Where we are taking the benchmark next.
Harder tasks, wider coverage
We are adding harder tasks that cover more CWEs, especially underrepresented ones. We are also adding tasks that every frontier model fails today, and repositories where many different kinds of vulnerability are possible.
Rubric-based scoring
We are adding rubric-based judgement. An agentic judge grades each planted bug against its own criteria, checks for defense in depth, and confirms that working features still run. Every fix earns a continuous score from 0 to 1, so a partial fix scores above no fix at all.
Rubric-based scoring (shipped in v1)
v1 adds a three-judge agentic panel on top of the deterministic gate. It verifies the weakness is closed, working features still run, and the fix was made honestly, and its verdict is reported beside the programmatic score. Next: per-vulnerability partial credit, so a partial fix scores above no fix at all.
A rotating held-out set
We will refresh the held-out set with newly disclosed vulnerabilities on a regular schedule. This keeps the benchmark hard to memorize, so scores reflect real capability rather than recalled fixes.
Multi-turn audits
We will add tasks that run over several turns. The agent can investigate, ask questions, and refine its fix, closer to how a real security review works.
Frequently asked questions
What is CWE-bench?
CWE-bench is a defensive cybersecurity benchmark from Collinear AI. It gives a coding agent a checkout of a real open-source repository and one instruction, audit the code and fix what you find. Results on a held-out set of 100 tasks across 54 CWEs were independently evaluated and published by Artificial Analysis.
CWE-bench is a defensive cybersecurity benchmark from Collinear AI. It gives a coding agent a checkout of a real open-source repository and one instruction, audit the code and fix what you find. Results on a held-out set of 120 tasks across 73 CWEs were independently evaluated and published by Artificial Analysis.
How well do AI coding agents do at finding and fixing vulnerabilities?
Not well yet. On CWE-bench the best model solves 47.8% of tasks at pass@1, and 10 of the 100 tasks are unsolved by every model tested. More than half of every model’s attempts still fail, which is the gap Collinear’s training data is built to close.
Not well yet. On CWE-bench v1 the best model solves 68% of tasks at pass@1, and 8 of the 120 tasks are unsolved by every model tested. Every model still fails at least three in ten of its attempts, which is the gap Collinear’s training data is built to close.
How are agents graded on CWE-bench?
By a deterministic programmatic verifier. A task counts as solved only when the exploit is blocked and every pre-existing test still passes. There is no partial credit and no LLM judge in the leaderboard gate, so the reward cannot be farmed.
By a deterministic programmatic verifier: a proof-of-concept exploit must no longer work and every pre-existing test must still pass, with no partial credit. v1 adds a panel of three independent LLM judges from different model families; each gets a shell in a fresh copy of the patched repository, checks the fix against a fixed checklist, and passes it only if every item holds. The majority verdict of the panel is reported beside the programmatic score, not in place of it.
Who evaluated and published the CWE-bench results?
Artificial Analysis independently evaluated and published the results. The held-out evaluation set is private to Collinear and Artificial Analysis and is not available to any external organization, which keeps the benchmark uncontaminated.
How can I improve a coding agent’s security performance?
Collinear AI offers a 1,000+ task training corpus of real audit-and-patch tasks, built to raise an agent’s ability to find and fully remediate vulnerabilities across larger repositories and multiple CWEs. Contact [email protected] to request access.
Can I download the private evaluation set?
No. The 100-task held-out evaluation set is private to Collinear and Artificial Analysis and is not available to any external organization. Keeping it unreleased is what stops it being trained against, so the scores stay uncontaminated.
No. The 120-task held-out evaluation set is private to Collinear and Artificial Analysis and is not available to any external organization. Keeping it unreleased is what stops it being trained against, so the scores stay uncontaminated.
Can I submit a model for evaluation?
Yes. Collinear runs models against the held-out set and reports the deterministic pass@1 result. Contact [email protected] to arrange an evaluation.
Yes. Collinear runs models against the held-out set and reports the deterministic pass@1 result with the judge-panel verdict alongside. Contact [email protected] to arrange an evaluation.
Can I license the separate training corpus?
Yes. The 1,000+ task training corpus of real audit-and-patch tasks is available for teams building defensive-security capability, and evaluation tasks never appear in training deliveries. Contact [email protected] to request access.
What is Collinear AI?
Collinear AI builds hard, verifiable RL environments and training data for frontier AI labs, spanning coding, computer use, and enterprise tool use. CWE-bench is its defensive-security benchmark. Learn more at collinear.ai.
Acknowledgements
CWE-bench is built and verified by Collinear AI's research and engineering team. Tasks are built on open-source projects. CWE is a classification maintained by MITRE. OWASP mappings follow the OWASP Top 10 (2025).
CWE is a trademark of The MITRE Corporation.
If you use CWE-bench in your research, please cite:
@misc{cwebench2026,
title = {CWE-bench: A Defensive Cybersecurity Benchmark for Coding Agents},
author = {{Collinear AI}},
year = {2026},
howpublished = {\url{https://cwe-bench.com}},
note = {100 held-out audit-and-patch tasks across 54 CWEs.}
}@misc{cwebench2026v1,
title = {CWE-bench v1: Harder to Game, Fairer to Measure},
author = {{Collinear AI}},
year = {2026},
howpublished = {\url{https://cwe-bench.com}},
note = {120 held-out audit-and-patch tasks across 73 CWEs.}
}