CWE-bench v2 is a public benchmark of 100 tasks that tests AI agents on finding and fixing security weaknesses in real open-source repositories, with fully programmatic and deterministic scoring. The tasks cover 91 different CWEs, requiring models to understand multiple vulnerability surfaces rather than recognize familiar patterns, and each task includes a repository checkout, instruction, and verifier curated by LLM judge panels and hand-reviewed. Among 11 evaluated models, Gemini 4 Argon leads with 67.8% pass@1, but even the best model fails 19% of tasks, and 95 tasks are solved by at least one model. On cost, DeepSeek V4.1 Flash achieves 57.0% pass@1 at $0.04 per rollout, Claude Opus 5.5 adds ten points at $1.83, and Gemini 4 Argon edges ahead at $3.92, forming the Pareto frontier. The benchmark is open-source and can be run via Harbor, with tasks also usable as reinforcement learning environments for training defensive agents.
Context
CWE-bench v2 is a public benchmark for evaluating AI agents on cybersecurity vulnerability detection and repair in open-source repositories. It provides 100 tasks covering 91 CWEs, with programmatic scoring and a held-out set for unbiased evaluation.
Approach / What changed
The benchmark uses Harbor to run tasks as Dockerized environments with deterministic verifiers. Each task includes a repository checkout, instruction, and programmatic checks curated by LLM judge panels and hand-reviewed. Scoring is pass@1 and pass@4 over multiple attempts.
Takeaways
1
CWE-bench v2 covers 91 different CWEs across 100 tasks, requiring models to understand multiple vulnerability surfaces rather than recognize familiar patterns.
2
Gemini 4 Argon achieves the highest pass@1 at 67.8%, but even the best model fails 19% of tasks, indicating the long tail remains unsolved.
3
The benchmark is open-source and can be run via Harbor, with tasks also usable as reinforcement learning environments for training defensive agents.
FAQ
How is scoring done in CWE-bench v2?
Scoring is fully programmatic and deterministic, with pass@1 and pass@4 metrics. Each task has a verifier that checks for correct behavior, not specific fix structure, and results are reproducible.
How can I evaluate my model on the held-out set?
To evaluate on the held-out set, contact [email protected]. The held-out set is maintained to track progress on tasks no model has seen.
What is the cost-performance trade-off among top models?
DeepSeek V4.1 Flash achieves 57.0% pass@1 at $0.04 per rollout, Claude Opus 5.5 adds ten points at $1.83, and Gemini 4 Argon edges ahead at $3.92, forming the Pareto frontier.
A five-checkpoint loop for chip-design teams: pin reference artifacts, trace multistage agent runs, refine tool descriptions, and re-run a composite benchmark so outputs are verified, not merely plausible.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.