Loading…
Topic
CWE-bench
1 post about CWE-bench. Every summary links to the original.
- 1 post
- 1 company
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
CWE-bench v2: More coverage, quality, and transparency
CWE-bench v2 is a public benchmark of 100 tasks that tests AI agents on finding and fixing security weaknesses in real open-source repositories, with fully programmatic and deterministic scoring. The tasks cover 91 different CWEs, requiring models to understand multiple vulnerability surfaces rather than recognize familiar patterns, and each task includes a repository checkout, instruction, and verifier curated by LLM judge panels and hand-reviewed. Among 11 evaluated models, Gemini 4 Argon leads with 67.8% pass@1, but even the best model fails 19% of tasks, and 95 tasks are solved by at least one model. On cost, DeepSeek V4.1 Flash achieves 57.0% pass@1 at $0.04 per rollout, Claude Opus 5.5 adds ten points at $1.83, and Gemini 4 Argon edges ahead at $3.92, forming the Pareto frontier. The benchmark is open-source and can be run via Harbor, with tasks also usable as reinforcement learning environments for training defensive agents.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.