---
title: "CWE-bench v2: More coverage, quality, and transparency"
description: "CWE-bench v2 is a public benchmark of 100 tasks that tests AI agents on finding and fixing security weaknesses in real open-source repositories, with fully programmatic and deterministic scoring. The tasks cover 91 different CWEs, requiring models to understand multiple vulnerability surfaces rather than recognize familiar patterns, and each task includes a repository checkout, instruction, and verifier curated by LLM judge panels and hand-reviewed. Among 11 evaluated models, Gemini 4 Argon leads with 67.8% pass@1, but even the best model fails 19% of tasks, and 95 tasks are solved by at least one model. On cost, DeepSeek V4.1 Flash achieves 57.0% pass@1 at $0.04 per rollout, Claude Opus 5.5 adds ten points at $1.83, and Gemini 4 Argon edges ahead at $3.92, forming the Pareto frontier. The benchmark is open-source and can be run via Harbor, with tasks also usable as reinforcement learning environments for training defensive agents."
---

# CWE-bench v2: More coverage, quality, and transparency

[Hugging Face](https://yomu.fyi/company/hugging-face) · Gonzalo Gonzalez, Ajay Jagannath, Nazneen Rajani, Nathan Habib · Nov 3, 2026

**Type:** Announcement

## Summary

CWE-bench v2 is a public benchmark of 100 tasks that tests AI agents on finding and fixing security weaknesses in real open-source repositories, with fully programmatic and deterministic scoring. The tasks cover 91 different CWEs, requiring models to understand multiple vulnerability surfaces rather than recognize familiar patterns, and each task includes a repository checkout, instruction, and verifier curated by LLM judge panels and hand-reviewed. Among 11 evaluated models, Gemini 4 Argon leads with 67.8% pass@1, but even the best model fails 19% of tasks, and 95 tasks are solved by at least one model. On cost, DeepSeek V4.1 Flash achieves 57.0% pass@1 at $0.04 per rollout, Claude Opus 5.5 adds ten points at $1.83, and Gemini 4 Argon edges ahead at $3.92, forming the Pareto frontier. The benchmark is open-source and can be run via Harbor, with tasks also usable as reinforcement learning environments for training defensive agents.

## Context

CWE-bench v2 is a public benchmark for evaluating AI agents on cybersecurity vulnerability detection and repair in open-source repositories. It provides 100 tasks covering 91 CWEs, with programmatic scoring and a held-out set for unbiased evaluation.

## Approach / What changed

The benchmark uses Harbor to run tasks as Dockerized environments with deterministic verifiers. Each task includes a repository checkout, instruction, and programmatic checks curated by LLM judge panels and hand-reviewed. Scoring is pass@1 and pass@4 over multiple attempts.

## Takeaways

- CWE-bench v2 covers 91 different CWEs across 100 tasks, requiring models to understand multiple vulnerability surfaces rather than recognize familiar patterns.
- Gemini 4 Argon achieves the highest pass@1 at 67.8%, but even the best model fails 19% of tasks, indicating the long tail remains unsolved.
- The benchmark is open-source and can be run via Harbor, with tasks also usable as reinforcement learning environments for training defensive agents.

## FAQ

### How is scoring done in CWE-bench v2?

Scoring is fully programmatic and deterministic, with pass@1 and pass@4 metrics. Each task has a verifier that checks for correct behavior, not specific fix structure, and results are reproducible.

### How can I evaluate my model on the held-out set?

To evaluate on the held-out set, contact cyber@collinear.ai. The held-out set is maintained to track progress on tasks no model has seen.

### What is the cost-performance trade-off among top models?

DeepSeek V4.1 Flash achieves 57.0% pass@1 at $0.04 per rollout, Claude Opus 5.5 adds ten points at $1.83, and Gemini 4 Argon edges ahead at $3.92, forming the Pareto frontier.

**Tags:** [AI](https://yomu.fyi/topic/ai), [Benchmark](https://yomu.fyi/topic/benchmark), [CWE-bench](https://yomu.fyi/topic/cwe-bench), [Cybersecurity](https://yomu.fyi/topic/cybersecurity), [LLMs](https://yomu.fyi/topic/llm)

- Source: [Hugging Face](https://huggingface.co/blog/cwe-bench)
- Source URL: https://huggingface.co/blog/cwe-bench
- Ingested by Yomu: 2026-11-03T18:01:09.497Z

[Read original post](https://huggingface.co/blog/cwe-bench)
