How CBRN Risk Evaluations Get Built for Frontier Models
A five-stage walkthrough of how CBRN risk evaluations are commissioned, designed, graded, and converted into safety-level decisions for frontier AI models.
Hugging FaceDavid CorvoysierAnnouncement
The torch-neuronx model support Space on Hugging Face checks whether Transformer models run unmodified on AWS Trainium accelerators. It offers three views: Selected (curated top models), Trending, and Downloads, each showing the result, number of Trainium2 cores, wall time including compilation, and peak device memory. Users can look up a specific model by typing its name and select the Neuron compiler (neuronx-cc) and Transformers library versions to inspect compatibility. Each test loads the model on a trn2 instance, shards with tensor or expert parallelism, generates 32 tokens, and checks fluency against a CPU run or judge model. The underlying dataset is publicly available as Parquet files in the aws-neuron/torch-neuronx-model-support dataset, enabling direct querying with Pandas to retrieve results.
AWS Trainium users need a way to determine which Hugging Face Transformers models run unmodified on the accelerator. The torch-neuronx model support Space was created to answer that question by testing popular models and providing a searchable interface.
A Hugging Face Space tests each model on a trn2 instance, shards weights using tensor parallelism (or expert parallelism for mixture-of-experts models), generates 32 tokens with static cache and torch.compile(backend=‘neuron’), checks output fluency against a CPU run or judge model, and stores results as Parquet files in a public dataset.
The Space lets users search any Hugging Face Transformers model by name and see whether it runs unmodified on AWS Trainium, along with the number of Trainium2 cores used, wall time, and peak memory.
Each test loads the model on a trn2 instance, shards it with tensor or expert parallelism, generates 32 tokens, and validates fluency using a CPU comparison for single-core runs or a judge model for sharded runs.
All test results and run scripts are available as Parquet files in the aws-neuron/torch-neuronx-model-support dataset, which can be queried directly with Pandas.
Switch to the Trending or Downloads view and type part of the model's name. The Space follows Hub rankings and shows whether the model was tested or left out, with reasons.
A PASS means the model loaded on a trn2 instance, sharded over the smallest number of Trainium2 cores that fit its weights, generated 32 tokens, and produced fluent English output compared to a CPU run or judged by a small model.
Yes, the Space provides the exact Neuron stack version, packages to install, and a run.py script. It also offers a command to sync the NEFF cache from a public bucket to skip compilation.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
A five-stage walkthrough of how CBRN risk evaluations are commissioned, designed, graded, and converted into safety-level decisions for frontier AI models.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.