AWSRahul Gupta, Jwala DhamalaAnnouncement
Amazon partnered with AVERI to conduct a third-party review of its internal safety evaluation datasets for frontier AI models, covering harmful-content, cybersecurity, privacy, and copyright risks. AVERI’s team, led by Miles Brundage, independently examined a random sample of prompts for diversity, difficulty calibration, and methodological soundness without input from Amazon’s evaluation team. The review validated comprehensive topical coverage, diverse writing styles and conversation types, and low contamination risk from public datasets. It also identified opportunities to improve, such as expanding linguistic diversity across all risk categories, adding more multistep adversarial prompts to simulate sophisticated misuse, and maintaining evaluation freshness by removing outdated references and clarifying grading criteria. Amazon incorporated the feedback by refreshing test sets with more-challenging adversarial scenarios, introducing a difficulty calibration framework with tiered levels, establishing a quarterly review cadence, and broadening multilingual prompts across all risk categories.
Frontier AI models require rigorous safety evaluation. Amazon invests heavily in internal safety testing spanning harmful-content, cybersecurity, privacy, and copyright compliance. AVERI is an independent nonprofit dedicated to third-party evaluation of AI systems. Their mandate was to examine the quality and rigor of Amazon’s own safety evaluation datasets.
Amazon provided AVERI with a random sample of prompts from its internal safety-evaluation suite. AVERI’s team independently examined the prompts for diversity, difficulty calibration, and methodological soundness without input from Amazon’s evaluation team.
AVERI validated several strengths of Amazon’s evaluation process: comprehensive topical coverage, diverse testing approaches in writing style and turn structure, and low contamination risk from publicly available datasets.
Opportunities for improvement included expanding linguistic and stylistic diversity more evenly across risk categories, adding multistep adversarial prompts to probe deeper reasoning, and maintaining evaluation freshness by removing time-limited references and clarifying grading criteria.
Amazon made concrete improvements: refresh test sets with more-challenging adversarial scenarios, introduce a difficulty calibration framework with tiered levels, establish a quarterly review cadence, and broaden multilingual coverage across all risk categories.
AVERI's role was to examine the quality and rigor of Amazon's own safety tests, not just test model outputs. They assessed whether the evaluation prompts stress-test safety boundaries effectively.
AVERI recommended expanding linguistic diversity more evenly across risk categories, calibrating attack sophistication with more multistep adversarial prompts, maintaining evaluation freshness to avoid time-limited references and unclear grading criteria, and ensuring evaluation signal quality by addressing superficial refusal cues.
Amazon made concrete improvements including test set refresh with more-challenging adversarial scenarios, introduction of a difficulty calibration framework with tiered levels, establishment of a quarterly review cadence, and broader multilingual coverage across all risk categories.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
An adversarial prefix is a challenge prepended to a prompt to see whether a model holds its original decision. This diagram maps the layers that turn that pressure test into a measurable, reviewable signal.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
