From Ringing Line to Resolved Contact: How a Call Center Works
Follow one contact through a call center: arrival, routing, handling by a human or AI agent, resolution, and the cost-per-contact math that decides which path to use.
A five-stage walkthrough of how CBRN risk evaluations are commissioned, designed, graded, and converted into safety-level decisions for frontier AI models.
CBRN evaluations ask whether a frontier model could meaningfully assist with chemical, biological, radiological, or nuclear harm, and they feed safety-level decisions. The pipeline defines which risk thresholds matter, commissions external teams with domain expertise, and demands evaluations that are hard to pass, absent from training data, efficient, and high volume. Grading uses executable checks on real behavior rather than a model's own claims, with coverage tracked across a representative model set and pinned versions. Engineers should care because evaluation quality, not just model capability, decides whether a release gate is trustworthy.
Demand for high-quality safety evaluations currently outpaces supply, and the hardest ones need domain expertise a frontier lab may not hold internally. The initiative funds outside organizations to develop them while giving partners direct collaboration with domain experts, so results stay independent of the team being evaluated.
The stated bar is that an evaluation be sufficiently difficult, not present in training data, efficient, and able to run at high volume, and that it be developed with genuine domain expertise. Those properties keep results meaningful as models improve and keep the evaluation cheap enough to rerun.
Grade the end state rather than the transcript: executable checks on the final artifact catch agents that report success without doing the work. Recording real request and response traffic and replaying it as a fixed suite also surfaces behavior changes that intent-based tests miss.
The posts behind it
The torch-neuronx model support Space on Hugging Face checks whether Transformer models run unmodified on AWS Trainium accelerators. It offers three views: Selected (curated top models), Trending, and Downloads, each showing the result, number of Trainium2 cores, wall time including compilation, and peak device memory. Users can look up a specific model by typing its name and select the Neuron compiler (neuronx-cc) and Transformers library versions to inspect compatibility. Each test loads the model on a trn2 instance, shards with tensor or expert parallelism, generates 32 tokens, and checks fluency against a CPU run or judge model. The underlying dataset is publicly available as Parquet files in the aws-neuron/torch-neuronx-model-support dataset, enabling direct querying with Pandas to retrieve results.
ThinkingBox evaluates agent performance by comparing the final database state with the expected outcome, catching agents that claim completion without correct database changes. Its benchmark, ThinkingBox-Bench, contains 507 tasks across domains like retail, auto insurance, travel, neobank, and consulting, each with executable checks on terminal database state. The authors used these executable outcomes as rewards for reinforcement learning, fine-tuning Qwen3.8-27B with GRPO for 50 steps on a training pool selected by offline policy-relative difficulty, which filters tasks that are neither already solved nor hopeless. After training, observed 20/20 tasks completed correctly on all twenty attempts rose from 38 (7.50%) to 107 (21.10%), while pass@1 increased from 51.70% to 60.78%, with a small decline in at-least-once coverage. The results show that database-state grading can serve as a reward signal to improve agent repeatability, and the training loop and data curation method are released on GitHub.
Anthropic is opening applications for a guest investigator program in the life sciences, targeting principal investigators and their research groups at academic institutions, nonprofits, or government labs. Over 8–12 weeks, selected teams will collaborate directly with Anthropic researchers on problems in causal human biology, structural and computational molecular biology, medicinal chemistry, drug discovery, and clinical science. Anthropic will provide access to its most capable models and computing resources at no cost, and the resulting analyses, artifacts, and tools will be published with open-source data and code. Applicants must explain how they would benefit from direct collaboration and make a reasonable case that their outputs—such as preliminary discoveries or open-source datasets—can be produced within the collaboration period. Guest investigators retain full rights to their data throughout the project and to any independent follow-on research, and applications are open for approximately 14 days.
Many regression tests verify what developers intended a system to do, not what it actually does, leaving a critical gap. Hyrum’s Law states that once a system has enough users, every observable behavior becomes depended on, so unplanned quirks and accidental defaults are a common source of regressions. Characterization tests, coined by Michael Feathers, address this by capturing the current actual behavior of code as a test, regardless of whether that behavior is correct. They are rarely written because writing them is a slog—requiring testing many inputs manually—and because it feels wrong to assert behavior that appears buggy. A powerful method is to record production traffic at the network level, capturing requests, responses, and side effects like database calls, then replaying those recordings as a repeatable test suite that flags any behavior change, and helps during refactoring, framework upgrades, rewrites, and AI-assisted changes.
PyTorch is the dominant AI framework, operating on tensors, and AWS announced TorchNeuron, an open-source backend translating PyTorch code to run on Trainium. TorchNeuron’s adaptive eager mode compiles small subgraphs, overlapping compilation with execution to hide compile time while preserving source correlation for debugging. It employs multi-level caching of NEFF files and coordinates accelerators across an UltraServer to avoid redundant compilation, reducing time to first training step by a third. To evaluate performance, AI agents scanned Hugging Face repositories, classified models by architecture, and identified 841 representative models covering the full spectrum of architectures. From these, they derived a shape registry of tens of thousands of tensor dimensions for staged testing, ensuring accurate and efficient execution on Trainium.
Anthropic announced a new initiative to fund third-party organizations to develop evaluations for measuring advanced AI capabilities and safety risks. The initiative prioritizes three areas: AI Safety Level assessments for cybersecurity, CBRN, model autonomy, and other national security risks; advanced capability and safety metrics including science, harmfulness, multilingual, and societal impacts; and infrastructure, tools, and methods for evaluation development. The company seeks evaluations that are sufficiently difficult, not in training data, efficient, high-volume, and developed with domain expertise. Anthropic offers funding and direct collaboration with its domain experts from Frontier Red Team and other teams. The goal is to catalyze a robust third-party evaluation ecosystem to address the current limited landscape where demand for high-quality safety evaluations outpaces supply.
More like this
Follow one contact through a call center: arrival, routing, handling by a human or AI agent, resolution, and the cost-per-contact math that decides which path to use.
A circuit breaker turns dependency failures into fast failures, then probes for recovery. See the state loop and how half-open probes can quietly consume worker capacity.
Cilium uses eBPF programs in the Linux kernel to enforce identity-based network policies and capture L7 observability for Kubernetes workloads. This diagram traces the path from packet arrival to policy verdict and Hubble flow.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.