Anthropic raises Series B to build robust AI

Summary

Anthropic, an AI safety and research company, raised $580 million in Series B financing to build large-scale experimental infrastructure for studying and improving the safety properties of computationally intensive AI models. Since its 2021 founding, the company has researched steerability, robustness, and interpretability, including mathematically reverse engineering small language models and investigating pattern-matching behavior in larger ones. It developed baseline techniques and reinforcement learning methods for making models more “helpful and harmless,” released an alignment-focused dataset, and analyzed sudden performance changes and their societal impacts. The stated goal is to create models with stronger implicit safeguards, fewer after-training interventions, and tools for examining whether those safeguards work, alongside policy and societal research. The round was led by FTX CEO Sam Bankman-Fried and followed a $124 million Series A in 2021; Anthropic said its team numbered around 40 people.

Takeaways

1. 1

   The Series B raised $580 million and was led by Sam Bankman-Fried, following Anthropic’s $124 million Series A in 2021.

2. 2

   Anthropic’s safety work includes mathematically reverse engineering small language models, studying pattern-matching behavior, and applying reinforcement learning to make models more “helpful and harmless.”

3. 3

   The research aims to produce models with better implicit safeguards and fewer after-training interventions, while enabling closer examination of whether those safeguards work.