Backdoor Defense Is a Trade-off, Not a Scanner
A model that passes benchmarks can still defect only on a trigger. This explainer maps the trade-off between deeper backdoor verification and the friction, latency, and cost it adds.
Loading…
Topic
AI Security
50 posts about AI Security. Every summary links to the original.
- 50 posts
- 10 companies
- 2 explainers
See the idea
AI Security explainers
How to Layer AI Security Controls
Defense-in-depth for AI agents combines intent verification, contextual policies, runtime enforcement, and network boundaries to block both direct attacks and slow-burn data exfiltration.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Donating another $20 million to Public First Action
The announcement says an additional $20 million contribution to Public First Action brings total support to $40 million. Public First Action is described as nonpartisan, educating the public about AI and working across party lines on safeguards; the donations are restricted to public education and policy and cannot influence elections. It cites rapidly improving models, including Claude Mythos Preview’s discovery of thousands of high-severity vulnerabilities, and says the model was released to a limited set of cyber defenders through Project Glasswing. The policy agenda calls for verifiable safety claims, civil penalties, deployment controls for catastrophic risks, independent evaluation, security programs, and tighter controls on advanced chips and illicit model access.
Responsible AI: Governance, Principles, and Practical Guide
Responsible AI is presented as a lifecycle-wide practice for designing, developing, deploying, and monitoring AI systems with fairness, transparency, accountability, privacy, safety, and human oversight as requirements. The guide connects technical controls—secure encrypted data pipelines, documented dataset provenance, demographic bias audits, adversarial robustness tests, access controls, and continuous monitoring—with governance mechanisms including named model owners, cross-functional oversight, model-risk assessments, and immutable decision logs. For generative AI, it recommends output policies, training-data leakage testing, guardrails, and red-team testing, while model cards, automated fairness checks, independent audits, and incident response plans support transparency and accountability. Regulatory preparation includes mapping systems to the EU AI Act’s risk categories and documenting design, training data, and intended use; the NIST AI Risk Management Framework and OECD AI Principles are identified as governance references.
Blocking Slow-Burn Attacks: Contextual Policies in Omnigent
Omnigent’s post examines how a vendor-review assistant can leak confidential pricing terms when an attacker hides an indirect prompt injection in a shared runbook. Because the malicious workflow is divided into ordinary actions, stateless checks approve each step even though the session as a whole is unsafe. The demonstration compares an unprotected run, which sends the summary externally, with a contextual policy that stores a running risk score, adds 30 for each document read, and denies email after the score exceeds 50. It also shows that agents cannot remove or disable policies, new policies require human approval, and any denial prevails when policies are combined. Runtime enforcement therefore preserves the block even when the agent has been misled.
Progress from our Frontier Red Team
Anthropic’s Frontier Red Team reports on national-security risks from frontier AI models, drawing on work across four releases over the past year. It finds early-warning signs of rapid progress in dual-use capabilities: Claude reached undergraduate-level performance on some cybersecurity evaluations and exceeded expert baselines in selected biology tasks, while current models remain below thresholds for substantially elevated national-security risk. The team compares model generations on CTF and Cybench tests, realistic roughly 50-host cyber ranges, virology troubleshooting, LabBench, and controlled biosecurity studies, including expert red-teaming. Models still fail autonomously in realistic network environments and make critical mistakes in weaponization planning, although researcher-built tools enabled an attack resembling a known large-scale theft of personally identifiable information; the team is expanding automated evaluations and pursuing government partnerships and stronger safeguards.
Frontier threats red teaming for AI safety
Frontier threats red teaming applies adversarial testing to evaluate whether AI models could create national-security risks, using biology as a test project. Anthropic worked with biosecurity experts to define threat models, probe models for more than 150 hours, learn jailbreak techniques, and build quantitative, repeatable evaluations in a secure interface without public-deployment safety monitoring. The project found that frontier models can sometimes produce sophisticated, accurate, detailed biological knowledge, with capability increasing alongside model size and potentially with tool access; the authors assess unmitigated risks as possibly near-term. Training changes and classifier-based filters reduced harmful outputs or access to chained expert information, and these mitigations were deployed in the public-facing frontier model. Future work will test LLM speedups over search engines, examine newer and smaller adapted models, and expand collaboration, disclosure, and independent evaluation.
Contextual Policies in Omnigent: Using session state to better govern AI agents
Omnigent, an open-source meta-harness for AI agents, introduces contextual policies to make agent controls safer and less disruptive than per-action allow, deny, or approval rules. Policies receive session events, maintain private state such as tools used, documents read, accumulated risk, initial intent, and model spend, then allow, deny, transform, or escalate the next action. Omnigent’s server intercepts tool calls from supported agents and applies these policies consistently, while examples include Google Drive restrictions based on documents created or marked confidential, risk thresholds that require approval for later email or file-sharing actions, and budget thresholds that pause or redirect work to a cheaper model. Intent-based authorization limits tools according to the user’s opening request, applying least privilege across supported harnesses and custom agents. The project is described as open source and alpha, with the server providing one interception layer for agents using different harnesses.
Building safeguards for Claude
Claude’s Safeguards team describes a lifecycle approach to reducing misuse while keeping the model useful, combining policy, training, evaluation, real-time enforcement, and monitoring. The team uses a Unified Harm Framework spanning physical, psychological, economic, societal, and individual autonomy harms, and policy vulnerability testing with external experts to shape policies and defenses. Before release, Claude undergoes safety, risk, and bias evaluations, including capability uplift testing in high-risk domains; findings can lead to new detection methods or guardrails. After deployment, prompted or fine-tuned Claude classifiers monitor violations for response steering and account actions, while human review and CSAM hash matching support enforcement. Ongoing analysis uses privacy-preserving topic clusters, hierarchical summarization, and threat intelligence to identify aggregate misuse and novel attacks.
Barracuda makes security logs conversational with Genie
Barracuda Managed XDR needed to make security-log investigations accessible beyond analysts who knew each vendor’s schema and could write SQL. Its AI-Powered Log Search integrates Databricks Genie into the XDR dashboard: Genie translates natural-language questions into SQL, runs them against customer logs, and returns tabular results with the generated query. Security-specific metadata in Unity Catalog helps map requests across network, server, cloud, email, and endpoint sources, while multi-turn context supports iterative filtering. Secure views enforce organization-level filtering before execution; scoped service principals and SQL validation add further controls against cross-tenant access. Barracuda reports reducing routine investigation time from hours to minutes and estimates that self-service could free 83 to 100 SOC analyst hours monthly, or 1,000 to 1,200 annually.
Government of Alberta uses Claude to find and fix cybersecurity vulnerabilities
The Government of Alberta’s Ministry of Technology and Innovation has used Claude Code with Claude Opus and Sonnet since 2025 to review, secure, and modernize systems spanning 27 provincial ministries. About 50 autonomous agents scanned 466 million lines across 3,400 repositories in roughly 20 hours, using a rules-engine first pass followed by reviews that cite exact files and lines; Alberta estimates a traditional review could take around 6.5 years. Claude Code also generated and tested fixes, wrote missing tests, and rebuilt some obsolete applications, with Ministry engineers approving patches before release. Specialized red-team and blue-team agents now run continuous checks against roughly 95 security controls, while the Alberta AI Academy trains employees and members of the public. The government plans to consolidate 185 legacy applications into 16 reusable modern applications, a future goal rather than a reported result.
Scaling Security Alert Triage With Specialized Agents on Databricks
Databricks describes a security-alert triage system designed to review low-severity alerts that historically received less attention because of their volume and low fidelity. An initial single-agent prompt escalated 50% of alerts, so the team built a fleet of 17 source-specific agents plus a Threat Intelligence agent, using Structured Streaming to ingest alerts, enrich evidence, and route each alert. The pipeline combines deterministic benign-signal filtering, historical and behavioral context, specialized prompt functions, optional tools, structured dispositions, and cost controls before sending escalations to human analysts. Those analysts provide ground truth through MLflow traces and labels, supporting ongoing evaluation across escalate, monitor, and close decisions. After triaging more than 18,000 alerts, the system reached a 3.2% escalation rate, a 10.5-second median triage time, and more than 6,500 analyst hours saved in 30 days; escalated low-severity alerts were roughly 10 times more likely to be true positives than HIGH and MEDIUM alerts.
Claude's extended thinking
Claude 3.7 Sonnet introduces extended thinking, letting users toggle deeper reasoning and developers set a thinking budget. The same model spends additional computation on sequential reasoning, while agent training supports iterative function calls, computer use, and open-ended tasks. The post examines visible thought processes, noting benefits for trust and research but also limitations around faithfulness, safety, and jailbreak risk; it therefore treats the feature as a research preview. Reported examples include stronger OSWorld performance, playing Pokémon Red through tens of thousands of interactions and progressing through three Gym Leaders, plus logarithmic math gains as thinking tokens increase. Safety work includes encrypted thoughts for some harmful content and computer-use prompt-injection mitigations that raise prevention from 74% to 88%.
More details on Fable 5’s cyber safeguards and our jailbreak framework
Claude Fable 5 has been redeployed globally, accompanied by additional detail about its cybersecurity safety classifiers and a proposed AI jailbreak severity framework. The classifiers distinguish prohibited use, high-risk dual use, low-risk dual use, and benign use, with intended responses ranging from blocking to allowing with monitoring; the model uses a larger safety margin than previous models, accepting more false positives to increase protection against harmful outcomes. Other safeguards include access controls, model safety training, and offline monitoring, while a HackerOne program invites researchers to submit potential cyber jailbreaks. The draft framework measures capability gain against the available baseline and illustrates scoring with Log4Shell cases, where severity changes as discoverability and existing tools change.
Redeploying Claude Fable 5
Access to Claude Fable 5 and Mythos 5 was restored after US export controls had temporarily suspended both models for all users. The post says Fable 5 will return globally on July 1 across Claude Platform, Claude.ai, Claude Code, and Claude Cowork, while Mythos 5 access has resumed for some US organizations and broader partner access remains under discussion. Anthropic describes a new safety classifier intended to block the bypass reported by Amazon researchers in more than 99% of cases; blocked Fable 5 requests are sent to Opus 4.8, although the classifier increases false positives in coding and debugging. It also outlines defense-in-depth safeguards, a proposed shared industry framework for assessing jailbreaks, and expanded government collaboration on pre-release evaluation, information sharing, and joint research.
Claude Fable 5 and Claude Mythos 5
Claude Fable 5 and Claude Mythos 5 are presented as a joint launch of the same underlying model, with Fable 5 intended for general use and Mythos 5 initially reserved for selected cyberdefenders and infrastructure providers. Fable 5 uses conservative safeguards that route some sensitive queries to Claude Opus 4.8, while Mythos 5 lifts safeguards in selected areas through Project Glasswing and a planned trusted-access program. The post reports strong results in software engineering, knowledge work, vision, long-context tasks, cybersecurity, protein design, molecular biology, and genomics, including a reported 50-million-line Ruby migration completed in a day. Both models are priced at $10 per million input tokens and $50 per million output tokens, with Fable 5 available through the Claude API and staged subscription access.
Palana (Part 2): Architecting isolation, identity, and auditability for AI agents
Palana is Grab’s Kubernetes-native secure execution platform for autonomous AI agents, designed to provide isolated, attributable, and auditable environments. It provisions per-agent namespaces, service accounts, EFS-backed storage, ingress, policies, and proxy-mediated egress, while admission webhooks inject proxy settings and enforce pod restrictions. Human identity remains authoritative for RBAC, while sanitized identifiers are reserved for object names and Vault paths; secrets are split between agent-readable values and proxy-only credentials, and the LLM wrapper derives agent identity from Kubernetes context. Layer 3/4 controls provide containment, Layer 7 OPA-backed proxy policies govern host and method access, and structured signals support audit and idle shutdown that preserves /data and platform state. The production lessons emphasize platform-level controls, a safe self-service path, namespace boundaries, proxy-mediated authentication, and lifecycle management as prerequisites for useful autonomy.
What’s new in Databricks Data + AI Platform security and compliance at Data + AI Summit 2026
At Data + AI Summit 2026, Databricks announced security and compliance capabilities for scaling Genie, Lakebase, serverless workloads, and AI-powered applications without relying solely on manual provisioning, static network controls, or siloed compliance programs. Automatic Identity Management (AIM) for Microsoft Entra ID is generally available on AWS and Google Cloud, AIM for Okta is in Public Preview, and Context-Based Ingress is in Public Preview across all three clouds for policies based on network source, identity, and access scope. Private Network Gateway, in Private Preview on Azure Databricks, provides one secure connection from serverless workloads to private networks, while expanded Private Link support extends to Lakebase and other services. Compliance additions include Azure Serverless coverage, HITRUST across AWS, Azure, and Google Cloud, ISMAP on Azure and AWS, expanded AWS GovCloud availability, and planned FedRAMP High support on Azure Commercial.
Building an open ecosystem for AI governance with Unity AI Gateway
Databricks announced the Unity AI Gateway partner ecosystem, extending enterprise AI governance beyond models to runtime interactions among models, agents, MCP servers, skills, and AI tools. Built on Unity Catalog, the gateway lets organizations apply policies, monitor activity, manage spend, and govern AI across providers and frameworks, while integrating security, identity, and governance products they already use. The announcement groups the integrations into runtime AI security, observability and guardrails; agent identity and access governance; and AI observability and risk monitoring. Named integrations include Alice, CrowdStrike Falcon AI Detection and Response, Cyera, HiddenLayer, Netskope, Noma Security, Obsidian Security, Openlayer, Okta, Ping Identity, SailPoint, and Saviynt, with described capabilities including prompt-injection detection, data-loss prevention, agent discovery, authorization, and lifecycle governance.
Introducing Omnigent: A Meta-Harness to Combine, Control and Share Your Agents
Omnigent is an open-source meta-harness designed to combine, control, and share agents across different harnesses, models, and interfaces. Databricks says users currently juggle multiple agents and copy context between them, while builders struggle to combine or replace harnesses with incompatible interfaces. Its runner wraps terminal-based agents and SDKs in sandboxed sessions with a uniform API for messages, files, streamed text, and tool calls; a server adds policies, sharing, and access through terminal, web, mobile, native Mac OS, and APIs. Features include live collaboration, hosted sandbox execution, contextual security and cost policies, OS isolation, and multi-harness authoring. Released in alpha under Apache 2.0, Omnigent is intended to provide a durable layer above changing agents and harnesses.
Statement on the US government directive to suspend access to Fable 5 and Mythos 5
Anthropic says the US government issued an export-control directive requiring it to suspend Fable 5 and Mythos 5 access for every foreign national, including foreign-national Anthropic employees, inside or outside the United States. The directive arrived at 5:21 p.m. ET and did not specify the national-security concern; Anthropic says it understands the basis to be a potential narrow, non-universal jailbreak involving a request to inspect a codebase and fix software flaws. The company says it reviewed the underlying report, found only previously known minor vulnerabilities, and judged the demonstrated capability widely available from other models, including OpenAI’s GPT-5.5. Anthropic is complying by disabling both models for all customers and says it is working to restore access.
Governing agent autonomy with Auto-review
Auto-review governs local agent autonomy as a continuum, allowing routine work to proceed while reviewing actions whose consequences depend on context, user intent, and security stakes. A small classifier agent runs before tool calls in the same RPC stream as the parent agent, and can inspect workspace contents with ReadFile, Grep, Glob, and ListDir when a command alone is insufficient. Its blocks return explanations rather than automatically prompting users, enabling the parent agent to narrow or replace a risky action. Evaluation used 6,122 deduplicated labeled rows from internal developer sessions plus synthetic high-consequence cases, and tested tool use, overrides, parse failures, and behavioral flapping. Early results show the classifier blocks around 4% of reviewed actions, while about 7% of Auto-review chats experience at least one interruption.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.