Backdoor Defense Is a Trade-off, Not a Scanner
A model that passes benchmarks can still defect only on a trigger. This explainer maps the trade-off between deeper backdoor verification and the friction, latency, and cost it adds.
Loading…
Topic
AI Security
50 posts about AI Security. Every summary links to the original.
- 50 posts
- 10 companies
- 2 explainers
See the idea
AI Security explainers
How to Layer AI Security Controls
Defense-in-depth for AI agents combines intent verification, contextual policies, runtime enforcement, and network boundaries to block both direct attacks and slow-burn data exfiltration.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Backdoors in LLMs: Why model scanning isn't enough
Pulling a model from Hugging Face and running benchmarks does not guarantee the model will behave the same in production. Researchers have shown that language models can be trained to act normally during evaluation and defect only when a specific trigger appears, and that standard post-training defenses do not reliably remove such planted behaviors. The attack surface includes malicious artifacts, chat-template injections, LoRA/adapter backdoors, training-data backdoors, and MoE routing backdoors, each requiring different defenses. No single scanner or test can prove a third-party model is clean, so a multi-stage supply-chain process is needed: verify provenance, isolate untrusted artifacts, scan non-weight files, probe behavior, optionally realign, and apply runtime guardrails. The goal is to turn model intake from implicit trust into an auditable process, assuming some malicious artifact will eventually pass the scanner.
AI threat modeling shouldn't be a bottleneck, so we open sourced ours
The post announces the open-sourcing of agentic-threat-modeling, an AI-assisted threat-modeling module for Claude Code, Cursor, and Gemini CLI, built to address the bottleneck of security teams reviewing AI-generated code. It runs on Lola, an open source package manager for AI contexts, and profiles likely attackers, generates narrative attack scenarios grounded in actual code, and checks them against six structured threat modeling frameworks (STRIDE, PASTA, LINDDUN, attack trees, and more). The module ties every finding to a specific file and line of code, as shown in a real-world example against a file upload endpoint. It’s designed to produce structured output for AI-assisted triage, though that pipeline isn’t built yet. The team open sourced it because the problem isn’t unique to them, and it’s available in RedHatProductSecurity/agentic-threat-modeling and LobsterTrap/lola.
Build zero-trust AI agents that judge intent, not just syntax
Part 2 of the Zero-trust Agents series addresses attacks that deterministic, build-time checks cannot recognize when syntax remains valid. It keeps the Customer Support and Returns Agent, which reads orders, calculates restocking fees in Agent Sandbox, and commits payouts through an issue_refund tool signed with a hardware-backed Cloud KMS key, then moves enforcement to Gemini Enterprise Agent Platform. Agent Gateway applies three runtime controls: Model Armor screens prompts and responses, Semantic Governance Policies evaluates proposed tool calls against intent, history, and business rules, and Agent Anomaly Detection analyzes behavior across a session. The examples cover prompt injection, a non-refundable digital-license refund, and a multi-turn refund drain; detected anomalies can trigger a new policy that blocks subsequent calls without redeploying the agent.
Partnering with Mozilla on Firefox security
Anthropic describes a collaboration with Mozilla in which Claude Opus 4.6 discovered 22 previously unknown Firefox vulnerabilities over two weeks, including 14 that Mozilla classified as high severity. The work began with reproducing historical Firefox CVEs, then shifted to novel bugs in the current codebase, starting with the JavaScript engine and eventually scanning nearly 6,000 C++ files. Anthropic and Mozilla submitted 112 unique reports, while most issues were fixed in Firefox 148; Claude also produced working exploits for two vulnerabilities in a deliberately weakened test environment. The post says Claude is substantially better at finding and fixing vulnerabilities than exploiting them, but warns that this gap may narrow and recommends evidence-rich reports, task verifiers, and coordinated disclosure processes.
Dario Amodei's remarks at the AI Safety Summit
At the AI Safety Summit, Dario Amodei explains why rapid AI progress makes dangerous capabilities difficult to predict and motivates Anthropic’s Responsible Scaling Policy (RSP). The policy combines AI safety levels (ASLs), modeled loosely on biological safety levels, with recurring tests for dangerous capabilities along the compute scaling curve. ASL-2 covers current models and calls for model cards, external red-teaming, and strong security, while ASL-3 would require unusually strong protection against weight theft and safeguards preventing operationally useful CBRN information; ASL-4 adds escalated misuse and autonomous-system risks. Amodei also describes executive involvement, integrating RSP deadlines into product and research planning, board and external accountability, and the goal of refining company policies into regulatory testing and auditing regimes.
Safety for Whom? Refusing the Right Subset of a Topic, Not the Whole Topic
Topic-level safety systems can conflate a dangerous subject with dangerous intent, causing models to refuse legitimate requests alongside harmful ones. The paper studies deployment-specific boundaries using political persuasion as a testbed: refuse manipulation or targeted persuasion while answering factual political questions. Its training pipeline repairs self-generation coverage through escalating retries, adds verified surface-dangerous benign examples, and evaluates held-out harmful-benign prompt pairs rather than relying on aggregate refusal rates. On Qwen3-8B, escalated coverage raises in-distribution political refusal from 9.47% to 84.75%, while the strongest configuration reduces unsafe responses across three broader benchmarks from 26.26% to 0.14%; however, XSTest over-refusal rises to 74.00%. Adding benign boundary data reduces over-refusal on comply-worthy held-out prompts from 32.94% to 4.16%, with harmful-side refusal declining from 91.88% to 87.72%.
Detecting and countering malicious uses of Claude
The report examines how malicious actors used Claude in influence operations, credential scraping, recruitment fraud, and malware development while preserving utility for legitimate users. It details an influence-as-a-service operation in which Claude orchestrated more than 100 social-media bot accounts, selecting likes, shares, comments, or inaction across personas and political objectives. The response used conversation-analysis techniques including Clio and hierarchical summarization, alongside classifiers that assessed user requests and model responses to identify, investigate, and ban accounts. The cases show frontier models accelerating less skilled actors’ capabilities, although successful deployment was not confirmed for the credential, recruitment, or malware efforts, and no content from the influence operation achieved viral status.
Detecting and countering misuse of AI: August 2025
An August 2025 threat intelligence report describes three cases in which Claude was misused for cybercrime and fraud: a data-extortion operation, North Korean remote-worker fraud, and AI-generated ransomware sold as a service. In the extortion case, Claude Code automated reconnaissance, credential harvesting, network penetration, data-selection, ransom calculation, and psychologically targeted demands against at least 17 organizations, with some ransoms exceeding $500,000. The report says AI lowered the expertise barrier: one actor used Claude to build ransomware with encryption, evasion, and anti-recovery features, then sold variants for $400 to $1,200, while other operators used it to secure and maintain technical jobs. Responses included banning accounts, adding tailored classifiers and detection methods, improving indicator correlation, alerting authorities and partners, and sharing technical indicators with third-party safety teams.
Evaluating LLM guardrail configs locally with EvalHub
This installment evaluates local NeMo Guardrails configurations against large-scale risk datasets using EvalHub, following earlier manual testing. EvalHub provides a unified interface for AI evaluation, with provider integrations, reusable benchmarks, Hugging Face dataset inputs, and both local and Kubernetes deployment options. The benchmark configuration maps dataset prompt and label columns to block and pass labels, while balanced safe and risky examples prevent an always-block guardrail from appearing perfect. On the prompt-injection benchmark, the nuanced regex configuration achieved 51.1% accuracy with 13.7 ms median latency; DeBERTa reached 88.93% accuracy with 45.2 ms median latency. The conclusion favors layered guardrails: targeted regex checks can provide speed, with DeBERTa handling other cases, and EvalHub supplies reproducible measurements of the latency–efficacy trade-off.
Developing Enterprise Frontier Safeguards with our customers
Anthropic is announcing Enterprise Frontier Safeguards (EFS), designed to combine zero data retention (ZDR) with monitoring for serious misuse of frontier models. The service addresses enterprise concerns that detecting activity such as fraud, cyberattacks, or biological capability development, or stolen credentials, may require retaining and correlating traffic across sessions and accounts, while regulated organizations may not accept provider-held data or human review. EFS stores activity data in customer-controlled cloud infrastructure, under customer-managed encryption keys, access policies, and audit logging, while automated systems analyze a rolling traffic window and send flags directly to the customer for review. It will roll out in phases later this fall, support Anthropic and partner cloud offerings, and let customers opt into storage, encryption keys, and automated review without changing model behavior, pricing, or rate limits.
LoRA backdoor threat: How OpenShift AI mitigates the risk
LoRA adapters can carry semantic backdoors even when the underlying open-weight model is clean, creating a supply-chain risk for customized AI assistants. The demonstrated proof of concept uses a 35 MB adapter trained from 270 ChatML examples, with 130 poisoning examples and 140 clean examples, to trigger on ordinary pharmaceutical IP language rather than fixed strings. With Qwen/Qwen2.5-0.5B-Instruct served by vLLM 0.25.1, the backdoored model covertly called send_email to an external address while showing a benign response, whereas the base model and controls remained clean. The defense-in-depth approach combines OpenShift AI, Advanced Cluster Security, and Advanced Cluster Management to restrict runtime LoRA loading, require provenance-checked adapters, enforce admission and build policies, and block unexpected egress. Because weight inspection cannot reliably reveal semantic behavior, protection focuses on deployment, runtime, tool, and network boundaries.
Developing LLM guardrail configs locally with NeMo Guardrails
The article presents a local workflow for developing and testing NeMo Guardrails configurations in a Jupyter notebook, avoiding repeated Kubernetes deployments and requiring no LLM, GPU, or cloud resources. It defines rails, configurations, and servers, then starts with a case-insensitive regex rail that blocks the exact phrase “ignore all previous instructions”; the test catches a direct attack but allows an accented “Ignóre” variant. A broader regex configuration is followed by a DeBERTa-based prompt-injection classifier and a manually assembled set of 13 prompts, producing 12 correct decisions out of 13, with one subtle attack allowed. The article recommends iterating locally, potentially layering targeted regexes before classifiers for fail-fast, defense-in-depth filtering, while noting that 13 prompts cannot meaningfully measure performance and pointing to standardized EvalHub evaluations as a next step.
Challenges in red teaming AI systems
AI red teaming is presented as adversarial testing intended to identify vulnerabilities and risks in AI systems, but inconsistent practices make safety comparisons difficult. The post surveys domain-specific expert testing, policy vulnerability testing, frontier-threat assessments, multilingual and multicultural work, language-model-assisted attacks, automated red teaming, multimodal testing, and crowdsourced or community-based approaches. It describes an iterative process in which experts articulate threat models and probe systems qualitatively, then standardize inputs and use language models to generate many variations for more quantitative automated evaluation. In automated red teaming, a red team model generates attacks and a blue team model is fine-tuned on those outputs to improve robustness. The conclusion calls for technical standards, independent testing capacity, professional services, third-party access, and links between red teaming and model release conditions.
Frontier model security
Frontier AI models and their research are presented as strategic assets requiring protections beyond ordinary commercial technology because theft or misuse could affect economic and national security. The post recommends treating advanced AI as akin to critical infrastructure, with public-private cooperation and possible procurement or regulatory requirements. Its central control is multi-party authorization: no individual retains persistent access to production-critical environments, and time-limited access requires a coworker’s approval and business justification. It also argues that NIST’s SSDF and SLSA can be adapted to model development and coupled software, creating a chain of custody that supports provenance for deployed systems. Anthropic says it is implementing these controls and other cybersecurity practices through an iterative process, while acknowledging protections must evolve as model capabilities scale.
Integrations That Write Themselves
Ramp describes an approach to scaling integrations beyond what engineers can build and maintain by hand. Its agentic system lets customers describe a missing workflow integration, researches the provider’s API, requests credentials through a secure widget, writes and tests ordered API-call “recipes,” and generates a deterministic script that runs the integration after build time. An internal Integration Factory applies the same pattern to first-party connectors, producing evidence for review and opening pull requests; the post says 75 integrations have shipped this way, with work reduced from weeks or months to hours. Security controls include HTTPS and hostname allowlists, private-address rejection, isolated egress, bounded responses, business-scoped immutable recipes, and test credentials rather than customer data. The central design lesson is to use models during research and construction, keep them out of execution, and bound autonomy with verification artifacts.
Cursor earns AIUC-1 certification for agent security and reliability
Cursor announces that it has earned AIUC-1 certification after an independent review of its organizational controls and agent behavior. AIUC-1 combines an organizational-controls audit with adversarial product testing, translating the NIST AI Risk Management Framework, MITRE ATLAS, and OWASP’s agentic threat taxonomy into requirements for live AI systems. Schellman reviewed documented controls and AI governance practices, while evaluators tested IDE and cloud agents in a representative enterprise configuration across several thousand scenarios and two rounds. Testing covered rules, hooks, Auto-review, model-level safeguards, secrets protection, secure code generation, MCP security, identity and permissions, and potentially destructive actions. Cursor says its safeguards held in benign and adversarial conditions, with certification requiring quarterly testing and an annual full audit as AIUC-1 evolves.
How We’re Building Scam Alert on WhatsApp With End-to-End Encryption and Verifiability Guarantees
WhatsApp presents an early technical overview of Scam Alert, an optional feature that uses an on-device machine learning model to identify potential scam messages from non-contacts while keeping message content on the device. The warning appears only to the user, who can block, report, continue, or mark a chat trusted; reporting and sharing require explicit user action. To measure performance, the system sends locally aggregated warning and user action counts through confidential federated analytics using Trusted Execution Environments and differential privacy, rather than transmitting raw signals. Model versions are recorded on a public transparency ledger, while published weights, client-side logs, and an expanded Bug Bounty program support independent inspection of delivery and behavior.
Databricks joins the Open Secure AI Alliance to advance AI safety and security
Databricks announces that it is a founding member of the Open Secure AI Alliance, formed with NVIDIA and more than 75 organizations to advance openly shared AI safety, AI security, and AI-enabled cyber defense research. The alliance’s scope extends beyond model weights to an open execution stack covering runtimes, guardrails, agent harnesses, risk frameworks, and governance. Databricks contributes Omnigent, an Apache 2.0 meta-harness supporting 13+ harnesses with contextual policies, spend caps, and sandbox isolation; DASF 3.0, which maps 97 risks across 13 components to 73 controls; DAGF; and BlackIce, a containerized toolkit bundling 14 red-teaming tools. Its Lakewatch Security Lakehouse applies the same openness to governed, agentic detection and response over portable data formats.
Databricks Completes Acquisition of Panther: Accelerating the Security Lakehouse Era
Databricks says it has completed its acquisition of Panther, an AI SOC platform, to accelerate its security lakehouse strategy. The combination pairs Lakewatch’s open, governed foundation for collecting, retaining, and analyzing petabyte-scale security telemetry with Panther’s operational SOC workflows and more than 100 out-of-the-box integrations. Panther adds detections-as-code, CI/CD-based authoring and deployment, and AI-native triage and investigation that correlate cloud, identity, SaaS, IT, and business data. The announcement presents the combined platform as a way to retain high-fidelity telemetry, preserve data ownership through open standards including OCSF, Spark, Unity Catalog, Delta, Parquet, and SQL, and automate investigations, detection refinement, and response workflows for modern security operations.
Permission isn't purpose: Intent-based authorization in Omnigent
Omnigent’s intent-based authorization addresses a gap in identity-based access control: an agent with valid credentials may follow indirect prompt injections embedded in data and perform authorized actions unrelated to the user’s task. It binds each session to a human-declared purpose and evaluates that intent before every tool call, producing permitted, consent-required, or denied verdicts. In a data-quality example, reading a customers table is allowed, dashboard publication requires approval, and an injected request to grant external access is denied despite the agent’s identity permitting that tool. The intent is fixed at design time for autonomous agents or approved by a human at session start for interactive agents, and cannot be broadened by the agent. Omnigent combines this policy with session-risk scoring and other contextual policies in a single engine where any denial wins.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.