Loading…
AI Security
16 posts about AI Security. Every summary links to the original.
Red Hat ·
Developing LLM guardrail configs locally with NeMo Guardrails
The article presents a local workflow for developing and testing NeMo Guardrails configurations in a Jupyter notebook, avoiding repeated Kubernetes deployments and requiring no LLM, GPU, or cloud resources. It defines rails, configurations, and servers, then starts with a case-insensitive regex rail that blocks the exact phrase “ignore all previous instructions”; the test catches a direct attack but allows an accented “Ignóre” variant. A broader regex configuration is followed by a DeBERTa-based prompt-injection classifier and a manually assembled set of 13 prompts, producing 12 correct decisions out of 13, with one subtle attack allowed. The article recommends iterating locally, potentially layering targeted regexes before classifiers for fail-fast, defense-in-depth filtering, while noting that 13 prompts cannot meaningfully measure performance and pointing to standardized EvalHub evaluations as a next step.
Rob GeadaRamp ·
Integrations That Write Themselves
Ramp describes an approach to scaling integrations beyond what engineers can build and maintain by hand. Its agentic system lets customers describe a missing workflow integration, researches the provider’s API, requests credentials through a secure widget, writes and tests ordered API-call “recipes,” and generates a deterministic script that runs the integration after build time. An internal Integration Factory applies the same pattern to first-party connectors, producing evidence for review and opening pull requests; the post says 75 integrations have shipped this way, with work reduced from weeks or months to hours. Security controls include HTTPS and hostname allowlists, private-address rejection, isolated egress, bounded responses, business-scoped immutable recipes, and test credentials rather than customer data. The central design lesson is to use models during research and construction, keep them out of execution, and bound autonomy with verification artifacts.
Shubhayan SrivastavaDatabricks joins the Open Secure AI Alliance to advance AI safety and security
Databricks announces that it is a founding member of the Open Secure AI Alliance, formed with NVIDIA and more than 75 organizations to advance openly shared AI safety, AI security, and AI-enabled cyber defense research. The alliance’s scope extends beyond model weights to an open execution stack covering runtimes, guardrails, agent harnesses, risk frameworks, and governance. Databricks contributes Omnigent, an Apache 2.0 meta-harness supporting 13+ harnesses with contextual policies, spend caps, and sandbox isolation; DASF 3.0, which maps 97 risks across 13 components to 73 controls; DAGF; and BlackIce, a containerized toolkit bundling 14 red-teaming tools. Its Lakewatch Security Lakehouse applies the same openness to governed, agentic detection and response over portable data formats.
Katie Cummiskey, Maria Pere-Perez, Arun Pamulapati, Nishith SinhaDatabricks Completes Acquisition of Panther: Accelerating the Security Lakehouse Era
Databricks says it has completed its acquisition of Panther, an AI SOC platform, to accelerate its security lakehouse strategy. The combination pairs Lakewatch’s open, governed foundation for collecting, retaining, and analyzing petabyte-scale security telemetry with Panther’s operational SOC workflows and more than 100 out-of-the-box integrations. Panther adds detections-as-code, CI/CD-based authoring and deployment, and AI-native triage and investigation that correlate cloud, identity, SaaS, IT, and business data. The announcement presents the combined platform as a way to retain high-fidelity telemetry, preserve data ownership through open standards including OCSF, Spark, Unity Catalog, Delta, Parquet, and SQL, and automate investigations, detection refinement, and response workflows for modern security operations.
Andrew Krioukov, Jack Naglieri, Taylor Kain, Dave HerraldPermission isn't purpose: Intent-based authorization in Omnigent
Omnigent’s intent-based authorization addresses a gap in identity-based access control: an agent with valid credentials may follow indirect prompt injections embedded in data and perform authorized actions unrelated to the user’s task. It binds each session to a human-declared purpose and evaluates that intent before every tool call, producing permitted, consent-required, or denied verdicts. In a data-quality example, reading a customers table is allowed, dashboard publication requires approval, and an injected request to grant external access is denied despite the agent’s identity permitting that tool. The intent is fixed at design time for autonomous agents or approved by a human at session start for interactive agents, and cannot be broadened by the agent. Omnigent combines this policy with session-risk scoring and other contextual policies in a single engine where any denial wins.
Nishith Sinha, Matei ZahariaResponsible AI: Governance, Principles, and Practical Guide
Responsible AI is presented as a lifecycle-wide practice for designing, developing, deploying, and monitoring AI systems with fairness, transparency, accountability, privacy, safety, and human oversight as requirements. The guide connects technical controls—secure encrypted data pipelines, documented dataset provenance, demographic bias audits, adversarial robustness tests, access controls, and continuous monitoring—with governance mechanisms including named model owners, cross-functional oversight, model-risk assessments, and immutable decision logs. For generative AI, it recommends output policies, training-data leakage testing, guardrails, and red-team testing, while model cards, automated fairness checks, independent audits, and incident response plans support transparency and accountability. Regulatory preparation includes mapping systems to the EU AI Act’s risk categories and documenting design, training data, and intended use; the NIST AI Risk Management Framework and OECD AI Principles are identified as governance references.
Databricks StaffBlocking Slow-Burn Attacks: Contextual Policies in Omnigent
Omnigent’s post examines how a vendor-review assistant can leak confidential pricing terms when an attacker hides an indirect prompt injection in a shared runbook. Because the malicious workflow is divided into ordinary actions, stateless checks approve each step even though the session as a whole is unsafe. The demonstration compares an unprotected run, which sends the summary externally, with a contextual policy that stores a running risk score, adds 30 for each document read, and denies email after the score exceeds 50. It also shows that agents cannot remove or disable policies, new policies require human approval, and any denial prevails when policies are combined. Runtime enforcement therefore preserves the block even when the agent has been misled.
Nishith Sinha, Matei ZahariaContextual Policies in Omnigent: Using session state to better govern AI agents
Omnigent, an open-source meta-harness for AI agents, introduces contextual policies to make agent controls safer and less disruptive than per-action allow, deny, or approval rules. Policies receive session events, maintain private state such as tools used, documents read, accumulated risk, initial intent, and model spend, then allow, deny, transform, or escalate the next action. Omnigent’s server intercepts tool calls from supported agents and applies these policies consistently, while examples include Google Drive restrictions based on documents created or marked confidential, risk thresholds that require approval for later email or file-sharing actions, and budget thresholds that pause or redirect work to a cheaper model. Intent-based authorization limits tools according to the user’s opening request, applying least privilege across supported harnesses and custom agents. The project is described as open source and alpha, with the server providing one interception layer for agents using different harnesses.
Matei Zaharia, David Nasi, Xiangrui Meng, Kecheng Cao, Tomu HirataBarracuda makes security logs conversational with Genie
Barracuda Managed XDR needed to make security-log investigations accessible beyond analysts who knew each vendor’s schema and could write SQL. Its AI-Powered Log Search integrates Databricks Genie into the XDR dashboard: Genie translates natural-language questions into SQL, runs them against customer logs, and returns tabular results with the generated query. Security-specific metadata in Unity Catalog helps map requests across network, server, cloud, email, and endpoint sources, while multi-turn context supports iterative filtering. Secure views enforce organization-level filtering before execution; scoped service principals and SQL validation add further controls against cross-tenant access. Barracuda reports reducing routine investigation time from hours to minutes and estimates that self-service could free 83 to 100 SOC analyst hours monthly, or 1,000 to 1,200 annually.
Barracuda XDR Engineering TeamScaling Security Alert Triage With Specialized Agents on Databricks
Databricks describes a security-alert triage system designed to review low-severity alerts that historically received less attention because of their volume and low fidelity. An initial single-agent prompt escalated 50% of alerts, so the team built a fleet of 17 source-specific agents plus a Threat Intelligence agent, using Structured Streaming to ingest alerts, enrich evidence, and route each alert. The pipeline combines deterministic benign-signal filtering, historical and behavioral context, specialized prompt functions, optional tools, structured dispositions, and cost controls before sending escalations to human analysts. Those analysts provide ground truth through MLflow traces and labels, supporting ongoing evaluation across escalate, monitor, and close decisions. After triaging more than 18,000 alerts, the system reached a 3.2% escalation rate, a 10.5-second median triage time, and more than 6,500 analyst hours saved in 30 days; escalated low-severity alerts were roughly 10 times more likely to be true positives than HIGH and MEDIUM alerts.
Leanne Shapton, Connor Hanify, Sam PezzinoWhat’s new in Databricks Data + AI Platform security and compliance at Data + AI Summit 2026
At Data + AI Summit 2026, Databricks announced security and compliance capabilities for scaling Genie, Lakebase, serverless workloads, and AI-powered applications without relying solely on manual provisioning, static network controls, or siloed compliance programs. Automatic Identity Management (AIM) for Microsoft Entra ID is generally available on AWS and Google Cloud, AIM for Okta is in Public Preview, and Context-Based Ingress is in Public Preview across all three clouds for policies based on network source, identity, and access scope. Private Network Gateway, in Private Preview on Azure Databricks, provides one secure connection from serverless workloads to private networks, while expanded Private Link support extends to Lakebase and other services. Compliance additions include Azure Serverless coverage, HITRUST across AWS, Azure, and Google Cloud, ISMAP on Azure and AWS, expanded AWS GovCloud availability, and planned FedRAMP High support on Azure Commercial.
Jason Wu, Samrat Ray, Filippo Seracini, Alex Esibov, Vijay Raja, Kelly Albano, Robert Zhang, Mia Penfold LopezBuilding an open ecosystem for AI governance with Unity AI Gateway
Databricks announced the Unity AI Gateway partner ecosystem, extending enterprise AI governance beyond models to runtime interactions among models, agents, MCP servers, skills, and AI tools. Built on Unity Catalog, the gateway lets organizations apply policies, monitor activity, manage spend, and govern AI across providers and frameworks, while integrating security, identity, and governance products they already use. The announcement groups the integrations into runtime AI security, observability and guardrails; agent identity and access governance; and AI observability and risk monitoring. Named integrations include Alice, CrowdStrike Falcon AI Detection and Response, Cyera, HiddenLayer, Netskope, Noma Security, Obsidian Security, Openlayer, Okta, Ping Identity, SailPoint, and Saviynt, with described capabilities including prompt-injection detection, data-loss prevention, agent discovery, authorization, and lifecycle governance.
David Nasi, Kelly Albano, Ashish KathapurkarIntroducing Omnigent: A Meta-Harness to Combine, Control and Share Your Agents
Omnigent is an open-source meta-harness designed to combine, control, and share agents across different harnesses, models, and interfaces. Databricks says users currently juggle multiple agents and copy context between them, while builders struggle to combine or replace harnesses with incompatible interfaces. Its runner wraps terminal-based agents and SDKs in sandboxed sessions with a uniform API for messages, files, streamed text, and tool calls; a server adds policies, sharing, and access through terminal, web, mobile, native Mac OS, and APIs. Features include live collaboration, hosted sandbox execution, contextual security and cost policies, OS isolation, and multi-harness authoring. Released in alpha under Apache 2.0, Omnigent is intended to provide a durable layer above changing agents and harnesses.
Matei Zaharia, Kasey Uhlenhuth, Corey ZumarHow to safeguard AI workloads with Unity AI Gateway Guardrails
Unity AI Gateway’s beta LLM Guardrails extend earlier Gateway guardrails with LLM-powered evaluation, pre-built protections, and tunable custom rules for AI security, compliance, and sensitive-data protection. An Acme marketing scenario maps policies to PII redaction on inputs, jailbreak and prompt-injection blocking, unsafe-content blocking on outputs, and a custom rule that blocks competitor references. Teams configure these guardrails on an endpoint, optionally use Log mode, select evaluator endpoints, and test representative requests while tracking activity in inference tables. Built-in controls behave as expected in the examples, while the custom rule initially performs unreliably until a more specific prompt and gpt-5-4-mini evaluator improve triggering without degrading other tests. Inference tables record request outcomes, evaluator calls, latency, statuses, token counts, and shared request_id values for auditing and refinement.
Tim LortzResponsible AI Governance: A Practical Framework for Business Leaders
Responsible AI governance is presented as an operational framework for leaders overseeing systems that can produce biased outputs, expose sensitive data, and create regulatory, financial, or reputational harm. It draws on the NIST AI RMF and OECD AI principles, maps to EU AI Act requirements, and uses human dignity, fairness, privacy, accountability, transparency, and security as governance values. The program starts with a living inventory recording purpose, ownership, training-data sources, affected populations, review dates, model lineage, and third-party status, followed by risk classification and assessments based on potential impact. It calls for lifecycle controls including bias mitigation, security testing, human review, drift monitoring, audits, incident exercises, and confidential concern reporting. The roadmap recommends piloting governance on a highest-risk product line, scaling controls across business units, and reviewing the framework annually or after major incidents, regulatory updates, or portfolio changes.
Databricks StaffA Modern AI Risk Management Framework
AI systems introduce probabilistic behavior, model drift, adversarial manipulation, bias, and auditability challenges that traditional IT risk practices may not address. The framework described combines NIST AI RMF, EU AI Act, and ISO/IEC 23894:2023, using Govern, Map, Measure, and Manage as its operational structure while recognizing regulatory and societal context. It recommends cross-functional governance, clear risk ownership, an AI-BOM covering systems, data flows, dependencies, and accountability, plus continuous measurement of fairness, explainability, data quality, security vulnerabilities, and harm likelihood and severity. Across the AI lifecycle, the text identifies risks including data poisoning, model drift, malicious library injection, prompt injection, hallucinations, and platform vulnerabilities, and points to the AI RMF Playbook for checklists, timelines, and governance artifacts.
Databricks Staff
