---
title: "Databricks"
description: "Data and AI platform for data engineering, analytics, machine learning, and generative AI."
---

# Databricks
> Data and AI platform for data engineering, analytics, machine learning, and generative AI.

## Articles

### [Stop rogue AI: How Unity Catalog secures your agent actions](https://yomu.fyi/post/stop-rogue-ai-how-unity-catalog-secures-your-agent-actions.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ahmed Bilal
- Published: May 19, 2026
- Type: Problem & solution

Agentic AI systems connected to external tools can take destructive production actions while operating within delegated authority, yet without restrictions on tool selection or records of their calls. Unity Catalog now governs MCP tools with fine-grained access control, service policies, and payload logging, while Unity AI Gateway enforces decisions in real time. Policies are SQL functions receiving actor and context information; they can allow, deny, or require user consent, inspect arguments and caller properties, and block calls before execution. Every tool call is recorded in a Unity Catalog-managed Delta table with the tool name, arguments, result, user identity, and allow-or-deny status, making the activity queryable with SQL. The service policies and payload logging are available as a Gated Beta, with early access offered through Databricks account teams.


### [Databricks context engineer associate: the industry’s first certification for reliable AI agent systems](https://yomu.fyi/post/databricks-context-engineer-associate-the-industry-s-first-certificati.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Rachel Canetta, James Kantor, Trang Le
- Published: May 19, 2026
- Type: Announcement

Databricks announces the Databricks Certified Context Engineer Associate, described as the industry’s first certification purpose-built for context-aware AI systems. It assesses whether practitioners can design, assemble, and govern the information AI agents receive at inference time, including system prompts, retrieval, memory, external tools, context-window management, and governance. Candidates work with Databricks technologies such as AI Search, Lakebase, MLflow, MCP, and Unity Catalog, while advanced coverage includes multi-agent systems, long-horizon workflows, and evaluation of context decisions. The beta exam will be available free to Data + AI Summit attendees, who may take it once; beta results are expected in 6–8 weeks.


### [How to safeguard AI workloads with Unity AI Gateway Guardrails](https://yomu.fyi/post/how-to-safeguard-ai-workloads-with-unity-ai-gateway-guardrails.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Tim Lortz
- Published: May 19, 2026
- Type: Tutorial

Unity AI Gateway’s beta LLM Guardrails extend earlier Gateway guardrails with LLM-powered evaluation, pre-built protections, and tunable custom rules for AI security, compliance, and sensitive-data protection. An Acme marketing scenario maps policies to PII redaction on inputs, jailbreak and prompt-injection blocking, unsafe-content blocking on outputs, and a custom rule that blocks competitor references. Teams configure these guardrails on an endpoint, optionally use Log mode, select evaluator endpoints, and test representative requests while tracking activity in inference tables. Built-in controls behave as expected in the examples, while the custom rule initially performs unreliably until a more specific prompt and gpt-5-4-mini evaluator improve triggering without degrading other tests. Inference tables record request outcomes, evaluator calls, latency, statuses, token counts, and shared request\_id values for auditing and refinement.


### [What’s new in Unity AI Gateway: service policies, guardrails, observability, and cost controls for AI agents and MCPs](https://yomu.fyi/post/what-s-new-in-unity-ai-gateway-service-policies-guardrails-observabili.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: David Nasi, Kelly Albano
- Published: May 19, 2026
- Type: Announcement

Unity AI Gateway is expanding runtime governance for production AI agents, model calls, and MCP tool interactions as teams face rising costs, unclear behavior, and limited control. The Beta adds LLM-based guardrails, token-level cost attribution with per-user alerts and hard budget limits, payload logging, and MCP service policies. Guardrails use a model and prompt to evaluate inputs, outputs, or both in real time, while inference tables and Unity Catalog system tables centralize governed records of usage and interactions. For MCPs, administrators can define SQL policies as Unity Catalog functions using agent identity, user context, and request parameters to constrain tool access and sensitive actions. The capabilities are available in Beta, with payload logging and service policies offered through gated enrollment, and are intended to improve observability, compliance, and cost control.


### [MemEx: A Programmable Scratchpad for LLM Agents](https://yomu.fyi/post/memex-a-programmable-scratchpad-for-llm-agents.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: The Databricks AI Research Team
- Published: May 19, 2026
- Type: Explainer

MemEx addresses the context-window bottleneck in LLM agents, where large tool outputs persist in the shared prompt even when later steps need only a small portion. It gives the agent a persistent, typed Python kernel: tool results remain as objects in scope, code can filter, summarize, and compose them, and only printed values are materialized as context tokens. MemEx preserves existing Tool Calling schemas while adding typed submit(), persistent state, backend-agnostic execution, and asynchronous spawn\_agent() sub-agents, enabling multi-step workflows and parallel trajectory analysis. Across nine-model evaluations on OfficeQA Pro and Enterprise Structured Retrieval, it was reported as better and cheaper than parallel structured Tool Calling; examples include roughly 10-point OfficeQA gains for some models and Qwen structured-retrieval accuracy rising from 18% to 36% and 20% to 38% at lower token cost.


### [How Deutsche Börse built a generative AI tool to tackle the large-scale migration of   Zeppelin notebooks to Databricks](https://yomu.fyi/post/how-deutsche-bo-rse-built-a-generative-ai-tool-to-tackle-the-large-sca.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Evan Pandya, Tobi Wole-Fasanya
- Published: May 19, 2026
- Type: Problem & solution

Deutsche Börse Group is migrating a high volume of Zeppelin notebooks used by more than 2,000 users from Cloudera to Databricks, after Cloudera’s 2027 Zeppelin decommissioning and the group’s move to cloud analytics. Because the notebooks combine SQL, Python, custom interpreters, data-source references, visualizations, widgets, and scheduling logic, the team rejected fully rule-based rewriting. Its Zeppelin to Databricks Notebook Converter, delivered as a Databricks App, deterministically maps paragraphs to cells, translates interpreter syntax, reformats metadata as .ipynb JSON, and preserves original content. A generated, environment-specific prompt then gives Genie context about interpreters, data sources, and configuration patterns so it can reconstruct logic, ask clarifying questions, and handle the variable work. The hybrid workflow reduced redevelopment to 15–20 minutes per notebook, while keeping SQL, Python, visualizations, widgets, Oracle and HDFS references, scheduling, and custom business logic untouched; large-scale testing is next.


### [Announcing the Databricks analytics engineer learning pathway](https://yomu.fyi/post/announcing-the-databricks-analytics-engineer-learning-pathway.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Maroua Lazzarou, Pratyarth Rao
- Published: May 18, 2026
- Type: Announcement

Databricks is launching the Analytics Engineer Learning Pathway, a curriculum for SQL practitioners that covers data modeling, pipelines, metrics, and Genie spaces on the lakehouse. The pathway teaches learners to transform raw data into governed, AI-ready semantic models and metric views, the foundation for analytics, dashboards, and AI agents. Courses cover Analytics Fundamentals, production SQL ETL with Materialized Views, Streaming Tables, AUTO CDC, and Lakeflow Jobs, plus data modeling with Delta Lake and Unity Catalog. The curriculum also addresses metric views, Genie spaces, governance with Unity Catalog permissions and ABAC policies, and Spark Declarative Pipelines with expectations, event logs, and metrics. Courses are available in self-paced and instructor-led formats through Databricks Academy, with the full pathway included in active learning subscriptions.


### [The question your commercial data should already be able to answer](https://yomu.fyi/post/the-question-your-commercial-data-should-already-be-able-to-answer.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Christina Busmalis, Tony Farias
- Published: May 18, 2026
- Type: Announcement

In ATTR-CM, a progressive and often fatal form of heart failure, 70 to 80 percent of patients do not know they have the disease, creating a need for better commercial prioritization than static call lists provide. Databricks and Veeva position embedded Genie agents and AI/BI dashboards in Veeva Vault CRM as a way to connect commercial data bidirectionally and answer role-specific questions inside existing workflows. Sales reps can see geographic HCP views with suspected patient counts, formulary scores, office availability, NRx-weighted priorities, and talking points, then rebuild a day after a cancellation. MSLs can generate cited pre-call briefs from approved sources such as Veeva Link, PubMed, ClinicalTrials.gov, and ASNC guidelines. Territory managers receive personalized views of call patterns, unworked signals, and dormant HCPs, while Unity Catalog provides shared access, lineage, and compliance governance.


### [PipelineIQ: Forward‑Looking Sales Intelligence That Drives Action](https://yomu.fyi/post/pipelineiq-forward-looking-sales-intelligence-that-drives-action.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Sam Le Corre, Dael Williamson, Luis Herrera
- Published: May 15, 2026
- Type: Problem & solution

PipelineIQ addresses the administrative drain and unreliable revenue predictability caused by incomplete, inconsistent, and backward-looking CRM data in B2B sales. Rather than build another forecasting system, it applies prescriptive analytics to identify forward signals and produce immediate actions for reps and managers. Built on Databricks, it uses Foundation Model APIs, Unity Catalog, Delta Lake, and AI/BI Dashboards; its confidence scorer sends CRM fields to ai\_query() with a Gemma 3 12B model, scores eight MEDDPICC dimensions from 0–10, and limits missing fields to scores of 3 or below. Weighted confidence is refreshed daily, with a fail-safe override to Low when a use case has more than three active blockers. Dashboards and Genie queries connect evidence-based risk explanations, remediation steps, and portfolio views to sales execution.


### [Backstage with Lakebase, part 2](https://yomu.fyi/post/backstage-with-lakebase-part-2.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Cameron Casher, Kevin Hartman, Surya Sai Turaga
- Published: May 15, 2026
- Type: Explainer

Part 2 shows how moving Backstage’s operational Postgres database to Databricks Lakebase brings security, auditing, masking, and cost controls under Unity Catalog. The proof of concept used Lakehouse Federation to expose the catalog as the lakebase\_bs foreign catalog, replacing cross-service RDS investigations with Unity Catalog grants and system.access.audit records. Audit events identify OAuth users and source IPs, while billing attributed 31.6130 DBU to production and 0.0107 DBU to a dropped test branch. Branch-level attribute masking propagates automatically to feature, CI, and QA copies, while the post cites a Perforce report stating that 60% of organizations experienced breaches or theft in non-production environments. It also introduces LakebaseOps, with three agents and seven scheduled jobs, and Lakebase MCP, exposing 46 governed tools across four profiles, so DBAs can design policies and workflows instead of handling repetitive provisioning.


### [Expanded interoperability with Unity Catalog Open APIs](https://yomu.fyi/post/expanded-interoperability-with-unity-catalog-open-apis.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Alex Jiang, Tathagata Das
- Published: May 14, 2026
- Type: Announcement

Unity Catalog’s expanded Open APIs address data silos by letting organizations use multiple compute engines while retaining centralized governance and a single copy of data. In beta, Apache Spark, Apache Flink, and DuckDB can create, read, write, and stream to or from UC managed Delta tables, with catalog commits providing serialized commits, transactional safety, and auditability. Delta Kernel, an open source Java and Rust library, abstracts low-level protocol details, helping connectors integrate external writes with catalog-managed commits while Predictive Optimization continues to run on accessed tables. Credential vending, now GA for tables, issues short-lived, scoped cloud credentials and supports M2M OAuth plus automatic refresh; volume credential vending is in Public Preview for unstructured data. The roadmap includes functionality for fine-grained row- and column-level ABAC on external reads, while external managed-table access remains in beta.


### [From manual to autonomous: how AI agents are transforming electric grid operations](https://yomu.fyi/post/from-manual-to-autonomous-how-ai-agents-are-transforming-electric-grid.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Julien Debard, Edward Tavares
- Published: May 14, 2026
- Type: Explainer

Electric utilities are facing rising demand, retiring generation, extreme weather, aging infrastructure, and fragmented operational data that manual processes cannot manage at scale. AI agents are presented as a human-centered alternative that synthesizes heterogeneous data, learns from outcomes, and progresses from human-approved recommendations to exception-based control and eventually autonomous operations within defined parameters. Hawaiian Electric used a Retrieval Augmented Generation proof-of-concept with Databricks AI Search, Unity Catalog, and Lakeflow Declarative Pipelines to query regulatory documents and provide page-specific citations. The system reduced response times from five minutes to five seconds and was implemented in two weeks, while the article describes broader potential for predictive maintenance, outage response, load forecasting, and customer service.


### [Data quality is the AI strategy](https://yomu.fyi/post/data-quality-is-the-ai-strategy.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Aly McGue
- Published: May 13, 2026
- Type: Explainer

NYU Langone Health’s AI strategy starts with data quality, arguing that healthcare AI cannot be reliable when source data is fragmented or inconsistent. The institution standardized on common transactional platforms, including one electronic health record and one ERP system, established authoritative data sources, and fixes data at the source rather than mapping it in the warehouse layer. Its Databricks-based unified data and AI platform, with Unity Catalog, supports clinicians, analysts, scientists, and corporate users across care, operations, and research, while real-time feeds power emergency-room decision-support models. Mherabi also describes a three-layer analytics model: structured visualizations, conversational tools such as Genie, and answers delivered in formats suited to the user. The stated conclusion is that upstream data discipline, governance, literacy, and adaptable platforms provide the foundation for trustworthy AI and timely clinical insight.


### [The Rosetta stone of CPS: Claroty’s AI-powered library](https://yomu.fyi/post/the-rosetta-stone-of-cps-claroty-s-ai-powered-library.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ben Hazan, Anton Berlinsky, Ohad Avni, Itay Wagner, Guy Zalcman, Dor Bdolach, Ravid Ariely, Gal Sberro
- Published: May 13, 2026
- Type: Problem & solution

CPS asset identification is difficult because 88% of CPS assets do not transmit an exact product code and 76% use product codes that differ from the vendor's official records. Claroty's AI-Powered CPS Library addresses this entity-resolution problem with a hybrid architecture that combines classic matching, statistical inference, generative AI, NLP, reasoning agents, and human-in-the-loop review to reconcile protocol data, vendor records, manuals, and vulnerability information. Its Databricks-based lakehouse uses Delta Lake, Unity Catalog, Spark pipelines, Model Serving, MLflow evaluations, Lakebase, and Databricks Apps to maintain governed, auditable mappings across a catalog of more than 17 million assets. The post reports 25% improved vulnerability-attribution accuracy and says early tests gave 56% of analyzed devices new or updated recommendations for outdated firmware, while low-confidence mappings remain subject to expert review.


### [Clinical operations intelligence belongs on the Lakehouse](https://yomu.fyi/post/clinical-operations-intelligence-belongs-on-the-lakehouse.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Nicholas Siebenlist, Surya Sai Turaga
- Published: May 13, 2026
- Type: Problem & solution

The Site Feasibility Workbench addresses fragmented clinical operations data by placing a site-selection decision-support application, its models, and data on Databricks. Released as a fully open-source Databricks App, it combines a FastAPI backend, React frontend, Unity Catalog-governed data, Lakebase operational state, and AI/BI Genie within the workspace. TA-segmented LightGBM models use the organization’s CTMS, EDC, and IRT history alongside real-world evidence, patient access, qualification, Open Payments, and protocol-execution signals to rank sites. Each prediction stores SHAP attributions in a governed Unity Catalog Delta table, while MLflow tracks training runs and model versions; saved shortlists persist to Lakebase and Genie answers governed-data questions. The six-step workflow supports protocol selection through final shortlisting, including diversity considerations; the release is a decision-support layer rather than a source of record, and deployment reportedly takes about 30 minutes before security review and validation.


### [ABAC row filtering and column masking policies, governed tags, and data classification are now generally available in Unity Catalog](https://yomu.fyi/post/abac-row-filtering-and-column-masking-policies-governed-tags-and-data.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Adriana Ispas, Kristen Wilder, Jacqueline Li, Corey Sunwold, Menglei Sun, Viswesh Periyasamy
- Published: May 13, 2026
- Type: Announcement

Unity Catalog now generally offers three complementary data-governance capabilities: Attribute-Based Access Control (ABAC) policies for row filtering and column masking, Governed Tags, and agentic Data Classification. They address per-object access rules, coordination gaps, and manual detection by letting governance teams define tag-based policies once, automatically classify sensitive data, and protect matching objects across catalogs and schemas. Governed tags provide an account-level vocabulary inherited across catalogs, schemas, tables, and columns, while ABAC applies row filters and column masks using tag-based conditions. Classification uses built-in compliance classifiers, custom classifiers, metadata, pattern recognition, and large language models, with human-in-the-loop validation and false-positive exclusions. General availability adds 10x larger policy limits, support for 10,000+ policies per metastore, lifecycle management through SQL, APIs, UI, and Terraform, expanded compliance coverage, and custom classifiers in beta.


### [How CFOs in consulting can recover margin with Databricks](https://yomu.fyi/post/how-cfos-in-consulting-can-recover-margin-with-databricks.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Rajaram Suresh, Kateryna Savchyn, Pavithra Rao, Mark Hahn, Kim Hatton, Sharmey Shah, Lisa Fox
- Published: May 12, 2026
- Type: Problem & solution

Consulting firms are facing margin pressure despite growing deal pipelines, with 2024 EBITDA margins at 9.8%, utilization at 68.9%, and revenue growth at 4.6%. The post attributes this to fragmented finance workflows spanning Salesforce, SAP, Workday, Concur, and spreadsheets, plus inconsistent metrics and manual reconciliation. It proposes using Databricks to create a governed data layer with Lakeflow, Lakehouse, and Unity Catalog, then adding Genie for natural-language analysis and finance apps that trigger workflows and write approvals back in real time. A global consulting firm reportedly cut cash forecasting cycles by 3–5 days and reduced reporting-cycle FTE hours by 80%, while Databricks’ own close fell from 15 days to 8. The proposed rollout spans 10–12 weeks and begins with one tightly scoped workflow such as revenue close, unbilled WIP, T&E leakage, or practice-margin analysis.


### [The Convergence of Open Table Formats and Open Catalogs: Catalog Commits is Generally Available](https://yomu.fyi/post/the-convergence-of-open-table-formats-and-open-catalogs-catalog-commit.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Benjamin Mathew, Michelle Leon, Lukas Rupprecht, Ryan Johnson
- Published: May 12, 2026
- Type: Announcement

Catalog Commits is generally available for Unity Catalog managed Delta tables, aligning Delta with Iceberg’s catalog-oriented model and making the catalog responsible for table discovery, access, and latest table state. The change addresses three coordination problems: metadata “split brain” when engines write directly to storage, fragmented multi-engine governance, and the historical inability to coordinate atomic writes across multiple tables. With Catalog Commits enabled, Unity Catalog brokers table access through standardized APIs, keeping catalog and table state synchronized and enabling consistent authorization, holistic auditability, automated optimizations, and multi-statement, multi-table ACID transactions on Databricks. The release supports Databricks products and engines including Delta Spark, Delta Flink, Starburst Trino, DuckDB, and StreamNative, while Delta Kernel provides a shared path for connector support.


### [Announcing Databricks student fellows](https://yomu.fyi/post/announcing-databricks-student-fellows.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Trang Le, Elise Hollowed, Joe Nash
- Published: May 12, 2026
- Type: Announcement

Databricks is launching the Student Fellows Program, an opportunity for university and college students interested in computer science, AI, and data engineering to become leaders in their campus communities. Fellows receive specialized training from Databricks experts, build hands-on skills with the Databricks platform, and organize campus activities such as hackathons, tech talks, and study groups while connecting peers with resources. The initiative also offers opportunities for free or discounted certification exam vouchers and practical experience intended to support applications for internships with Databricks, its customers, or partners. Student Fellows’ primary mission is to foster a community of learners and serve as a bridge between Databricks and their university communities, with the first cohort currently accepting applications.


### [Faster Queries and New Capabilities with the Open-Source Databricks JDBC Driver](https://yomu.fyi/post/faster-queries-and-new-capabilities-with-the-open-source-databricks-jd.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Toussaint Webb, Gopal Lal, Kaitlin Baumgardner
- Published: May 12, 2026
- Type: Announcement

The open-source Databricks JDBC driver 3.x and later releases address connectivity needs affecting dashboards, spreadsheets, applications, and other data workflows. Compared with the legacy 2.x driver, they deliver up to 30% faster retrieval of large query results while preserving backward compatibility. Architectural changes add full Arrow-based transfer for JDK 16+, a first-class asynchronous execution API, stream-based ingestion into Databricks Volumes, and integration with the Statement Execution API. The driver also supports stored procedures, multi-statement transactions, Unity Catalog metric views, query tags, geospatial and complex data types, plus built-in client telemetry for query latency, metrics, and errors. Its open-source ownership and Databricks maintenance are presented as enabling faster fixes, new features, transparency, contributions, and tighter platform integration.


[Newer posts](https://yomu.fyi/company/databricks/page/22.md) · [Older posts](https://yomu.fyi/company/databricks/page/24.md)