---
title: "Databricks"
description: "Data and AI platform for data engineering, analytics, machine learning, and generative AI."
---

# Databricks
> Data and AI platform for data engineering, analytics, machine learning, and generative AI.

## Articles

### [How security teams can report cyber risk to boards](https://yomu.fyi/post/how-security-teams-can-report-cyber-risk-to-boards.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Taylor Kain
- Published: May 22, 2026

Boards are seeking visibility into cyber risk, but technical reports often fail to connect security posture with business impact or financial exposure. The post explains that compliance and cyber risk leaders can use Databricks Genie to query vulnerability posture, asset criticality, threat intelligence, control data, and historical incident costs in a governed environment. It recommends probabilistic financial modeling, including Monte Carlo simulation, to run randomized attack scenarios and produce loss distributions; Value-at-Risk framing can make those results familiar to directors. This approach replaces qualitative red/amber/green reporting with expected-loss ranges, supports investment prioritization, and enables trend analysis and board-ready answers, while the suggested cadence combines quarterly strategic briefings, monthly operational reviews, and incident-triggered updates.


### [Transforming industries with conversational AI: Partner solutions built on Databricks Genie](https://yomu.fyi/post/transforming-industries-with-conversational-ai-partner-solutions-built.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Amit Singh
- Published: May 21, 2026

Databricks presents a first cohort of partner-built industry solutions that use Genie to reduce the “Analyst Bottleneck,” where leaders wait for custom SQL or dashboard updates. Genie provides a conversational analytics layer grounded in Unity Catalog metadata and business semantics, using specialized AI agents to return governed, secure, traceable answers to natural-language questions. The featured accelerators span communications, media and entertainment, financial services, healthcare and life sciences, manufacturing and energy, public sector, and retail, with examples including churn prediction, next-best-action recommendations, billing anomaly investigation, KYC monitoring, actuarial analysis, and commercial insights. Across the examples, solutions combine domain expertise with Databricks platform capabilities such as machine learning, Agent Bricks, Lakebase, Serverless SQL, and auditable governance to support self-service analysis and operational decisions.


### [From emissions reporting to decarbonization decisions](https://yomu.fyi/post/from-emissions-reporting-to-decarbonization-decisions.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Caitlin Gordon
- Published: May 21, 2026

Energy companies have built sophisticated infrastructure for Scope 1, 2, and 3 emissions reporting across complex asset portfolios, but that infrastructure is often oriented toward historical disclosure rather than forward-looking action. Sustainability leaders still need timely answers about which assets warrant intervention, what operational choices drive carbon intensity, and whether current performance is tracking toward commitments. Databricks Genie addresses this gap by letting users query emissions, operational, and financial data in natural language, linking results to dispatch decisions, fuel purchases, and asset utilization. It supports multi-scope analysis, traceability to source data, and scenario questions such as retiring an asset or increasing renewable PPA volume. The stated outcome is a shift from compliance reporting toward data-driven decarbonization decisions, with Genie available today for energy-sector use.


### [You’ve built the media products, now make them personalized](https://yomu.fyi/post/you-ve-built-the-media-products-now-make-them-personalized.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Elena Tesser
- Published: May 21, 2026

Media companies may have launched streaming services, digital editions, and mobile apps, yet still face a digital product intelligence gap: product teams must answer behavioral data questions quickly enough to personalize experiences and improve engagement. Databricks Genie gives nontechnical product leaders a conversational interface that translates natural-language questions into SQL queries, visualizations, and actionable insights over governed enterprise data, without requiring code or analyst handoffs. The agent queries governed Delta Lake tables fed by streaming clickstreams, views, and session signals, combining event-level behavior, A/B tests, audience segments, and cross-platform data in conversational answers. The source says internal benchmark accuracy improved from 32% to over 90% through multi-LLM orchestration, specialized knowledge search, and parallel reasoning, and presents Genie as reliable enough for production personalization decisions.


### [From "What Happened?" to "What Will Happen?"](https://yomu.fyi/post/from-what-happened-to-what-will-happen.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ryuta Yoshimatsu, Javier Poveda Panter, Dominik Safaric, Philipp Singer, Diana Kriuchkova, Sauraj Gambhir, Dael Williamson, Bryan Smith
- Published: May 21, 2026

Databricks Genie makes descriptive analytics accessible in natural language, but predictive questions still require specialized data science workflows and carefully prepared datasets. This post presents a multi-agent supervisor deployed as a Databricks App, combining Genie, TabPFN, and Agent Bricks to turn business questions into predictions. The orchestrator asks Genie to use governed Lakehouse data, schemas, relationships, and semantics to generate labeled training data through SQL, then sends it to TabPFN, which predicts in a single forward pass without feature preprocessing, model selection, or hyperparameter tuning. The resulting conversational experience supports descriptive and predictive analytics with Unity Catalog lineage and access control, while an MLflow GenAI evaluation harness monitors reliability and regressions. Its central limitation is that predictions depend on Genie producing a meaningful dataset with a clear label, so missing signals, joins, outcomes, or agent omissions can make results unreliable.


### [Unlock seamless and cost-effective marketing campaigns with Lakebase](https://yomu.fyi/post/unlock-seamless-and-cost-effective-marketing-campaigns-with-lakebase.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Thomas Nguyen
- Published: May 20, 2026

Retail marketing teams often store customer segments in OLTP databases, creating underused capacity between campaigns and synchronization work whenever new segments are requested. Lakebase, Databricks’ implementation of a lakebase architecture, separates storage from compute and uses elastic, serverless Postgres that can scale to zero when idle and up for bursty campaign traffic. The integration with the Lakehouse uses managed Synced Tables for customer segments and Lakehouse Sync for continuous CDC-based replication of operational data into Unity Catalog Delta tables. The SAP Engagement Cloud example configures a Lakebase Autoscaling project, native Postgres credentials, an ISRG Root X1 certificate, and snapshot synchronization for datasets with more than 10% updated. The post concludes that this design lowers idle and sizing costs, reduces pipeline maintenance, and supports low-latency point lookups, while warning that Lakebase is not optimized for large scans or classic OLAP.


### [Governing AI agents at scale with Unity Catalog](https://yomu.fyi/post/governing-ai-agents-at-scale-with-unity-catalog.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: David Nasi, Stefania Leone
- Published: May 20, 2026

With AI agents proliferating across organizational functions, governance must answer which agents access sensitive data and how they act. Traditional approaches either scatter logs and permissions across systems or restrict deployment, while agents require controls over autonomous tool use and runtime behavior. Unity Catalog and Unity AI Gateway address this through four pillars: delegated access, data-centric governance, cost intelligence, and open interoperability. The design passes user identity through agents with on-behalf-of tokens, evaluates MCP tool calls with Service Policies, applies inline guardrails, and records model, access, trace, and usage data in lakehouse tables. It also connects data quality, classification, spend, and outcomes, while supporting multiple frameworks and model providers through shared governance.


### [How telecom CFOs can make smarter network capex decisions with AI](https://yomu.fyi/post/how-telecom-cfos-can-make-smarter-network-capex-decisions-with-ai.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Elena Tesser
- Published: May 20, 2026

Telecom CFOs face multi-year, multi-billion-dollar network capex decisions involving spectrum, fiber, and 5G densification amid technology uncertainty and competitive dynamics. The post argues that operators can improve allocation by unifying network quality, customer churn, ARPU, billing, and investment history in a queryable environment. Databricks Genie provides a governed natural-language interface over that enterprise data, allowing finance leaders without SQL or data engineering backgrounds to ask cross-domain questions at geographic granularity and receive answers from systems of record. Example analyses compare ARPU and churn after 5G densification with markets still on the deployment roadmap, while scenario modeling can test accelerated deployment against historical returns. The stated outcome is better evidence for strategic judgment, shifting conversations from generic industry benchmarks toward comparable investments in the operator’s own network.


### [How Databricks Genie improves retail personalization](https://yomu.fyi/post/how-databricks-genie-improves-retail-personalization.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Sarah Duffy
- Published: May 20, 2026

Retail personalization often stalls when CX leaders must wait for analysts to answer questions about segment behavior, loyalty performance, channel preference, or churn risk, narrowing the window for action. Databricks Genie addresses this access gap by letting business users query unified structured and unstructured enterprise data in plain English instead of SQL. Its retail differentiators include identity-resolved queries across channels and devices, lifecycle-stage awareness, campaign response and control-group data integration, and privacy controls enforced through Unity Catalog. The source says Genie reduces routine analyst requests rather than replacing data science teams, while enabling merchandisers, category managers, loyalty marketers, and CX leaders to self-serve operational questions. It cites 7-Eleven’s use of Databricks SQL, Unity Catalog, and AI/BI Genie to launch, refine, and measure personalized offers within a secure, unified platform.


### [Databricks for Good and Virtue Foundation: Partnering to Connect Medical Volunteers to Critical Health Services in 72 Countries](https://yomu.fyi/post/databricks-for-good-and-virtue-foundation-partnering-to-connect-medica.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Priyanka Mehta, Shaunak Sen
- Published: May 20, 2026

Virtue Foundation’s VF Match platform connects medical professionals with volunteer opportunities across 72 low and low-middle income countries, but scaling its initial LLM proof of concept into production required a robust data pipeline. The Databricks-based Foundational Data Refresh combines Overture Maps and Bright Data, processes more than 25 million web pages with targeted GPT extraction steps, and uses Spark, Photon, and Lakeflow Jobs to orchestrate checkpointed, modular workloads. Splink resolves duplicate facilities and nonprofits across messy sources; Photon reduced the slowest entity-resolution partition from 30 minutes to approximately two minutes, a 15x improvement. A prototype VF Agent adds natural-language access through LangGraph, Databricks Model Serving, AI Search, and Genie, while the resulting system provides an up-to-date global view of healthcare infrastructure and volunteer needs.


### [Introducing Arrow UDFs in PySpark: A Faster, Leaner Replacement for Pandas UDFs](https://yomu.fyi/post/introducing-arrow-udfs-in-pyspark-a-faster-leaner-replacement-for-pand.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ruifeng Zheng, Yicong Huang
- Published: May 20, 2026

Apache Spark’s Pandas UDFs improve Python UDF throughput through Arrow-based serialization and batching, but Pandas/Arrow conversion can add copies, limit zero-copy behavior, and weaken support for complex datatypes. Native Arrow UDFs, introduced in Databricks Runtime 18.0, operate directly on Arrow data without converting inputs to Pandas or NumPy objects, preserving columnar layout and enabling vectorized processing through Arrow’s compute and memory model. The new @arrow\_udf decorator supports scalar, aggregate, and table functions, while type-hinted @udf definitions, mapInArrow, and applyInArrow extend the interface to DataFrame and grouped operations; iterator-based modes can amortize initialization costs. In a comparison, an Arrow UDF ran about 10% faster than a Pandas UDF and used about 40% less memory, with better complex-datatype support.


### [Automate Data & KPI Monitoring with SQL Alerts](https://yomu.fyi/post/automate-data-kpi-monitoring-with-sql-alerts.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Srilekha Dornadula
- Published: May 19, 2026

Databricks SQL Alerts is generally available as an automated alternative to manually checking dashboards and rerunning monitoring queries for business KPIs, pipeline health, and data quality. Teams define a SQL query, evaluation condition, schedule, and notification destinations, then receive alerts when results cross configured guardrails; standalone alerts can also run within Lakeflow Jobs. GA supports authoring in the SQL editor, notifications through email, Slack, PagerDuty, Microsoft Teams, or webhooks, and production management through Git, Declarative Automation Bundles, APIs, Terraform, and SDKs. The post’s revenue example compares yesterday’s result with a seven-day average and triggers when revenue\_pct\_change is below -5%, while the SQL Alert task can expose OK, TRIGGERED, or ERROR to downstream pipeline logic.


### [How to Build Real-Time Fraud Detection using Spark Real-Time Mode and Lakebase](https://yomu.fyi/post/how-to-build-real-time-fraud-detection-using-spark-real-time-mode-and.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Sixuan He, Navneeth Nair
- Published: May 19, 2026

Card fraud operates in seconds, while organizations need to score and block suspicious transactions before settlement without adding a separate streaming stack. This Solution Accelerator presents an open-source, end-to-end implementation on Databricks, covering transaction ingestion, Spark Real-Time Mode processing, fraud scoring, Lakebase feature serving, and a Databricks Apps monitoring dashboard. RTM processes streaming workloads in milliseconds and targets sub-300ms latency, while Lakebase supplies a managed, serverless Postgres serving layer with sub-millisecond reads. The design keeps batch ETL, training, streaming, and governance within the same platform. Deployment progresses from a synthetic rate-source quick start to Kafka-based parsing, stateful per-card velocity tracking with TTL, enrichment, weighted scoring, routing, and optional MLflow-trained RandomForest scoring, followed by a Streamlit app for live decisions.


### [How Databricks Genie improves supply chain visibility with real-time AI analytics](https://yomu.fyi/post/how-databricks-genie-improves-supply-chain-visibility-with-real-time-a.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Caitlin Gordon
- Published: May 19, 2026

Supply chain leaders often have extensive operational data yet still respond reactively because predictive signals such as supplier lead-time trends, inventory velocity, weather, and commodity prices remain siloed and difficult to synthesize. Databricks Genie is presented as a plain-language intelligence layer that lets leaders interrogate operational and external data in the flow of work rather than relying on analyst-led BI sessions. Users can ask questions combining supplier tiers, lead-time changes, inventory coverage, contract terms, and production schedules, then follow with what-if queries about financial exposure if conditions worsen. The post says Genie returns answers in seconds, supports near-real-time monitoring and proactive alerting, and enables shared, governed answers for procurement, operations, and finance, shifting decisions from reactive reporting toward earlier evidence-based action.


### [A CFO’s guide to managing value-based care financial performance](https://yomu.fyi/post/a-cfo-s-guide-to-managing-value-based-care-financial-performance.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Adam Crown
- Published: May 19, 2026

Value-based care changes healthcare finance by tying payment to outcomes, cost efficiency, and equity rather than service volume, while shifting greater financial risk to providers under ACO, bundled, and capitated contracts. A CFO must therefore track attributed populations, PMPM cost trends, utilization, clinical outliers, quality thresholds, and contract performance against benchmarks. Databricks Genie is presented as a conversational interface over integrated attribution, claims, clinical, and benchmark data, allowing questions such as PMPM divergence among high utilizers and DRG drivers to be answered in real time. The source also identifies incorrect attribution and risk-adjustment coding errors as major threats, and describes predictive risk stratification and quality-gap forecasting as ways AI can support earlier intervention.


### [Stop rogue AI: How Unity Catalog secures your agent actions](https://yomu.fyi/post/stop-rogue-ai-how-unity-catalog-secures-your-agent-actions.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ahmed Bilal
- Published: May 19, 2026

Agentic AI systems connected to external tools can take destructive production actions while operating within delegated authority, yet without restrictions on tool selection or records of their calls. Unity Catalog now governs MCP tools with fine-grained access control, service policies, and payload logging, while Unity AI Gateway enforces decisions in real time. Policies are SQL functions receiving actor and context information; they can allow, deny, or require user consent, inspect arguments and caller properties, and block calls before execution. Every tool call is recorded in a Unity Catalog-managed Delta table with the tool name, arguments, result, user identity, and allow-or-deny status, making the activity queryable with SQL. The service policies and payload logging are available as a Gated Beta, with early access offered through Databricks account teams.


### [Databricks context engineer associate: the industry’s first certification for reliable AI agent systems](https://yomu.fyi/post/databricks-context-engineer-associate-the-industry-s-first-certificati.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Rachel Canetta, James Kantor, Trang Le
- Published: May 19, 2026

Databricks announces the Databricks Certified Context Engineer Associate, described as the industry’s first certification purpose-built for context-aware AI systems. It assesses whether practitioners can design, assemble, and govern the information AI agents receive at inference time, including system prompts, retrieval, memory, external tools, context-window management, and governance. Candidates work with Databricks technologies such as AI Search, Lakebase, MLflow, MCP, and Unity Catalog, while advanced coverage includes multi-agent systems, long-horizon workflows, and evaluation of context decisions. The beta exam will be available free to Data + AI Summit attendees, who may take it once; beta results are expected in 6–8 weeks.


### [How to safeguard AI workloads with Unity AI Gateway Guardrails](https://yomu.fyi/post/how-to-safeguard-ai-workloads-with-unity-ai-gateway-guardrails.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Tim Lortz
- Published: May 19, 2026

Unity AI Gateway’s beta LLM Guardrails extend earlier Gateway guardrails with LLM-powered evaluation, pre-built protections, and tunable custom rules for AI security, compliance, and sensitive-data protection. An Acme marketing scenario maps policies to PII redaction on inputs, jailbreak and prompt-injection blocking, unsafe-content blocking on outputs, and a custom rule that blocks competitor references. Teams configure these guardrails on an endpoint, optionally use Log mode, select evaluator endpoints, and test representative requests while tracking activity in inference tables. Built-in controls behave as expected in the examples, while the custom rule initially performs unreliably until a more specific prompt and gpt-5-4-mini evaluator improve triggering without degrading other tests. Inference tables record request outcomes, evaluator calls, latency, statuses, token counts, and shared request\_id values for auditing and refinement.


### [What’s new in Unity AI Gateway: service policies, guardrails, observability, and cost controls for AI agents and MCPs](https://yomu.fyi/post/what-s-new-in-unity-ai-gateway-service-policies-guardrails-observabili.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: David Nasi, Kelly Albano
- Published: May 19, 2026

Unity AI Gateway is expanding runtime governance for production AI agents, model calls, and MCP tool interactions as teams face rising costs, unclear behavior, and limited control. The Beta adds LLM-based guardrails, token-level cost attribution with per-user alerts and hard budget limits, payload logging, and MCP service policies. Guardrails use a model and prompt to evaluate inputs, outputs, or both in real time, while inference tables and Unity Catalog system tables centralize governed records of usage and interactions. For MCPs, administrators can define SQL policies as Unity Catalog functions using agent identity, user context, and request parameters to constrain tool access and sensitive actions. The capabilities are available in Beta, with payload logging and service policies offered through gated enrollment, and are intended to improve observability, compliance, and cost control.


### [MemEx: A Programmable Scratchpad for LLM Agents](https://yomu.fyi/post/memex-a-programmable-scratchpad-for-llm-agents.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: The Databricks AI Research Team
- Published: May 19, 2026

MemEx addresses the context-window bottleneck in LLM agents, where large tool outputs persist in the shared prompt even when later steps need only a small portion. It gives the agent a persistent, typed Python kernel: tool results remain as objects in scope, code can filter, summarize, and compose them, and only printed values are materialized as context tokens. MemEx preserves existing Tool Calling schemas while adding typed submit(), persistent state, backend-agnostic execution, and asynchronous spawn\_agent() sub-agents, enabling multi-step workflows and parallel trajectory analysis. Across nine-model evaluations on OfficeQA Pro and Enterprise Structured Retrieval, it was reported as better and cheaper than parallel structured Tool Calling; examples include roughly 10-point OfficeQA gains for some models and Qwen structured-retrieval accuracy rising from 18% to 36% and 20% to 38% at lower token cost.


[Newer posts](https://yomu.fyi/company/databricks/page/14.md) · [Older posts](https://yomu.fyi/company/databricks/page/16.md)