---
title: "Databricks"
description: "Data and AI platform for data engineering, analytics, machine learning, and generative AI."
---

# Databricks
> Data and AI platform for data engineering, analytics, machine learning, and generative AI.

## Articles

### [Smart Routing in Unity AI Gateway: Match frontier quality with 30%+ lower cost per task](https://yomu.fyi/post/smart-routing-in-unity-ai-gateway-match-frontier-quality-with-30-lower.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ankit Mathur, Ivan Zhou, Bryan Qiu, Rohit Agrawal, Elise Gonzales, Kelly Albano
- Published: Aug 13, 2026

Unity AI Gateway’s Smart Routing, now in Beta, addresses the cost and choice overload of assigning coding tasks to increasingly diverse models and harnesses. It uses task-aware routing: a small, low-latency model classifies the initial task description and metadata, then a policy selects a cheaper, medium, or frontier model; Omnigent can also choose the coding harness and route sub-agents. The design keeps consecutive turns on one model to preserve cache-hit rates, while escalating complex work and delegating simpler tasks to lower-cost options. On internal coding workloads, Smart Routing achieved 35% savings and outperformed any single model at 65% of Opus 5’s cost; public benchmarks reported matched Opus 5 performance with 56% savings. The team identifies mid-session reassessment, real-session evaluation, and affordable switching as continuing challenges.


### [Databricks Network Configuration delivery to Tens of Millions of Serverless VMs](https://yomu.fyi/post/databricks-network-configuration-delivery-to-tens-of-millions-of-serve.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Manish Bansal, Yankai Zhang, Chen He
- Published: Aug 12, 2026

Databricks’ serverless platform launches tens of millions of VMs daily, and each VM needs network configuration before serving customer workloads. The original design synchronously aggregated data from multiple upstream services on the cluster-creation critical path, producing 5,000-ms p99 RPC latency and availability bottlenecks as request volume reached billions per day. The replacement uses change events through a message queue, asynchronous per-workspace recomputation, partition-local snapshots, and periodic reconciliation to recover from missed events. The serving path now performs a single snapshot-store read without upstream calls, while events carry workspace and resource identifiers rather than customer data. After rollout, p99 RPC latency fell to 125 ms, server success rate rose to 99.99%, upstream call volume dropped 86%, and the legacy synchronous framework was deprecated.


### [How a major freight railroad scaled pipeline creation with Genie Code](https://yomu.fyi/post/how-a-major-freight-railroad-scaled-pipeline-creation-with-genie-code.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Dinesh Chandrasekaran, Subhadip Chanda, Julia Powell, Gal Oshri
- Published: Aug 12, 2026

A major Canadian freight railroad with roughly 20,000 route miles needed to modernize a decades-old analytical estate while handling hundreds of pipelines and rising demand for real-time analytics and AI. Using Databricks Genie Code, Unity Catalog, custom Agent Skills and a Streamlit Databricks App, the team created a metadata-grounded pipeline-generation workflow driven by compact YAML prompts. The workflow discovers schemas, maps source fields, and emits six artifacts—DDL, historical and streaming ingestion, incremental merges, and automated tests—using deterministic patterns for audit columns, deduplication, change-sequence guards and soft-delete reconciliation. Human-reviewed Source-to-Target Mapping preserves business interpretation, while Genie Code handles discovery, orchestration and artifact generation within governed Databricks execution. The reported result was more than 90% automation for new table ingestion, reducing delivery from days per table to minutes and supporting single, multi-table and bulk generation.


### [The Future of Data Analytics: Why AI is rewriting the Analyst’s Job Description](https://yomu.fyi/post/the-future-of-data-analytics-why-ai-is-rewriting-the-analyst-s-job-des.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Emma Stowell, Angus Morshead, Ogo Odili
- Published: Aug 12, 2026

The post argues that AI is not eliminating data analysts; it is automating the technical work that has crowded out their business impact. Natural-language tools such as Databricks AI/BI, Genie Code, and Genie One can generate dashboards, answer ad hoc questions, and accelerate tasks such as customer segmentation, which the post says one analyst completed in half a day instead of two months. This shifts the analyst’s focus toward problem framing, context, validation, storytelling, governance, and directing AI agents, while keeping intent, accountability, and decision ownership human. For organisations, the proposed response is to hire and develop curiosity, business acumen, and communication, embed analysts near decision-makers, retain human review, and measure decisions influenced rather than dashboards delivered.


### [Taking AUTO CDC to the next level: Solving the hardest real-world use cases](https://yomu.fyi/post/taking-auto-cdc-to-the-next-level-solving-the-hardest-real-world-use-c.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Josh Seidel, Shanelle Roman, Sudhanva Huruli
- Published: Aug 11, 2026

The post describes new AUTO CDC capabilities for challenges that standard CDC patterns do not fully address, including out-of-order bitemporal timelines, partial updates, and durable audit history. Bitemporal AUTO CDC tracks business and system time independently through four managed interval columns, rewriting affected history when corrections arrive out of order so either point-in-time view can be reconstructed. Partial Updates are generally available and treat selected incoming NULL values as "do not update," preventing unchanged fields from overwriting stored values. The post also describes logging business and system timestamps in MLflow for reproducibility after VACUUM, and notes that the Python API for AUTO CDC Type 1 is being contributed to Apache Spark 4.2, with a SQL interface already merged for a next release.


### [Open-sourcing Metals v2: Databricks’ Java and Scala language server for multi‑million line codebases](https://yomu.fyi/post/open-sourcing-metals-v2-databricks-java-and-scala-language-server-for.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ólafur Páll Geirsson, Iulian Dragos, Alessandro Patti
- Published: Aug 11, 2026

Databricks open-sourced Metals v2, a Scala and Java language server designed to provide low-latency code intelligence for multi-million-line Bazel monorepos, including its 26M-line repository. The project extends Metals v1 by reworking three layers: a build-free, content-addressed mbt repository index; compiler-backed Scala and Java pipelines using a Metals-provided sourcepath; and metadata-first BSP integration that keeps build synchronization off the startup path. Its Java implementation uses javac APIs and a modified Turbine header compiler, which processes close to one million Java lines per second on one thread and supports interactive analysis at nearly 100,000 lines per second in benchmarks. Metals v2 is available in Cursor, VS Code, and Neovim under Apache 2.0, while Databricks reports Cursor’s share of Scala and Java file-open events rising from 40% to 78%.


### [Modern Risk Demands a Real-Time Foundation: The CRO’s Mandate](https://yomu.fyi/post/modern-risk-demands-a-real-time-foundation-the-cro-s-mandate.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Amit Kumar Jha, Amee Vora, Andrea DeSosa, Suresh Sethuramaswamy
- Published: Aug 11, 2026

The piece argues that modern risk failures stem less from modeling weakness than from slow, fragmented data architecture. It uses Silicon Valley Bank, Archegos, and the UK LDI crisis to show how batch processing, siloed counterparty data, and static stress tests can obscure liquidity, aggregate exposure, and feedback loops until intervention is difficult. Its proposed shift is a unified Databricks Data and AI Platform combining governed data, real-time aggregation, lineage, scenario analysis, and AI-assisted investigation across risk functions. Specific capabilities described include Unity Catalog permissions and lineage, VaR or Expected Shortfall deltas computed in seconds, and SQL-auditable natural-language queries with MLflow-backed model provenance. The conclusion is that real-time, defensible risk visibility supports the CRO’s expanding role in capital allocation and growth, with Raiffeisen Bank International, Morgan Stanley, and State Street cited as production examples.


### [Electric joins Databricks to bring WASM Postgres to AI agent sandboxes](https://yomu.fyi/post/electric-joins-databricks-to-bring-wasm-postgres-to-ai-agent-sandboxes.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Stas Kelvich, Yan Leshinsky, Nikita Shamgunov, Reynold Xin
- Published: Aug 11, 2026

Databricks says Electric is joining the company to extend its Postgres capabilities from the lakehouse to AI agent sandboxes. The post frames agentic applications as requiring both durable governed state and rapidly changing local context: agents choose data at runtime, operate in sandboxed environments where cloud databases are reached over the network, and collaborate while working in parallel. Electric’s PGlite is a lightweight Postgres database built for WebAssembly that runs inside an agent, application, browser tab, or device, while its real-time sync engine synchronizes distributed data with centralized Lakebase infrastructure. PGlite’s weekly downloads reportedly grew from 1 million to 13 million in twelve months, and the sync architecture is described as keeping agents’ shared context current. Combined, Lakebase, PGlite, and synchronization are presented as a Postgres-based foundation for collaborative agentic applications at production scale.


### [How to ground Genie Agents in both structured data and documents without losing governance](https://yomu.fyi/post/how-to-ground-genie-agents-in-both-structured-data-and-documents-witho.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Doyoung Jung
- Published: Aug 10, 2026

Databricks Genie Agents can combine structured tables and unstructured documents, but the central challenge is preserving data governance when one agent can access both. The proposed architecture runs agents with the end user’s credentials, using Unity Catalog identity synchronization, object privileges, ABAC policies, row filters, and column masks to enforce permissions at query time. Automatic Identity Management syncs users and groups from Microsoft Entra ID and Okta, while just-in-time provisioning carries current memberships into Databricks. Structured assets include tables and metric views, and governed tags can attach protection across newly tagged data; documents are governed through Unity Catalog volumes and their grants. In the Brickstore example, APAC and AMER managers ask the same question yet receive different, authorized results, with regional rows, masked email values, and shared metric definitions; the source recommends testing by impersonating users and planning volume boundaries.


### [Innocent until combined: Blocking the lethal trifecta with Omnigent Contextual Policies](https://yomu.fyi/post/innocent-until-combined-blocking-the-lethal-trifecta-with-omnigent-con.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Nishith Sinha, Arun Pamulapati, Omar Khawaja
- Published: Aug 10, 2026

An AI agent can exfiltrate private data when one session combines access to confidential information, attacker-controlled content, and an external communication channel, even though each capability is individually authorized. Omnigent's contextual policy tracks those three legs as session state: private data, untrusted content, and exfiltration, with human-defined tool or argument-based classifications. It allows ordinary actions until both prerequisite legs are lit, then denies the outbound call that would complete the trifecta. In the support assistant example, a user-supplied ticket instructs the agent to read internal Q3 revenue figures and email them externally; without the policy, the confidential number leaves in a reply, while the protected run blocks that email. A password-reset ticket still proceeds because it lights only the untrusted-content leg, and an empty internal lookup does not light the private-data leg.


### [Introducing FILE type: a native column type for multimodal data](https://yomu.fyi/post/introducing-file-type-a-native-column-type-for-multimodal-data.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Michael Armbrust, Burak Yavuz, Dejan Krakovic, John Spencer
- Published: Aug 10, 2026

The beta of FILE introduces a native column type for storing unstructured data such as documents, images, audio, and video alongside structured table data. FILE applies fine-grained governance, row- and column-level controls, ABAC, and lifecycle synchronization, so deleting a row also deletes the corresponding binary in object storage. Its lightweight pointers fetch file content only when a query requires it, while SQL, Python UDFs, built-in AI functions, and materialized views support multimodal processing. The example uses sampled dashcam frames and object detection to identify stops with no hazard, then joins extracted results with trip metadata; the announcement also describes connectors, incremental pipelines, and an open effort around Parquet, Delta Lake, Apache Iceberg, and Apache Spark.


### [Managing AI Coding Costs at Scale](https://yomu.fyi/post/managing-ai-coding-costs-at-scale.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Patrick Wendell, Akshat Bhatia, Vinay Gaba, Erich Elsen, Ivan Zhou
- Published: Aug 7, 2026

AI coding tools have improved velocity metrics at Databricks, but broad deployment creates exponentially growing costs that can threaten the efficiency gains they deliver. The post presents a cost-management playbook aimed at preserving broad, low-friction access while keeping aggregate spending within a roughly fixed per-user envelope. It emphasizes the coding-model efficiency frontier, automated evaluations of internal workloads, model-flexible harnesses or meta-harnesses, and request- or task-level routing to the cheapest capable model. Databricks reports that Unity AI Gateway Smart Router reduced average task cost by more than 30% while roughly matching the most expensive model’s quality, and that harness and caching changes cut generated tokens and associated costs by almost 50% without observed quality degradation. The proposed AI Gateway centralizes model access, budget policies, tool configuration, and coding-session traces for ongoing optimization.


### [What is an AI Assistant?](https://yomu.fyi/post/what-is-an-ai-assistant.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Aug 7, 2026

AI assistants are software systems that use language models, data retrieval, and reasoning to understand natural-language requests and return information or perform tasks. For enterprise data teams, they can generate SQL, build dashboards, troubleshoot pipelines, and automate repetitive work, but their usefulness depends on integration with data catalogs and governance rather than generic text generation alone. The described pipeline captures input, parses intent and context, retrieves knowledge, generates an output, and either delivers it for review or executes an action; advanced systems incorporate tool and API use plus feedback from corrections and approvals. The article categorizes assistants as voice, conversational, general-purpose, domain-specific, data and developer, or autonomous agents, while noting risks including hallucinations, privacy exposure, integration complexity, over-reliance, and bias. It presents assistants as augmenting human work through faster, more accessible, and more scalable decision support, not replacing human oversight.


### [What are Agentic Workflows?](https://yomu.fyi/post/what-are-agentic-workflows.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Aug 6, 2026

Agentic workflows are AI-driven processes that let one or more agents plan, execute, and refine multi-step tasks toward a defined goal with minimal human intervention. Unlike chatbots and fixed automation, they evaluate context and intermediate results at runtime, gather data, select tools dynamically, and adapt when outcomes are incomplete, incorrect, ambiguous, or unexpected. The described pattern combines agents and LLMs with APIs, databases, code executors, prompts, feedback mechanisms, memory and state management, and, for complex work, coordinated specialized agents. Production reliability depends on clear success criteria, action guardrails, human review points for high-stakes actions, audit trails, governance, and production-grade infrastructure. Examples include customer-service resolution, fraud investigation, supply-chain optimization, and event-driven document processing across systems.


### [What is Tool Calling?](https://yomu.fyi/post/what-is-tool-calling.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Aug 6, 2026

Tool calling lets an AI model interact with external tools, APIs, and systems to retrieve information or perform actions beyond its training data, turning a conversational model into an agent that can act. The process is an application-mediated loop: the model recognizes when outside help is needed, selects a tool from schema-described options, emits a structured request—typically JSON—and receives the result as context for its response or next step. Available tools can support information retrieval, code execution, process automation, and smart-device or IoT monitoring, while complex tasks may chain multiple calls into a workflow. The piece also distinguishes tool calling from function calling, describing the latter as a subset, and presents MCP as a standardization layer for describing, discovering, and invoking tools, with Databricks Agent Bricks offering governed integrations and agent-development support.


### [Introducing OfficeQA Pro V2: A New Benchmark for Enterprise Grounded-Reasoning](https://yomu.fyi/post/introducing-officeqa-pro-v2-a-new-benchmark-for-enterprise-grounded-re.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks AI Research Team
- Published: Aug 6, 2026

OfficeQA Pro V2 benchmarks whether AI agents generalize grounded reasoning to unfamiliar enterprise document collections rather than specialize in one corpus or task distribution. It contains 90 questions grounded in roughly 120,000 pages of U.S. Treasury accounting records spanning 1793–2024, including dense tables, charts, revised values, and changing reporting conventions. Questions were generated with asynth from seeded inputs, corpus evidence searches, traceable reasoning, and quality gates, then evaluated with deterministic exact match at 0.0% tolerance. Out-of-the-box frontier agents averaged 37.5% in one matched comparison, while competition systems averaged 41.1% and the winning team reached 63.3%; Databricks Genie raised matched-model accuracy by 24.0 percentage points on average, although failures remained in parsing, temporal reconciliation, and entity or category interpretation.


### [Kimi K3 from Moonshot AI is now available on Databricks through Unity AI Gateway](https://yomu.fyi/post/kimi-k3-from-moonshot-ai-is-now-available-on-databricks-through-unity.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: The Databricks Engineering Team
- Published: Aug 6, 2026

Kimi K3 from Moonshot AI is now available on Databricks through the Foundation Model API and Unity AI Gateway, giving customers native access to an open-weight model alongside proprietary models from Anthropic, OpenAI, and Google Gemini. The announcement positions Kimi K3 as a frontier-quality option for coding, agentic reasoning, and document understanding, with deployment flexibility, customizability, and zero data retention coverage. Databricks says the model runs against governed enterprise data through Unity Catalog, while Unity AI Gateway provides permissions, safety guardrails, audit trails, spend controls, routing, and a standardized REST API within Databricks’ security perimeter. The post cites a score of 57 and rank of #4 in Artificial Analysis’ Intelligence Index, internal benchmarks showing performance at the level of leading proprietary models, and 50–72% lower cost per task; availability includes US hosting for AWS and GCP workspaces.


### [Unity AI Gateway is Generally Available](https://yomu.fyi/post/unity-ai-gateway-is-generally-available.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: David Nasi, Stefania Leone, Kevin Stumpf, Kasey Uhlenhuth, Kelly Albano
- Published: Aug 4, 2026

Unity AI Gateway is generally available as a centralized way for enterprises to manage AI costs, security, governance, and model choice across agents and other AI assets. It positions Unity AI Gateway alongside Unity Catalog: Unity Catalog supplies identity, permissions, lineage, and auditing, while the gateway applies runtime guardrails and contextual policies across AI interactions. The gateway centralizes usage data in Unity Catalog, offering dashboards, Genie-powered analysis, granular cost attribution, budgets, hard spend caps, and a single-query API for frontier models from providers including Anthropic, OpenAI, Gemini, Kimi, and GLM. Smart Routing is in beta and dynamically selects models using quality, cost, performance, availability, and budget; the announcement says customers already use the gateway at scale, including more than one quadrillion tokens passing through it in the past year.


### [Databricks joins the Open Secure AI Alliance to advance AI safety and security](https://yomu.fyi/post/databricks-joins-the-open-secure-ai-alliance-to-advance-ai-safety-and.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Katie Cummiskey, Maria Pere-Perez, Arun Pamulapati, Nishith Sinha
- Published: Aug 4, 2026

Databricks announces that it is a founding member of the Open Secure AI Alliance, formed with NVIDIA and more than 75 organizations to advance openly shared AI safety, AI security, and AI-enabled cyber defense research. The alliance’s scope extends beyond model weights to an open execution stack covering runtimes, guardrails, agent harnesses, risk frameworks, and governance. Databricks contributes Omnigent, an Apache 2.0 meta-harness supporting 13+ harnesses with contextual policies, spend caps, and sandbox isolation; DASF 3.0, which maps 97 risks across 13 components to 73 controls; DAGF; and BlackIce, a containerized toolkit bundling 14 red-teaming tools. Its Lakewatch Security Lakehouse applies the same openness to governed, agentic detection and response over portable data formats.


### [The New Monday Morning Report: How Generative AI can deliver the insights your executives need.](https://yomu.fyi/post/the-new-monday-morning-report-how-generative-ai-can-deliver-the-insigh.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Roberto Robles Nacif
- Published: Aug 3, 2026

Retail and CPG joint planning meetings often spend time reconciling fragmented, stale, and inconsistent data instead of deciding how to adjust spend, inventory, or forecasts. The proposed Monday Morning Brief replaces the manually stitched weekly deck with an overnight agent-generated brief built from continuously refreshed point-of-sale, shipment, and inventory data alongside trade, promotion, forecast, and external signals. It uses Delta Sharing for a shared view, Genie Ontology for business context, Unity AI Gateway for permissions, guardrails, logging, rate limits, and human approval, and supports models across AWS, Azure, and Google Cloud. The agent scans item-store combinations, ranks material watchouts, answers cited natural-language follow-ups, and can draft actions such as purchase-order changes, while humans retain approval for consequential decisions. The proposed progression from report to ritual to intelligent decision system includes a 90-minute workshop, a day-30 pilot scope, and a first live Monday targeted for day 90.


[Newer posts](https://yomu.fyi/company/databricks/page/2.md) · [Older posts](https://yomu.fyi/company/databricks/page/4.md)