---
title: "Databricks"
description: "Data and AI platform for data engineering, analytics, machine learning, and generative AI."
---

# Databricks
> Data and AI platform for data engineering, analytics, machine learning, and generative AI.

## Articles

### [What is pgvector?](https://yomu.fyi/post/what-is-pgvector.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 17, 2026
- Type: Explainer

pgvector is an open-source PostgreSQL extension that adds vector storage, indexing, and similarity search for embeddings alongside relational application data. It supports semantic search, recommendations, retrieval-augmented generation (RAG), image similarity, anomaly detection, and deduplication without requiring a separate vector database for many workloads. The extension uses the vector data type and distance metrics including L2, cosine similarity, and inner product, with HNSW and IVFFlat indexes offering different speed and memory trade-offs. Vector queries can also combine relational filters with Postgres full-text search for hybrid retrieval. pgvector is positioned as a practical choice for existing Postgres deployments, while pgvectorscale can extend its usefulness as data and query demands grow; dedicated vector databases become more relevant at larger scales, and Databricks AI Search serves complementary lakehouse workloads.


### [Banks don't have an AI problem — they have a data platform problem](https://yomu.fyi/post/banks-don-t-have-an-ai-problem-they-have-a-data-platform-problem.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Naeem Rehman, Jennifer Miller
- Published: Apr 17, 2026
- Type: Explainer

The post argues that banks’ difficulty scaling AI stems less from model capability than from fragmented data, weak governance, and limited real-time access. Examples from CBA Live 2026 include model drift in credit scoring, fragmented customer signals, and a collections model that predicted, with 85% accuracy, how many days newly delinquent accounts would take to cure using governed data. It also presents Erica’s 3.2 billion interactions since 2018 as evidence that production AI requires continuous tuning, monitoring, and edge-case management, while generic models decay in frontline settings. The proposed Databricks architecture combines Lakehouse, Unity Catalog, Lakeflow, Lakebase, MLflow, Model Monitoring, Online Feature Store, Genie, and Agent Bricks to support governed analytics, low-latency decisions, auditable models, and constrained agent actions. Its conclusion is that a shared data platform should precede additional AI use cases because it improves deployment speed, trust, explainability, and regulatory defensibility.


### [Open Platform, Unified Pipelines: Why dbt on Databricks is Accelerating](https://yomu.fyi/post/open-platform-unified-pipelines-why-dbt-on-databricks-is-accelerating.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Srilekha Dornadula, Ramiz Bozai
- Published: Apr 16, 2026
- Type: Explainer

dbt provides structure for transforming raw data into curated datasets, but the post argues that its effectiveness depends on the underlying platform. Databricks is presented as an open lakehouse combining storage, compute, governance, orchestration, and monitoring to reduce duplicated data, fragmented permissions, and operational overhead. Lakeflow Jobs runs dbt alongside ingestion, downstream actions, retries, and dependencies, while Unity Catalog supplies persistent permissions, documentation, lineage, and fine-grained controls. The integration also supports Delta Lake and Apache Iceberg, query tags for cost tracking, Photon, Predictive Optimization, Liquid Clustering, and dbt materialized views. The post concludes these capabilities provide openness, integrated operations, governance, and price-performance for dbt workflows, citing one customer’s consolidation of notebooks and 7+ source systems and more than 2,900 customers already running dbt on Databricks.


### [Why your agents can't read enterprise documents — and how to fix it](https://yomu.fyi/post/why-your-agents-can-t-read-enterprise-documents-and-how-to-fix-it.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Archika Dogra, Sergei Tsarev, Erich Elsen
- Published: Apr 16, 2026
- Type: Announcement

Enterprise agents often reason well over clean text but struggle with scanned PDFs, inconsistent layouts, nested tables, images, and handwriting, making document processing an accuracy ceiling for business workflows. The post announces Document Intelligence, a pipeline of chainable AI Functions: ai\_parse\_document converts scans into layout-enriched structured text, while ai\_classify and ai\_extract support routing and entity extraction. On OfficeQA treasury-bond documents, preprocessing with ai\_parse\_document produced a 16% average performance gain across tested agent frameworks without changing their reasoning harness. Across invoice, contract, medical-note, and financial-filing extraction tasks, it reportedly delivered the highest accuracy at 5–7x lower cost than comparable pipelines, with serverless batch infrastructure scaling the same SQL call from 100 to 100,000. Databricks integrates ingestion, orchestration, governance, and agent building through Lakeflow Connect, Lakeflow Jobs or Spark Declarative Pipelines, Unity Catalog, and Agent Bricks, aiming to replace fragmented services with reusable workflows.


### [Building with Databricks Document Intelligence and Lakeflow](https://yomu.fyi/post/building-with-databricks-document-intelligence-and-lakeflow.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Giselle Goicochea, Joanna Zouhour
- Published: Apr 16, 2026
- Type: Tutorial

The post presents a production-oriented approach to Intelligent Document Processing (IDP) that brings PDFs, images, and office documents into Databricks’ governed data lifecycle. Lakeflow Connect ingests files from sources such as SharePoint and Google Drive into Unity Catalog Volumes and tables, applying access control, lineage, and auditing while supporting incremental reads and writes. Databricks’ Document Intelligence then uses ai\_parse\_document to preserve structure in VARIANT output, with ai\_extract, ai\_classify, and ai\_prep\_search available for enrichment, routing, and retrieval preparation. Lakeflow Jobs productionizes the workflow with orchestration, control flow, retries, triggers, serverless scaling, and observability across ingestion, parsing, and serving tasks. The post also recommends joining extracted data to master data, using service principals, and monitoring drift and extraction accuracy, presenting the combined platform as a way to create trusted, queryable datasets from unstructured content.


### [Databricks on Google Cloud: Innovate faster. Smarter. Together.](https://yomu.fyi/post/databricks-on-google-cloud-innovate-faster-smarter-together.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Taylor Hoss, Sarah Jack
- Published: Apr 16, 2026
- Type: Announcement

Databricks and Google Cloud are presenting their partnership at Google Cloud Next ’26 as a route from governed enterprise data to production AI, including autonomous agents and modernized workloads. In 2025, Databricks became a first-party provider of Gemini models, giving users direct API access to build, customize, and deploy agentic AI on governed data across all three Databricks clouds without moving data or managing infrastructure. The post reports more than 55% quarter-over-quarter growth in Gemini adoption, 85% year-over-year growth in Google Cloud consumption, and 4,000-plus Marketplace sign-ups, alongside support for Google Axion processors. Customer examples include Databricks SQL Serverless savings for Digital Turbine, faster Marketplace procurement for Dun & Bradstreet, regional compliance in Saudi Arabia and Brazil, and 30% faster analytics delivery through Lakeflow-powered pipelines. At the event, Databricks is offering demos, talks, expert sessions, and a free $400 trial through Google Cloud Marketplace.


### [What is Generative AI in Marketing?](https://yomu.fyi/post/what-is-generative-ai-in-marketing.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 16, 2026
- Type: Explainer

Generative AI in marketing creates content, insights and recommendations that can personalize experiences, optimize campaigns and improve performance. Unlike traditional analytics, which mainly reports on past results, it produces net-new outputs such as ad copy, audience segments, product recommendations, visual assets and strategic summaries. A typical workflow prepares campaign, customer and brand data, grounds or fine-tunes models, generates outputs, applies targeting and optimization, and uses human review and refinement. It distinguishes pretrained tools such as ChatGPT, Claude and Perplexity from customized models and broader AI transformation, noting trade-offs between speed, investment, relevance and strategic alignment. Successful adoption depends on high-quality, consented data, governance, privacy controls, human oversight, monitoring and cross-functional ownership; the conclusion presents AI as a force multiplier rather than a replacement for human creativity and judgment.


### [Introducing the Databricks Connector for Google Sheets: Real-Time, Governed Lakehouse Data in the Sheets Users Love](https://yomu.fyi/post/introducing-the-databricks-connector-for-google-sheets-real-time-gover.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Toussaint Webb
- Published: Apr 15, 2026
- Type: Announcement

Databricks has introduced a Google Sheets connector that brings live, governed lakehouse data into a familiar spreadsheet environment. The connector addresses reliance on CSV exports, snapshots, and custom pipelines, which left business teams with siloed copies, inconsistent metrics, and slower decisions. Built on Databricks SQL and Unity Catalog, it lets users query governed datasets, including Unity Catalog Metric Views, through a no-code GUI or SQL, while Unity Catalog manages permissions. Users can refresh results manually or schedule refreshes, keeping Sheets data current without exporting snapshots. The connector is Generally Available to all Databricks customers, and Nubank reports that it has enabled non-technical users to explore data in Sheets while preserving control.


### [What is operational analytics?](https://yomu.fyi/post/what-is-operational-analytics.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 15, 2026
- Type: Explainer

Operational analytics uses real-time or near-real-time data to monitor day-to-day operations and support immediate decisions, unlike traditional analytics, which generally relies on batch data to explain past performance. Its workflow collects logs, events, clickstreams, telemetry, transactions, and other signals; streams them into a unified platform for cleaning, transformation, and enrichment; analyzes incoming data with rules, models, or anomaly detection; and sends results to dashboards, alerts, or operational applications. This can help teams detect issues earlier, reduce mean time to detect (MTTD) and mean time to respond (MTTR), improve forecasting, coordinate across departments, and act on inventory, customer, or system changes. The approach also requires integrating heterogeneous systems, maintaining data quality and schemas, embedding insights into existing workflows, and building reliable low-latency pipelines, with tools such as Lakeflow, Databricks SQL, and AI and machine learning capabilities presented as examples.


### [Unity AI Gateway: How to connect agents to external MCPs securely](https://yomu.fyi/post/unity-ai-gateway-how-to-connect-agents-to-external-mcps-securely.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ahmed Bilal, Sunish Sheth
- Published: Apr 15, 2026
- Type: Tutorial

Databricks Unity AI Gateway provides a governed way to connect agents to external Model Context Protocol (MCP) servers while integrating management with Unity Catalog. The post addresses provider-specific OAuth app registration, client secrets, token refresh, permission auditing, and the lack of centralized visibility into agent access. It describes registering an external MCP server, choosing per-user OAuth or a shared principal, selecting GitHub, and letting Databricks handle OAuth registration, token exchange, and refresh; connections can then be tested in AI Playground or with DatabricksMCPClient and deployed through Agent Bricks. Unity Catalog supplies permissions and audit records, while MLflow Tracing captures requests and tool interactions, and supported providers include Glean, GitHub, Atlassian, Google Drive, and SharePoint across AWS, Azure, and GCP.


### [Expanding agent governance with Unity AI Gateway](https://yomu.fyi/post/expanding-agent-governance-with-unity-ai-gateway.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: David Nasi
- Published: Apr 15, 2026
- Type: Announcement

Unity AI Gateway extends Unity Catalog’s permissions, auditing, and policy controls to agentic workflows involving LLMs, MCP servers, and APIs. The release addresses limited visibility across multi-step actions with on-behalf-of user execution for MCP calls, configurable guardrails, and consistent governance across model providers without separate configurations for each provider and model choice can vary by task. Guardrails can detect PII, prompt injection, unsafe content, and data exfiltration, while logs capture identities, timestamps, MCP details, dollar costs, payloads, latency, status codes, and errors. OpenAI-compatible APIs, fallback models, and endpoint-, user-, or group-level rate limits support production operation, although some capabilities are in Beta or rolling out and availability is limited to supported Databricks regions.


### [Agentic reasoning in practice: Making sense of structured and unstructured data](https://yomu.fyi/post/agentic-reasoning-in-practice-making-sense-of-structured-and-unstructu.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: The Databricks AI Research Team
- Published: Apr 14, 2026
- Type: Benchmark

Enterprise questions that connect structured records with unstructured text are difficult for single-turn retrieval systems to answer reliably. The post evaluates Databricks Agent Bricks Supervisor Agent (SA), a declarative agent builder built on the internal aroll framework, across STaRK’s Amazon, MAG, and Prime domains and six KARLBench tasks. SA decomposes questions, routes sub-questions to Genie or Knowledge Assistant, combines results, and can reformulate queries or use SQL JOINs when initial retrieval is incomplete. It outperforms rerun SoTA baselines by 4% on STaRK-Amazon, 21% on MAG, and 38% on Prime, while delivering a 23% improvement on FinanceBench and consistent gains across KARLBench; the authors attribute the results to multi-step tool selection, reflection, and precise instructions rather than custom code.


### [Agent Bricks: The governed enterprise agent platform](https://yomu.fyi/post/agent-bricks-the-governed-enterprise-agent-platform.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Kasey Uhlenhuth
- Published: Apr 14, 2026
- Type: Announcement

Agent Bricks is presented as Databricks’ enterprise platform for building, deploying, and governing agents that operate on business data under real identities, permissions, and operational constraints. The platform combines multi-model and framework support, execution, routing, fallback, cost optimization, and unified governance through Unity Catalog and AI Gateway, including on-behalf-of token passing and observability across data, models, MCPs, and APIs. Its context layer uses metadata such as schemas, business definitions, lineage, permissions, and data-quality signals, while Genie Spaces, Document Intelligence, Knowledge Assistant, and Agent Mode address structured and unstructured business information. The announcement includes general availability for Document Intelligence, Custom Agents on Apps, and Supervisor Agent, plus AI Gateway guardrails, managed OAuth MCP Connectors, web search, and MLflow’s CLEARS evaluation framework; the post reports 70% higher accuracy than standard RAG and a 30% improvement in multi-step workflows.


### [8 AI and data trends shaping financial services in 2026](https://yomu.fyi/post/8-ai-and-data-trends-shaping-financial-services-in-2026.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Kim Hatton, Junta Nakai, Marcela Granados, Antoine Amend, Ashraf Safdar, Jennifer Miller, Andrea DeSosa, Rajaram Suresh
- Published: Apr 14, 2026
- Type: Explainer

The post argues that financial-services AI adoption is widespread, but execution—not model capability or strategy—is determining who captures value in 2026. It attributes stalled pilots to fragmented legacy infrastructure, inconsistent data, weak lineage, and insufficient control for governed, real-time workflows such as fraud detection, pricing, and personalization. Firms advancing further treat data as a managed asset, embed governance in data and model pipelines, and align data, analytics, and AI teams around shared definitions, workflows, and metrics. The proposed remedy is a unified lakehouse environment combining storage, compute, governance, lifecycle management, orchestration, streaming, and AI workflows, with Unity Catalog providing centralized access control, lineage, and auditing. The conclusion is that by the end of 2026, firms that embed AI into operational decisioning at scale will pull ahead of organizations still running pilots.


### [Building real-time product search on Databricks](https://yomu.fyi/post/building-real-time-product-search-on-databricks.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jiayi Wu, Luke Lefebure, Adam Gurary
- Published: Apr 14, 2026
- Type: Explainer

Real-time product search for an online marketplace must retrieve, filter, rank, and respond in milliseconds while balancing latency, relevance, revenue, and click-through rate. The Databricks architecture divides the pipeline into ingestion, retrieval, and refinement, using Lakeflow pipelines and AI Functions for source processing, AI Search for embeddings and hybrid retrieval, and Lakebase for sub-10ms operational signals. It recommends validating p95/p99 latency, throughput, retrieval quality, and engagement metrics, with MLflow supporting systematic evaluation and caching helping meet strict latency requirements. As an example, FOX Sports used AI Search for thousands of QPS and achieved a 2x improvement in query success rate, combining streaming ingestion, two-phase retrieval, model serving, caching, and trending searches.


### [Lovable + Databricks: Build Data-Driven Apps at the Speed of Thought](https://yomu.fyi/post/lovable-databricks-build-data-driven-apps-at-the-speed-of-thought.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Evan Pandya
- Published: Apr 13, 2026
- Type: Announcement

Databricks users outside data teams often depend on requests to access business data because they may not write SQL or have a Databricks license. The official Lovable connector lets teams connect Lovable to a Databricks environment and build custom internal applications with plain English. Lovable's AI-powered agent examines accessible data and builds and deploys an application, while Databricks remains the source of truth and data stays within its security perimeter. Data is queried from Databricks at runtime, avoiding ETL, data replication, and sync jobs. Examples include live revenue and pipeline dashboards, spreadsheet-replacing operational tools, internal Slack chatbots, and other business applications that teams can build without engineering or data-team support.


### [What is a Vector Database?](https://yomu.fyi/post/what-is-a-vector-database.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Oct 11, 2024
- Type: Explainer

Vector databases store and manage high-dimensional vector representations of data, grouping them by similarity rather than traditional row-and-column organization. They support semantic and similarity-based retrieval by converting text, images, audio, or video into embeddings, indexing them with methods such as product quantization and HNSW, and comparing query vectors with indexed vectors. These capabilities underpin use cases including RAG, hybrid search, recommendations, NLP, image and video recognition, and anomaly detection, while sharding, partitioning, caching, and replication support scale. The article distinguishes a vector database from a standalone index because it also provides storage, CRUD, metadata filtering, horizontal scaling, and data management functions, and describes Databricks AI Search as an integrated implementation with automatic scaling, optimization, security, and governance.


[Newer posts](https://yomu.fyi/company/databricks/page/28.md)