---
title: "Databricks"
description: "Data and AI platform for data engineering, analytics, machine learning, and generative AI."
---

# Databricks
> Data and AI platform for data engineering, analytics, machine learning, and generative AI.

## Articles

### [AI customer service: strategy, agents, and solutions guide](https://yomu.fyi/post/ai-customer-service-strategy-agents-and-solutions-guide.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jul 27, 2026

AI customer service combines natural language processing, machine learning, predictive analytics, generative AI, and automation to route requests, generate responses, and support resolution across channels. The guide distinguishes AI agents from scripted chatbots: agents reason over context, call external tools, and complete multi-step tasks such as checking shipping data and issuing a partial refund. It presents AI as a layer that absorbs routine volume while human agents retain judgment over complex or sensitive cases, citing potential gains including more than 70% query automation, up to 30% lower operating costs, and a 15% improvement in customer satisfaction. It recommends evaluating integration, decision transparency, production support, security, and compliance, then starting with one measurable, high-volume use case and expanding only after consistent resolution quality.


### [Building AI Literacy: Frameworks, Tools, and Practices](https://yomu.fyi/post/building-ai-literacy-frameworks-tools-and-practices.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jul 27, 2026

AI literacy is presented as the ability to understand how AI works, use its tools effectively, evaluate outputs critically, and recognize ethical implications. The guide organizes this fluency into functional, critical, and ethical domains, with progression from defining terms and crafting prompts to comparing tools, auditing outputs for bias, and designing human oversight. It explains that large language models generate probable continuations from statistical patterns in training data, so hallucinations and bias require verification against trusted sources rather than assuming factual correctness. For education and organizations, it recommends distributed curriculum or role-specific modules, hands-on assignments, measurable assessment checkpoints, governance policies, internal champions, iterative pilots, and longer-term impact metrics.


### [AI in supply chain: from demand forecasting to AI agents](https://yomu.fyi/post/ai-in-supply-chain-from-demand-forecasting-to-ai-agents.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jul 24, 2026

AI in supply chain management applies machine learning, generative AI and AI agents to demand forecasting, inventory optimization, supplier-risk assessment and logistics orchestration. It connects ERP records, point-of-sale feeds, supplier communications and external signals to support continuous decision-making rather than periodic planning. The guide maps use cases to required data foundations, covering demand-sensing pilots, warehouse task prioritization, agent guardrails, cross-system APIs and generative-AI grounding. It recommends starting with a product category or region, comparing model output with a baseline, and expanding only when accuracy and bias improve consistently. Reported figures include up to 85% forecast-accuracy improvement, up to 15% lower inventory carrying costs, and 23% lower fulfillment costs among organizations that deployed AI at scale.


### [AI in healthcare: applications and best practices](https://yomu.fyi/post/ai-in-healthcare-applications-and-best-practices.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jul 24, 2026

Healthcare AI applies artificial intelligence, machine learning, deep learning, and generative AI to clinical, administrative, and research workflows, using patient data, EHRs, medical imaging, and clinical documentation to support decisions and operational efficiency. It distinguishes AI models from algorithms and generative models, and describes supervised classifiers, time-series, natural-language-processing, and computer-vision systems used across healthcare. The data section emphasizes standardized models, consistent coding, interoperability through APIs and FHIR, and unified data lakehouse architecture, noting reported EHR prediction accuracy of 70-72% that varies by condition, data quality, and population. For generative documentation, audio transcription, large language models, templates, and Retrieval-Augmented Generation can draft notes, but clinician review remains required because fluent output may contain factual errors. It also covers imaging AI, privacy practices, FDA and European AI Act obligations, and the expectation that clinical AI will augment rather than replace professional judgment.


### [AI Applications in Finance: A Practical Use Case Guide](https://yomu.fyi/post/ai-applications-in-finance-a-practical-use-case-guide.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jul 24, 2026

AI in finance spans machine learning, natural language processing, and generative AI for credit scoring, fraud detection, algorithmic trading, finance automation, and decision support across banking, capital markets, and insurance. Finance teams rank use cases by revenue impact, risk reduction, and implementation effort, while data scientists clean and validate the underlying data. Credit scoring can combine traditional and alternative data, with confidence thresholds routing uncertain cases to human underwriters; trading strategies use historical backtests, monitoring, and versioned audit trails. Fraud systems monitor transactions in real time and prioritize alerts, while finance automation uses machine learning, rules-based logic, exception queues, and ERP integration. The guide recommends 90-to-120-day pilots with predefined metrics and ROI measurement before scaling, alongside explainable AI, model governance, and logged decisions for responsible deployment.


### [How the FDA Built an AI Platform That 85% of Its Staff Now Use Daily](https://yomu.fyi/post/how-the-fda-built-an-ai-platform-that-85-of-its-staff-now-use-daily.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Molly Just-Behr
- Published: Jul 23, 2026

The FDA built ELSA, a generative AI platform for its 16,000 staff, on Halo, a governed Databricks data foundation created to address fragmented systems across eight centers. Those centers had separate chatbots and data stores; consolidating 50 to 60 sources enabled faster sharing, real-time streaming, and centralized access controls through Unity Catalog. Within roughly two months, ELSA adoption rose from less than 1% to 85%, while staff began building hundreds of agents weekly from standard operating procedures, regulatory guidance, and center-specific documents. MCP servers layered over Unity Catalog make governed data and tooling accessible beyond data scientists, and Databricks ML and NLP capabilities through MLflow extracted starting materials and product-supplier-manufacturer relationships from millions of submission pages. A reviewer can now request grounded starting-material information for a drug application in about three minutes instead of days, while the FDA adapts center-specific MCP tools and extends the model across its organization.


### [Permission isn't purpose: Intent-based authorization in Omnigent](https://yomu.fyi/post/permission-isn-t-purpose-intent-based-authorization-in-omnigent.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Nishith Sinha, Matei Zaharia
- Published: Jul 23, 2026

Omnigent’s intent-based authorization addresses a gap in identity-based access control: an agent with valid credentials may follow indirect prompt injections embedded in data and perform authorized actions unrelated to the user’s task. It binds each session to a human-declared purpose and evaluates that intent before every tool call, producing permitted, consent-required, or denied verdicts. In a data-quality example, reading a customers table is allowed, dashboard publication requires approval, and an injected request to grant external access is denied despite the agent’s identity permitting that tool. The intent is fixed at design time for autonomous agents or approved by a human at session start for interactive agents, and cannot be broadened by the agent. Omnigent combines this policy with session-risk scoring and other contextual policies in a single engine where any denial wins.


### [Provisioning for the Agentic Era: How Databricks Built a Self-Serve Infrastructure Vending Machine](https://yomu.fyi/post/provisioning-for-the-agentic-era-how-databricks-built-a-self-serve-inf.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Evan Pandya, Greg Wood, Joel Thomas
- Published: Jul 23, 2026

Databricks built the Field Engineering Vending Machine (FEVM) to replace shared environments with isolated, centrally governed infrastructure for its growing field organization. Users or agents describe a use case and configuration details, receiving an environment assembled through a React frontend, Python backend, Databricks Apps, Terraform, Git Runner, and Lakebase across AWS, Azure, and GCP. FEVM supports templates and add-ons such as Lakebase, notebooks, and pre-packaged assets from UC Volumes, applies configurable time-to-live policies, and sends Slack notifications for provisioning, expiration, and deletion. It also manages shared resources independently, including catalogs that persist across workspace lifecycles, and provides administrative controls for limits and audits. At the time described, it managed more than 2,600 active deployments across three clouds and over 5,000 active users, after handling nearly 1,200 requests during one internal event.


### [Why A Frontier Data Agent Outperforms General Coding Agents in Quality and Cost](https://yomu.fyi/post/why-a-frontier-data-agent-outperforms-general-coding-agents-in-quality.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: The Databricks AI Research Team
- Published: Jul 23, 2026

An evaluation compared Genie Code with three widely used coding agents from major AI labs on 401 real internal data tasks, using a shared 20-minute wall-clock budget and independent grading. Genie Code achieved 76.6% accuracy at an estimated $0.55 per task, outperforming rivals at 55.9–72.1% accuracy and $0.91–$1.16 per task. Its advantage is attributed to semantic search across catalog and workspace assets, persistent memory of tables and business logic, and enterprise-context understanding, which reduce exploratory tool use. On discovery-heavy tasks, general-purpose agents often wandered through large workspaces or timed out on inefficient scans, while Genie Code averaged 8.3 tool calls per task. The benchmark found no accuracy-cost trade-off, although the authors note that Genie Ontology was disabled and expect it to strengthen the advantage.


### [Connect Amazon S3 data to Databricks with Delegated IAM Permissions](https://yomu.fyi/post/connect-amazon-s3-data-to-databricks-with-delegated-iam-permissions.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Gordon Wang
- Published: Jul 23, 2026

Databricks introduces a simpler flow for connecting Amazon S3 to Unity Catalog through an external location, addressing the manual work previously required to establish governed read and write access. Instead of writing 140-line IAM trust policies, configuring bucket permissions, deploying CloudFormation templates, and switching between consoles, users specify a bucket and access level, then verify permissions in AWS. AWS IAM temporary delegation lets Databricks provision a least-privilege IAM role, configure its cross-account trust policy, register the external location, and enable Auto Loader and File Events. The delegated authorization is time-bounded and expires after setup; provisioning actions are logged in AWS CloudTrail, and users without sufficient permissions can request access from an AWS administrator in the flow. The result is a single-session setup intended to reduce configuration errors while supporting governed S3 access for ingestion, pipelines, analytics, and LTAP workloads.


### [Introducing AI spend controls with Unity AI Gateway](https://yomu.fyi/post/introducing-ai-spend-controls-with-unity-ai-gateway.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Kevin Stumpf
- Published: Jul 23, 2026

Unity AI Gateway now offers AI Spend Controls, extending existing cost visibility with proactive budget alerts across models and workloads. The feature supports budgets at user, use-case, workspace, and account levels, plus shared and per-user thresholds that can trigger email alerts or enforce hard caps by stopping requests after a limit is exceeded. Configuration starts in account settings under Usage and Budgets, where administrators select Unity AI Gateway, optionally scope workspaces and resource tags, and define monthly limits and recipients. Budget status and trends are available in the Cost section, while customizable Cost Analytics dashboards use Unity Catalog system tables to attribute DBU and model-provider costs by identity, workspace, endpoint, tags, model, provider, and request tags. The release positions Databricks budgets, Unity AI Gateway, and Unity Catalog as a combined governance layer for controlling AI access, usage, and spend.


### [Simplify AI agent orchestration with Lakebase Postgres](https://yomu.fyi/post/simplify-ai-agent-orchestration-with-lakebase-postgres.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Li Yu, Michelle JanneyCoyle, Jon Cormack, Yarri Bryn, Alec Sorensen, Darshana Nair
- Published: Jul 22, 2026

CLA and Databricks built a production document-processing application for auditing that reduces extraction time from hours to minutes without compromising quality, using Databricks-native services including Lakebase Postgres, Databricks Apps, Lakeflow Jobs, MLflow, and Unity Catalog Volumes. Lakebase serves as the orchestration layer’s single source of truth for tasks and execution attempts, coordinating long-running work, retries, leases, priorities, rate limits, costs, and status visibility. The queue uses Postgres patterns including FOR UPDATE SKIP LOCKED, priority and FIFO ordering, expiring leases for crash recovery, and database-backed concurrency controls. Databricks Jobs process PDFs through intelligent document processing and vision/LLM calls, while MLflow Tracing records execution and cost details and dashboard updates combine fast Postgres data with slower billing queries. In production, this architecture avoids external brokers and schedulers while providing durable task management, real-time visibility, and per-task cost attribution.


### [The last mile: why great first-party data still doesn't make great marketing](https://yomu.fyi/post/the-last-mile-why-great-first-party-data-still-doesn-t-make-great-mark.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Michael Burton, Katy Yuan
- Published: Jul 21, 2026

Modern data platforms and sophisticated marketing systems can still fail to produce timely customer experiences when no bridge connects first-party data to campaign activation. The post explains Scott Brinker’s “composable canvas,” a five-ring architecture centered on a unified data core, with semantic layer, CaaS, decisioning, and apps and agents operating on shared data without repeated movement. In contrast, batch files, dashboards, and disconnected teams can leave autonomous agents unable to trigger campaigns or propensity models unused. It recommends closing this last mile through activation-ready data architecture, self-service marketing analytics, and one narrowly scoped AI agent tied to a measurable campaign outcome. Examples in the post include a 4X model conversion rate, a four-week account scoring launch, and grocery offer flows that changed weekly preparation from hours to minutes.


### [How Dow Built a Carbon Footprint Ledger on Databricks to Accelerate Sustainability at Scale](https://yomu.fyi/post/how-dow-built-a-carbon-footprint-ledger-on-databricks-to-accelerate-su.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jesse Grekowicz, Tim Licquia, Varun Mahajan
- Published: Jul 21, 2026

Dow built the Carbon Footprint Ledger (CFL) on Databricks to provide a transparent, verifiable way to track and communicate product carbon footprints amid customer Scope 3 commitments and tighter environmental disclosure requirements. The system combines carbon-accounting methodology assured against ISO 14067 and the GHG Protocol Product Standard with a calculation engine and certificate-issuing ledger that maintains residual balances. Within Dow’s Integrated Data Hub, Apache Spark unifies supply-chain, sustainability, operations, and other source data, while Unity Catalog governs access and Delta tables provide ACID transactions, time travel, and efficient upserts. An advanced optimization model, developed and deployed with MLflow, identifies lowest-greenhouse-gas production pathways and sends versioned, monitored results through the pipeline with lineage tracking. The implementation reduced PCF processing from weeks to a fraction of that, enabled portfolio-scale calculations, and supports verifiable certificates and commercial use of decarbonization investments.


### [Why R&D Data Belongs in the Lakehouse - and Why Agents Need It There](https://yomu.fyi/post/why-r-d-data-belongs-in-the-lakehouse-and-why-agents-need-it-there.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Sebastian Eberhardt, Dominik Bentele, Jonathan Bräuer
- Published: Jul 21, 2026

cellcentric describes how it built a governed lakehouse foundation for R&D data so industrial AI agents can reason over engineering context, not isolated source-system records. On Azure and Databricks, the Data Hub combines Unity Catalog governance, Lakehouse Federation, Delta Sharing, Declarative Automation Bundles, and an MCP interface for agents. Its Fuel Cell Passport unifies five enterprise systems, models seven hierarchy levels, refreshes daily, and uses a state-based temporal model for point-in-time configuration and rework-history questions. The platform treats context coverage as a data-quality metric, with 27 published products averaging 90% column-comment coverage, while authenticated user identity and Unity Catalog authorization constrain agent access. For many R&D and process-development workflows, work that once took weeks now ships in days, though complex multi-source investigations still require substantial domain review.


### [Announcing the Public Preview of Discover and Domains, powered by Unity Catalog](https://yomu.fyi/post/announcing-the-public-preview-of-discover-and-domains-powered-by-unity.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Stef Bran Melendez, Kelly Albano
- Published: Jul 21, 2026

Databricks announces the Public Preview of Domains and the Discover page, powered by Unity Catalog, to help people and agents identify relevant, high-quality, and safe data and AI assets. Domains organize tables, dashboards, notebooks, queries, metric views, Genie Agents, and apps by business structure, while Discover provides an internal marketplace with search, certification signals, popularity and trending indicators, and AI-powered recommendations. Data stewards can create domains and subdomains, certify assets, add descriptions and contacts, and curate page sections and pinned content. Domains extend Unity Catalog Semantics and feed Genie Ontology, giving agents business context for narrowing retrieval, prioritizing trusted assets, and interpreting metrics within each user’s existing permissions. The features are available in Public Preview through a Databricks workspace, where organizations can create domains and curate assets.


### [Branching databases like code: a CI/CD pattern for Lakebase, in production at Glaspoort](https://yomu.fyi/post/branching-databases-like-code-a-ci-cd-pattern-for-lakebase-in-producti.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Hadi Farhat, Gideon Spierings, Ricardo de Vries, Raymon Veldman
- Published: Jul 20, 2026

Glaspoort replaced a sprawl of outdated, one-off BI reports with a custom application that combines Databricks analytical tools with Lakebase, a serverless Postgres OLTP database. The application keeps curated lakehouse data in a read-only schema while application state occupies a separate schema, and supports development, acceptance, and production environments. Its CI/CD pattern treats database branches like code: long-lived environments branch directly from production, each pull request receives an ephemeral production-shaped branch, migrations are replayed, and a staging app image runs the full test suite against it. This topology makes periodic resets cheap, avoids deleting dependent branches, and preserves fresh test data; the team chose stacked promotion for velocity, retaining a separate crisis pipeline for urgent fixes.


### [Scaling document classification to 100k+ labels](https://yomu.fyi/post/scaling-document-classification-to-100k-labels.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jane Zhang, Arnav Singhvi
- Published: Jul 20, 2026

Large-taxonomy classification must map freeform text to 35,000–100,000 labels while meeting quality, cost, and throughput requirements. The benchmark compares vector search, a two-step vector-search-plus-AI-Classify workflow, and direct frontier-model calls across transaction, company, and biomedical datasets, using hybrid semantic and BM25 retrieval with Reciprocal Rank Fusion for the search stages. The combined workflow performed best, reaching 0.81 average accuracy versus 0.76 for Gemini 3.5 Flash, while costing roughly one-hundredth as much per document; its strongest average setting shortlisted the top 20 labels. Vector search alone was nearly 100 times cheaper than the workflow but scored more than 20 points lower, while full-taxonomy frontier calls could exceed context limits. Taxonomy updates can be handled by embedding and ingesting changed labels rather than retraining or redeploying classifiers.


### [The three ways AI unlocks transformation in Retail, Travel, and Consumer Goods](https://yomu.fyi/post/the-three-ways-ai-unlocks-transformation-in-retail-travel-and-consumer.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Rob Saker
- Published: Jul 20, 2026

The piece argues that retail, travel, and consumer-goods companies face one problem in three forms: signals are not trusted, arrive too late, or cost too much to process at scale. It contrasts business intelligence, which depends on structured schemas, predefined questions, and dashboards, with AI systems that read unstructured data, reason probabilistically across signals, and connect decisions to action. Examples include reported improvements from Harmons’ shelf scanning, faster consumer-insight work at a health and hygiene company, and travel applications spanning maintenance, pricing, and concierge services. Its proposed architecture combines broad ingestion and open storage with governance, evaluation, model and agent controls, and applications that operate on a substrate, presenting current, coherent data as the foundation for organizations that can act.


### [AI Agent Orchestration: A Guide for Enterprise Systems](https://yomu.fyi/post/ai-agent-orchestration-a-guide-for-enterprise-systems.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jul 20, 2026

AI agent orchestration coordinates multiple specialized agents by managing task assignment, shared state, communication, and execution sequencing across complex enterprise workflows. The guide contrasts autonomous agents, which reason about subtasks and adapt to intermediate results, with fixed workflows, and recommends decomposing processes into single-responsibility agents with documented input/output contracts and least-privilege access. It describes centralized, decentralized, hierarchical, hybrid, federated, and emergent patterns, noting that pattern selection trades control, resilience, scalability, and auditability according to risk. Implementation guidance includes assigning human owners, defining accuracy, latency, and escalation targets, establishing failure paths, instrumenting a minimal prototype, and measuring a pilot against documented baseline processes. Organizations using multi-agent systems report 35% faster task completion and a 30% efficiency increase with specialized agents, while high-risk or irreversible actions still require human approval gates and audit trails.


[Newer posts](https://yomu.fyi/company/databricks/page/4.md) · [Older posts](https://yomu.fyi/company/databricks/page/6.md)