---
title: "Databricks"
description: "Data and AI platform for data engineering, analytics, machine learning, and generative AI."
---

# Databricks
> Data and AI platform for data engineering, analytics, machine learning, and generative AI.

## Articles

### [Practical Guide To Python App Hosting](https://yomu.fyi/post/practical-guide-to-python-app-hosting.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jul 13, 2026

Python app hosting is presented as both an infrastructure choice and, for data-intensive or AI applications, a data-architecture choice. The guide defines a hosting stack comprising application code, runtime, dependency manager, application server and reverse proxy, and contrasts it with traditional web hosting. It compares shared hosting, VPSs and cloud VMs, PaaS, container platforms and serverless functions by control, operational effort, scalability and workload fit. It notes that long-running data and AI tasks such as model training, vector indexes, cached datasets, ETL jobs and lengthy inference pipelines can exceed practical serverless limits. The conclusion is that public APIs and lightweight web apps allow broad platform choice, while apps accessing governed data, model endpoints or AI agents should be evaluated by where data lives and how access is governed, with Databricks Apps presented as an option when data already resides in a lakehouse.


### [The agentic marketing stack starts with the data layer](https://yomu.fyi/post/the-agentic-marketing-stack-starts-with-the-data-layer.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Aly McGue
- Published: Jul 10, 2026

The piece argues that agentic marketing depends on modernizing the data and infrastructure foundation first, rather than starting with isolated AI use cases. At Acxiom, moving core products and client solutions from on-premises Hadoop to Databricks reportedly improved workload runtimes by 80 to 90 percent, reducing jobs that took 50+ hours or as long as 90+ hours to 2–3 hours. The migration also reduced manual pipeline and infrastructure work, freeing engineering capacity for products and client outcomes. Those capabilities now support agentic audience planning, media buying, campaign activation, performance analytics, code generation, testing, and ad variation, with a goal of connecting the full marketing value chain. Because workflows handle PII, generated content passes through legal approval, defined controls, and human oversight; Acxiom is also embedding its data in client platforms and privacy-safe clean rooms to make decisions more transparent and native to customer environments.


### [Introducing Feature Views](https://yomu.fyi/post/introducing-feature-views.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Nick Joung, Ian Ackerman, Julia Powell
- Published: Jul 10, 2026

Feature Views are presented as a framework for creating managed feature pipelines in Databricks, addressing duplicated logic, training/serving skew, backfills, online-store plumbing, and governance for real-time ML. A Feature View defines the source, entity, time-series column, and computation once; Databricks uses it to generate point-in-time-accurate historical data, then materializes pipelines for batch or streaming production use. Registered in Unity Catalog, materialized features receive lineage and access controls, while MLflow records model dependencies and Model Serving retrieves required features without custom lookup code. For Kafka-sourced streaming features, Spark Realtime Mode, Lakebase, and Model Serving support rolling-window updates with reported end-to-end p99 latency of 200ms from event to online availability. The announcement says Feature Views are in Public Preview and can be developed through the Feature Engineering Client SDK and Genie Code, with streaming materialization requiring an Enterprise-tier workspace in a region supporting Lakebase.


### [The ambulatory intelligence gap](https://yomu.fyi/post/the-ambulatory-intelligence-gap.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Morgan Wilkie, Bryan Smith, Holly Burke, Mary Ellen
- Published: Jul 10, 2026

Health systems face an ambulatory intelligence gap: access, provider capacity, referral retention, panel management, and financial performance are interconnected, while information remains scattered across disconnected systems. Health Catalyst’s Ambulatory Intelligence combines AI with nearly two decades of healthcare improvement expertise, deploying directly in a customer’s Databricks workspace so sensitive data stays within the health system’s environment. It uses a medallion-based semantic layer, Unity Catalog for governance, Lakebase for low-latency serving, and Genie alongside dashboards to help leaders investigate why metrics change. The solution ships with prebuilt metrics across Access Optimization, Revenue Intelligence, Panel Management, and Referral Insights, plus cross-domain scorecards and configurable terminology and workflows. Reported outcomes from supported improvement work include increased revenue and encounters at Thibodaux Regional, 55,000 closed care gaps at INTEGRIS Health, and higher outpatient visits with fewer cancellations without reschedules at WakeMed; future plans include models based on prior outcomes and agentic capabilities.


### [Ask, build, compose: What our 5th Genie Hackathon taught us about Databricks Genie](https://yomu.fyi/post/ask-build-compose-what-our-5th-genie-hackathon-taught-us-about-databri.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Shruti Prasanna, Rob Bajra
- Published: Jul 9, 2026

The fifth Databricks hackathon used Databricks Genie to test three ways of working with governed data: asking, building, and composing. Genie Agents give business users domain-specific natural-language access to curated data, while Genie Code helps analysts create metric views, functions, pipelines, and dashboards inside Databricks. The composition track uses Conversation APIs and a managed MCP server to make Genie an agent tool alongside custom logic, Model Serving, and other MCP servers, with Unity Catalog governing access. Projects included supervisory routing across 190 tables, governance and analytics products built in days, and multi-agent systems such as ShipBob’s overnight operations brief. Across the tracks, the stated lesson is that shared governance and semantic context let business users, builders, and engineers use Genie at different levels without abandoning grounded, permissioned data access.


### [Navigating a Synapse Migration to Databricks](https://yomu.fyi/post/navigating-a-synapse-migration-to-databricks.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Olga Romanova, Johannes Oehler
- Published: Jul 8, 2026

The post presents migration from Azure Synapse to Databricks as a way to reduce platform complexity while supporting data engineering, analytics, machine learning, real-time pipelines, and AI applications. It distinguishes migration paths for Dedicated SQL Pools, serverless SQL, Spark Pools, orchestration, governance, and downstream BI connectivity, noting that accumulated T-SQL logic and dependencies require the most redesign, testing, and validation. The proposed programme uses discovery and assessment with Lakebridge Profiler and Analyzer, then selects an architecture, sequences workloads, pilots a real use case, and scales migration in waves. It recommends automating code conversion, reconciling row counts, aggregations, hashes, and tolerances, avoiding direct Synapse distribution-to-Delta partition mappings, and planning compaction, enablement, and rollback. Reported examples include Casey’s halving operational data delivery times from eight hours to four and Italgas cutting workload costs by 73%.


### [Benchmarking Coding Agents on Databricks’ Multi-Million Line Codebase](https://yomu.fyi/post/benchmarking-coding-agents-on-databricks-multi-million-line-codebase.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Vinay Gaba, Ankit Mathur, Rishabh Singh, Patrick Wendell, Matei Zaharia
- Published: Jul 8, 2026

Databricks built a coding-agent benchmark from reviewed tasks drawn from its multi-million-line codebase, spanning languages including Scala, Go, Rust, Python, and TypeScript. It uses recent human-written pull requests, held-out tests, standard model and harness setups, and manual validation rather than an LLM judge; Git history was sealed after traces showed agents could recover solutions. Results clustered models and harnesses into three capability tiers and showed that quality, cost per task, and token price can diverge: GLM 5.2 statistically tied Opus 4.8 on quality while costing $1.28 versus $1.94 per task, whereas Sonnet 5 cost $2.09 versus Opus’s $1.94. The same model through different harnesses produced cost differences of more than 2x in some cases while quality remained the same; Pi sent about three times less context per turn. Databricks plans to shift routine work toward cheaper models, deploy open models such as GLM as daily drivers, and improve model-and-harness selection.


### [How to Evaluate an Enterprise Analytics Platform](https://yomu.fyi/post/how-to-evaluate-an-enterprise-analytics-platform.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jul 8, 2026

Enterprise analytics platform evaluations often overemphasize dashboard interfaces, although the larger decision concerns whether analytics, AI and agents share data, semantics and governance. The post distinguishes point solutions from a unified platform and proposes seven evaluation criteria: workload fit, architecture and openness, governance and compliance, performance and scalability, adoption and usability, AI and ML readiness, and total cost of ownership. It recommends mapping current and three-year workloads, testing production-scale data with realistic concurrency, measuring p95 latency, and examining governance, usability, contracts and operational complexity in a proof of concept. Lakehouse architecture, open formats such as Delta Lake and Apache Iceberg, and shared controls are presented as ways to reduce context gaps; Databricks is offered as a practical example using Unity Catalog, Genie and Agent Bricks. The conclusion favors a weighted, three-year assessment over a feature comparison.


### [How Imperial College London is accelerating dementia research with a modern data platform](https://yomu.fyi/post/how-imperial-college-london-is-accelerating-dementia-research-with-a-m.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Kacey Hertan
- Published: Jul 7, 2026

Imperial College London’s Care Research and Technology Centre (CR&T) needed to scale Minder, a platform that combines in-home sensors, sleep monitors, and electronic health records to support dementia care and research. Growing data volumes caused competing workloads, tightly coupled storage and compute, and limited access for researchers and clinicians. The re-architected platform validates IoT data through Kubernetes, stores it in Delta Lake on Azure Data Lake Storage using bronze, silver, and anonymized gold layers, and adds Databricks analytics with Unity Catalog governance while retaining FHIR-based EHR interoperability and Kubeflow for model deployment. Research-to-production workflows, reusable pipelines, dashboards, and timestamped data improve collaboration, accessibility, and reproducibility. During migration, uptime remained 100%; new sources integrated in as little as one month, model development fell to about one month, and insights reached 581 people living with dementia in five months.


### [Contextual Policies in Omnigent: Using session state to better govern AI agents](https://yomu.fyi/post/contextual-policies-in-omnigent-using-session-state-to-better-govern-a.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Matei Zaharia, David Nasi, Xiangrui Meng, Kecheng Cao, Tomu Hirata
- Published: Jul 7, 2026

Omnigent, an open-source meta-harness for AI agents, introduces contextual policies to make agent controls safer and less disruptive than per-action allow, deny, or approval rules. Policies receive session events, maintain private state such as tools used, documents read, accumulated risk, initial intent, and model spend, then allow, deny, transform, or escalate the next action. Omnigent’s server intercepts tool calls from supported agents and applies these policies consistently, while examples include Google Drive restrictions based on documents created or marked confidential, risk thresholds that require approval for later email or file-sharing actions, and budget thresholds that pause or redirect work to a cheaper model. Intent-based authorization limits tools according to the user’s opening request, applying least privilege across supported harnesses and custom agents. The project is described as open source and alpha, with the server providing one interception layer for agents using different harnesses.


### [Barracuda makes security logs conversational with Genie](https://yomu.fyi/post/barracuda-makes-security-logs-conversational-with-genie.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Barracuda XDR Engineering Team
- Published: Jul 6, 2026

Barracuda Managed XDR needed to make security-log investigations accessible beyond analysts who knew each vendor’s schema and could write SQL. Its AI-Powered Log Search integrates Databricks Genie into the XDR dashboard: Genie translates natural-language questions into SQL, runs them against customer logs, and returns tabular results with the generated query. Security-specific metadata in Unity Catalog helps map requests across network, server, cloud, email, and endpoint sources, while multi-turn context supports iterative filtering. Secure views enforce organization-level filtering before execution; scoped service principals and SQL validation add further controls against cross-tenant access. Barracuda reports reducing routine investigation time from hours to minutes and estimates that self-service could free 83 to 100 SOC analyst hours monthly, or 1,000 to 1,200 annually.


### [Automatic Upgrades: best practice features for your lakehouse tables](https://yomu.fyi/post/automatic-upgrades-best-practice-features-for-your-lakehouse-tables.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Elizabeth Bowman, Tom van Bussel
- Published: Jul 6, 2026

Automatic Upgrades gives Unity Catalog managed tables a hands-off path to adopting best-practice table features without manually auditing thousands of tables or running ALTER TABLE repeatedly. It observes table access over a 100-day window, verifies that every accessing Databricks client supports the feature, skips inactive or externally accessed tables, and applies eligible changes through a lightweight background job. New tables inherit a feature when it becomes a schema default, while explicit table properties take precedence. Supported capabilities include Row Tracking, Automatic Liquid Clustering, Deletion Vectors, Column Mapping, Parquet V2, Catalog Commits, and Checkpoint V2, with benefits spanning query speed, storage and write costs, interoperability, and reliability. Changes are visible in DESCRIBE HISTORY and Catalog Explorer, reversible per table, and not re-enabled after a user disables one.


### [Reimagining Data Modeling on the Lakehouse: Introducing Vibe Data Modeling](https://yomu.fyi/post/reimagining-data-modeling-on-the-lakehouse-introducing-vibe-data-model.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Amr Ali, Cary Moore, Roberto Bruno Martins, Abhijit Tilak
- Published: Jul 6, 2026

Vibe Data Modeling is presented as a way to build a governed Silver-layer data model from a plain-English description of a business, addressing the months or years often required to hand-build one or adapt a generic industry template. The single-notebook implementation uses four widgets and a multi-model LLM agent to organize organizations into divisions, domains, subdomains, products, and attributes, then generate a logical model and deploy it to Unity Catalog. Its pipeline advances through four generate-and-validate stages, applying 251 rules, deterministic structural gates, two architect reviews, and a retry loop that changes strategy when checks fail. The authoritative model.json also produces schemas, Delta tables, informational foreign keys, classification tags, metric views, an RDFS ontology, DBML, and synthetic sample data, while plain-English refinements create auditable, reversible versions.


### [Scaling Security Alert Triage With Specialized Agents on Databricks](https://yomu.fyi/post/scaling-security-alert-triage-with-specialized-agents-on-databricks.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Leanne Shapton, Connor Hanify, Sam Pezzino
- Published: Jul 6, 2026

Databricks describes a security-alert triage system designed to review low-severity alerts that historically received less attention because of their volume and low fidelity. An initial single-agent prompt escalated 50% of alerts, so the team built a fleet of 17 source-specific agents plus a Threat Intelligence agent, using Structured Streaming to ingest alerts, enrich evidence, and route each alert. The pipeline combines deterministic benign-signal filtering, historical and behavioral context, specialized prompt functions, optional tools, structured dispositions, and cost controls before sending escalations to human analysts. Those analysts provide ground truth through MLflow traces and labels, supporting ongoing evaluation across escalate, monitor, and close decisions. After triaging more than 18,000 alerts, the system reached a 3.2% escalation rate, a 10.5-second median triage time, and more than 6,500 analyst hours saved in 30 days; escalated low-severity alerts were roughly 10 times more likely to be true positives than HIGH and MEDIUM alerts.


### [OpenAI and Databricks at DAIS 2026: Making enterprise AI real](https://yomu.fyi/post/openai-and-databricks-at-dais-2026-making-enterprise-ai-real.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Margaret Amori
- Published: Jul 6, 2026

At Data + AI Summit 2026, Databricks and OpenAI presented a partnership centered on combining OpenAI’s frontier models and agents with Databricks’ enterprise context and control. The post describes GPT models and Codex running natively on Databricks, with Unity AI Gateway governing interactions, budgets, routing, auditing, and visibility, while Agent Tools provide governed enterprise data access through MCPs. Examples include OpenAI’s marketing data foundation on Databricks, which cut storage costs by $400,000 per month, and a Hertz application built in 11 business days with GPT-5.5 and Databricks that lifted lead conversion from 60–65% to 75–80%. The sessions argue that deployment, security, evaluation, monitoring, context, and sharing comprise most of the work around enterprise agents. A joint virtual event scheduled for August will address shipping agentic applications at scale, including a Stellantis case study.


### [The 3 questions to answer to take AI from experimentation to impact](https://yomu.fyi/post/the-3-questions-to-answer-to-take-ai-from-experimentation-to-impact.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Christy Maver
- Published: Jul 2, 2026

The post argues that enterprises moving AI from experimentation to impact should answer three questions: whether employees and governance are ready, whether tools are accessible, and whether workers have the capabilities to use them. It recommends secure, governed AI agents that let employees experiment safely, apply consistent oversight across workloads, and build skills without compromising business security; the text notes that fewer than half of companies have formal governance for autonomous workloads. AI should appear inside natural workflows, including single chat interfaces and embedded intelligence dashboards, with access to company data, automated identity management, consistent governance, and business logic across engagements. Finally, agents should provide contextually accurate, actionable intelligence and automation, challenge users’ thinking, suggest next steps, and take action rather than only answer questions.


### [Inside the infrastructure strategies propelling AI leaders](https://yomu.fyi/post/inside-the-infrastructure-strategies-propelling-ai-leaders.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Christy Maver
- Published: Jul 2, 2026

The post examines why enterprise AI efforts can become too expensive, slow, and difficult to scale, citing survey findings that 67% of organizations with disconnected data environments identify storage, movement, and duplication as their largest recurring AI cost. It presents three infrastructure considerations: delivering infrastructure at agentic speeds, streamlining data, and adopting systems built for AI scale. Examples include instant temporary environments with secure rollback and restoration, unified operational and analytical data stored separately from compute in low-cost cloud storage, and elastic scaling that can move from high concurrency to zero in seconds. The conclusion is that open, AI-ready, purpose-built databases can reduce pipeline complexity and costs, support experimentation, and let organizations align spending with unpredictable workloads while enabling faster AI innovation.


### [How we keep GPUs reliable across Databricks AI](https://yomu.fyi/post/how-we-keep-gpus-reliable-across-databricks-ai.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Steven Chen, Feng Wang, Bhavik Soni, Chengguang Yang, Albert Zhong, Naren Loganathan, Harsh Panchal, Jianwei Xie
- Published: Jul 1, 2026

Databricks AI describes GPU reliability as a fleet-scale problem involving crashed jobs, silent slowdowns, and numerical corruption during distributed training. Its approach combines stress testing with demanding reinforcement learning, agentic coding, and document intelligence workloads, plus gpu-monitor, a multi-stage service that checks nodes during bootstrap, continuously while active, and periodically between workloads. The checks cover compute burn-in, peer connectivity, NCCL correctness and bandwidth, RDMA, ECC and HBM memory, PCIe integrity, throttling, thermal conditions, XID errors, and fabric health. A production incident showed why cumulative InfiniBand port downtime matters: a single outage exceeded the lower-level NCCL\_IB\_TIMEOUT, killing a collective before the longer PyTorch watchdog timeout. Nodes that fail checks are quarantined, while inter-node probes sweep message sizes from 8 bytes to 2 GiB and apply latency or BusBW criteria; the system is continuously expanded as new failure modes emerge.


### [Celebrating the Winners of the 2026 Built-On Databricks Startup Challenge](https://yomu.fyi/post/celebrating-the-winners-of-the-2026-built-on-databricks-startup-challe.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Andrew Ferguson, Joslyn O'Connell, Sully Clark
- Published: Jul 1, 2026

The 2026 Built-On Databricks Startup Challenge recognized early-stage startups building B2B applications on Databricks, with winners presented at the 2026 Data + AI Summit. VisionHeight won the Grand Prize for an agentic threat-intelligence platform that maps adversary infrastructure across the Internet while it is being constructed, aiming to give defenders earlier warning. Linkup took second place with a production-grade Web Search API that independently crawls and indexes the open web at the fact level, delivering sourced results in about two seconds. Intelo placed third with five coordinated AI Agent Teams covering retail merchandising and planning, while Clarecast, Gemini Sports, and LakeFusion received Honorable Mentions. Judges assessed market potential, founding-team caliber, and innovative Databricks use; the announced Startup Program also offers qualifying startups up to $200,000 in combined Databricks and Neon credits.


### [Granular Usage Attribution for dbt Pipelines with Query Tags](https://yomu.fyi/post/granular-usage-attribution-for-dbt-pipelines-with-query-tags.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Heeren Sharma, Lennart Reschke, JooHo Yeo
- Published: Jul 1, 2026

Databricks Query Tags give dbt pipelines granular usage attribution when identical query-history rows obscure model ownership, cost, and performance. The dbt-databricks adapter version 1.11+ automatically injects model, materialization, dbt-core, and adapter-version metadata, while profile-level and model-level configurations add custom dimensions such as team, cost center, project, and environment. These tags are stored as a MAP<STRING, STRING> in system.query.history, where SQL can extract them for cost attribution, performance debugging, monitoring, and dashboards. In the reference project, four mart tables accounted for 92% of compute time, whereas staging views and metric views were near-instantaneous. The accompanying open-source project demonstrates Genie exploration, a self-monitoring AI/BI dashboard, metric-view tagging, and deployment with Databricks bundles, while recommending consistent profile hierarchies, environment tags, and limited custom metadata.


[Newer posts](https://yomu.fyi/company/databricks/page/6.md) · [Older posts](https://yomu.fyi/company/databricks/page/8.md)