---
title: "Databricks"
description: "161 posts about Databricks, summarised, each linking to the original."
---

# Databricks
> 161 posts about Databricks, summarised, each linking to the original.

## Articles

### [Unified context: The missing layer for enterprise AI coworkers](https://yomu.fyi/post/unified-context-the-missing-layer-for-enterprise-ai-coworkers.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Cynthya Peranandam, Christy Maver
- Published: Jul 16, 2026

Enterprise AI assistants often produce fluent answers yet fail to improve forecast calls, deal reviews, and operational standups because decision context is scattered across systems, teams, and competing definitions. Genie One addresses this by using a shared context layer spanning Databricks data, documents, SaaS applications, and operational systems, allowing questions and follow-up work to retain business meaning. Genie Ontology organizes terms, metrics, entities, and relationships into a living knowledge graph, learning from data, dashboards, queries, documents, and connected applications while ranking definitions and signals using usage and certified-asset links. Together with Unity Catalog, it applies permissions, certified data, shared definitions, and governance controls to answers, actions, and agents. The stated outcome is faster movement from decision preparation to action, with less manual reconciliation while preserving accuracy and control.


### [AI-Enabled Advisory Services for Higher Education](https://yomu.fyi/post/ai-enabled-advisory-services-for-higher-education.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Chad Ammirati, Zach Langford, Nicole Wong
- Published: Jul 15, 2026

Higher-education call centers face costly, limited-coverage monitoring of advisor conversations and brittle, slow methods for identifying student concerns from transcripts. The proposed workflow deploys OpenAI Whisper on Databricks Model Serving, applies AI Functions for sentiment, topics, intent, and rubric scoring, and uses Unity Catalog to govern the resulting data. For advisor quality, an LLM-as-a-judge evaluates every transcript against a reference-table rubric, returns a weighted 1–5 overall score and per-criterion scores, and routes flagged calls for targeted QA review instead of random sampling. For student insights, quarterly transcript enrichment feeds an Agent Bricks Knowledge Assistant for cited reasoning over raw calls and a Genie Space for structured trend queries, while LangGraph orchestrates UC SQL functions as tools. Together, these components let non-technical advisors, mentors, and QA managers query student interactions without reaching out to a data SME.


### [How Retail Finance teams are using Agentic AI to protect omni-channel margins](https://yomu.fyi/post/how-retail-finance-teams-are-using-agentic-ai-to-protect-omni-channel.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Sarah Duffy
- Published: Jul 14, 2026

Omni-channel retail has spread margin, cash, and markdown decisions across more channels, fulfillment paths, and return routes, while agentic systems increase the speed and complexity of change. The post presents ontology as a way to preserve the meaning and business context behind finance figures, keeping definitions, channels, and cost drivers current. Databricks Genie is described as a data-smart AI coworker that answers finance questions in plain language, grounding responses in an evolving ontology, source traces, permissions, and governed AI costs. It focuses on margin after fulfillment and returns, inventory cash tied up in the wrong place, and full-price revenue at risk from markdowns and returns, then prepares actions for a person to approve. Unilever deployed Genie to more than 1,200 finance and business users; analysis that took days now takes minutes, with expected multi-million-euro annual cost avoidance.


### [Practical Guide To Python App Hosting](https://yomu.fyi/post/practical-guide-to-python-app-hosting.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Jul 13, 2026

Python app hosting is presented as both an infrastructure choice and, for data-intensive or AI applications, a data-architecture choice. The guide defines a hosting stack comprising application code, runtime, dependency manager, application server and reverse proxy, and contrasts it with traditional web hosting. It compares shared hosting, VPSs and cloud VMs, PaaS, container platforms and serverless functions by control, operational effort, scalability and workload fit. It notes that long-running data and AI tasks such as model training, vector indexes, cached datasets, ETL jobs and lengthy inference pipelines can exceed practical serverless limits. The conclusion is that public APIs and lightweight web apps allow broad platform choice, while apps accessing governed data, model endpoints or AI agents should be evaluated by where data lives and how access is governed, with Databricks Apps presented as an option when data already resides in a lakehouse.


### [The agentic marketing stack starts with the data layer](https://yomu.fyi/post/the-agentic-marketing-stack-starts-with-the-data-layer.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Aly McGue
- Published: Jul 10, 2026

The piece argues that agentic marketing depends on modernizing the data and infrastructure foundation first, rather than starting with isolated AI use cases. At Acxiom, moving core products and client solutions from on-premises Hadoop to Databricks reportedly improved workload runtimes by 80 to 90 percent, reducing jobs that took 50+ hours or as long as 90+ hours to 2–3 hours. The migration also reduced manual pipeline and infrastructure work, freeing engineering capacity for products and client outcomes. Those capabilities now support agentic audience planning, media buying, campaign activation, performance analytics, code generation, testing, and ad variation, with a goal of connecting the full marketing value chain. Because workflows handle PII, generated content passes through legal approval, defined controls, and human oversight; Acxiom is also embedding its data in client platforms and privacy-safe clean rooms to make decisions more transparent and native to customer environments.


### [The ambulatory intelligence gap](https://yomu.fyi/post/the-ambulatory-intelligence-gap.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Morgan Wilkie, Bryan Smith, Holly Burke, Mary Ellen
- Published: Jul 10, 2026

Health systems face an ambulatory intelligence gap: access, provider capacity, referral retention, panel management, and financial performance are interconnected, while information remains scattered across disconnected systems. Health Catalyst’s Ambulatory Intelligence combines AI with nearly two decades of healthcare improvement expertise, deploying directly in a customer’s Databricks workspace so sensitive data stays within the health system’s environment. It uses a medallion-based semantic layer, Unity Catalog for governance, Lakebase for low-latency serving, and Genie alongside dashboards to help leaders investigate why metrics change. The solution ships with prebuilt metrics across Access Optimization, Revenue Intelligence, Panel Management, and Referral Insights, plus cross-domain scorecards and configurable terminology and workflows. Reported outcomes from supported improvement work include increased revenue and encounters at Thibodaux Regional, 55,000 closed care gaps at INTEGRIS Health, and higher outpatient visits with fewer cancellations without reschedules at WakeMed; future plans include models based on prior outcomes and agentic capabilities.


### [Ask, build, compose: What our 5th Genie Hackathon taught us about Databricks Genie](https://yomu.fyi/post/ask-build-compose-what-our-5th-genie-hackathon-taught-us-about-databri.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Shruti Prasanna, Rob Bajra
- Published: Jul 9, 2026

The fifth Databricks hackathon used Databricks Genie to test three ways of working with governed data: asking, building, and composing. Genie Agents give business users domain-specific natural-language access to curated data, while Genie Code helps analysts create metric views, functions, pipelines, and dashboards inside Databricks. The composition track uses Conversation APIs and a managed MCP server to make Genie an agent tool alongside custom logic, Model Serving, and other MCP servers, with Unity Catalog governing access. Projects included supervisory routing across 190 tables, governance and analytics products built in days, and multi-agent systems such as ShipBob’s overnight operations brief. Across the tracks, the stated lesson is that shared governance and semantic context let business users, builders, and engineers use Genie at different levels without abandoning grounded, permissioned data access.


### [Navigating a Synapse Migration to Databricks](https://yomu.fyi/post/navigating-a-synapse-migration-to-databricks.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Olga Romanova, Johannes Oehler
- Published: Jul 8, 2026

The post presents migration from Azure Synapse to Databricks as a way to reduce platform complexity while supporting data engineering, analytics, machine learning, real-time pipelines, and AI applications. It distinguishes migration paths for Dedicated SQL Pools, serverless SQL, Spark Pools, orchestration, governance, and downstream BI connectivity, noting that accumulated T-SQL logic and dependencies require the most redesign, testing, and validation. The proposed programme uses discovery and assessment with Lakebridge Profiler and Analyzer, then selects an architecture, sequences workloads, pilots a real use case, and scales migration in waves. It recommends automating code conversion, reconciling row counts, aggregations, hashes, and tolerances, avoiding direct Synapse distribution-to-Delta partition mappings, and planning compaction, enablement, and rollback. Reported examples include Casey’s halving operational data delivery times from eight hours to four and Italgas cutting workload costs by 73%.


### [How Imperial College London is accelerating dementia research with a modern data platform](https://yomu.fyi/post/how-imperial-college-london-is-accelerating-dementia-research-with-a-m.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Kacey Hertan
- Published: Jul 7, 2026

Imperial College London’s Care Research and Technology Centre (CR&T) needed to scale Minder, a platform that combines in-home sensors, sleep monitors, and electronic health records to support dementia care and research. Growing data volumes caused competing workloads, tightly coupled storage and compute, and limited access for researchers and clinicians. The re-architected platform validates IoT data through Kubernetes, stores it in Delta Lake on Azure Data Lake Storage using bronze, silver, and anonymized gold layers, and adds Databricks analytics with Unity Catalog governance while retaining FHIR-based EHR interoperability and Kubeflow for model deployment. Research-to-production workflows, reusable pipelines, dashboards, and timestamped data improve collaboration, accessibility, and reproducibility. During migration, uptime remained 100%; new sources integrated in as little as one month, model development fell to about one month, and insights reached 581 people living with dementia in five months.


### [Barracuda makes security logs conversational with Genie](https://yomu.fyi/post/barracuda-makes-security-logs-conversational-with-genie.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Barracuda XDR Engineering Team
- Published: Jul 6, 2026

Barracuda Managed XDR needed to make security-log investigations accessible beyond analysts who knew each vendor’s schema and could write SQL. Its AI-Powered Log Search integrates Databricks Genie into the XDR dashboard: Genie translates natural-language questions into SQL, runs them against customer logs, and returns tabular results with the generated query. Security-specific metadata in Unity Catalog helps map requests across network, server, cloud, email, and endpoint sources, while multi-turn context supports iterative filtering. Secure views enforce organization-level filtering before execution; scoped service principals and SQL validation add further controls against cross-tenant access. Barracuda reports reducing routine investigation time from hours to minutes and estimates that self-service could free 83 to 100 SOC analyst hours monthly, or 1,000 to 1,200 annually.


### [Scaling Security Alert Triage With Specialized Agents on Databricks](https://yomu.fyi/post/scaling-security-alert-triage-with-specialized-agents-on-databricks.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Leanne Shapton, Connor Hanify, Sam Pezzino
- Published: Jul 6, 2026

Databricks describes a security-alert triage system designed to review low-severity alerts that historically received less attention because of their volume and low fidelity. An initial single-agent prompt escalated 50% of alerts, so the team built a fleet of 17 source-specific agents plus a Threat Intelligence agent, using Structured Streaming to ingest alerts, enrich evidence, and route each alert. The pipeline combines deterministic benign-signal filtering, historical and behavioral context, specialized prompt functions, optional tools, structured dispositions, and cost controls before sending escalations to human analysts. Those analysts provide ground truth through MLflow traces and labels, supporting ongoing evaluation across escalate, monitor, and close decisions. After triaging more than 18,000 alerts, the system reached a 3.2% escalation rate, a 10.5-second median triage time, and more than 6,500 analyst hours saved in 30 days; escalated low-severity alerts were roughly 10 times more likely to be true positives than HIGH and MEDIUM alerts.


### [OpenAI and Databricks at DAIS 2026: Making enterprise AI real](https://yomu.fyi/post/openai-and-databricks-at-dais-2026-making-enterprise-ai-real.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Margaret Amori
- Published: Jul 6, 2026

At Data + AI Summit 2026, Databricks and OpenAI presented a partnership centered on combining OpenAI’s frontier models and agents with Databricks’ enterprise context and control. The post describes GPT models and Codex running natively on Databricks, with Unity AI Gateway governing interactions, budgets, routing, auditing, and visibility, while Agent Tools provide governed enterprise data access through MCPs. Examples include OpenAI’s marketing data foundation on Databricks, which cut storage costs by $400,000 per month, and a Hertz application built in 11 business days with GPT-5.5 and Databricks that lifted lead conversion from 60–65% to 75–80%. The sessions argue that deployment, security, evaluation, monitoring, context, and sharing comprise most of the work around enterprise agents. A joint virtual event scheduled for August will address shipping agentic applications at scale, including a Stellantis case study.


### [Celebrating the Winners of the 2026 Built-On Databricks Startup Challenge](https://yomu.fyi/post/celebrating-the-winners-of-the-2026-built-on-databricks-startup-challe.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Andrew Ferguson, Joslyn O'Connell, Sully Clark
- Published: Jul 1, 2026

The 2026 Built-On Databricks Startup Challenge recognized early-stage startups building B2B applications on Databricks, with winners presented at the 2026 Data + AI Summit. VisionHeight won the Grand Prize for an agentic threat-intelligence platform that maps adversary infrastructure across the Internet while it is being constructed, aiming to give defenders earlier warning. Linkup took second place with a production-grade Web Search API that independently crawls and indexes the open web at the fact level, delivering sourced results in about two seconds. Intelo placed third with five coordinated AI Agent Teams covering retail merchandising and planning, while Clarecast, Gemini Sports, and LakeFusion received Honorable Mentions. Judges assessed market potential, founding-team caliber, and innovative Databricks use; the announced Startup Program also offers qualifying startups up to $200,000 in combined Databricks and Neon credits.


### [Granular Usage Attribution for dbt Pipelines with Query Tags](https://yomu.fyi/post/granular-usage-attribution-for-dbt-pipelines-with-query-tags.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Heeren Sharma, Lennart Reschke, JooHo Yeo
- Published: Jul 1, 2026

Databricks Query Tags give dbt pipelines granular usage attribution when identical query-history rows obscure model ownership, cost, and performance. The dbt-databricks adapter version 1.11+ automatically injects model, materialization, dbt-core, and adapter-version metadata, while profile-level and model-level configurations add custom dimensions such as team, cost center, project, and environment. These tags are stored as a MAP<STRING, STRING> in system.query.history, where SQL can extract them for cost attribution, performance debugging, monitoring, and dashboards. In the reference project, four mart tables accounted for 92% of compute time, whereas staging views and metric views were near-instantaneous. The accompanying open-source project demonstrates Genie exploration, a self-monitoring AI/BI dashboard, metric-view tagging, and deployment with Databricks bundles, while recommending consistent profile hierarchies, environment tags, and limited custom metadata.


### [Forecasting at the speed of modern retail](https://yomu.fyi/post/forecasting-at-the-speed-of-modern-retail.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ryuta Yoshimatsu, Puneet Jain, Lourdes Angélica Martinez Medina, Lucas Bruand, Dael Williamson
- Published: Jul 1, 2026

Retail and CPG forecasting now spans hundreds of thousands, sometimes far more, time series across fragmented channels, promotions, and short-lived products, making legacy methods and manual exception management difficult. Multi-model forecasting addresses this complexity by evaluating a range of techniques against actual data and selecting the best-performing model for each series, but enterprise-scale experiments require scarce forecasting and distributed-systems expertise. Released in 2024, Databricks’ open-source Many Model Forecasting (MMF) integrates more than 35 statistical, deep-learning, and foundation time-series models and runs on distributed Databricks compute. MMF Agent, built on Genie Code, guides users through data quality, series classification, compute configuration, forecasting, post-processing, and model selection, while Unity Catalog helps it use organizational data context. The workflow is intended to reduce setup from days to hours, improve targeting and accuracy, and make rigorous forecasting more accessible while remaining customizable for technical teams.


### [Celebrating the Winners of the 2026 Built-On Databricks Startup Challenge - Cloned](https://yomu.fyi/post/celebrating-the-winners-of-the-2026-built-on-databricks-startup-challe-cloned.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Andrew Ferguson, Joslyn O'Connell, Sully Clark
- Published: Jul 1, 2026

The 2026 Built-On Databricks Startup Challenge recognized early-stage startups building B2B applications on the Databricks platform, with winners announced at the Data + AI Summit in San Francisco on June 16. VisionHeight won the grand prize for an agentic threat-intelligence platform that maps adversary infrastructure across the Internet while it is being built; Linkup placed second with a production-grade Web Search API delivering results in about two seconds, and Intelo placed third with an agentic workforce for retail merchandising and planning. Clarecast, Gemini Sports, and LakeFusion received honorable mentions for predictive intelligence, football squad planning, and an AI-native data foundation combining MDM, PIM, and LakeGraph. Judges assessed market potential, founding-team caliber, and innovative use of Databricks, while the startup-program offer provides qualifying startups up to $200,000 in credits across Databricks and Neon.


### [How Databricks is turning video into searchable, actionable intelligence](https://yomu.fyi/post/how-databricks-is-turning-video-into-searchable-actionable-intelligenc.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Justin Monaldo, Kacey Hertan, Yvan Aquino
- Published: Jun 26, 2026

Databricks presents video analysis as a data engineering problem for organizations with terabytes of footage that is difficult and expensive to review manually. An app accepts a video and a natural-language prompt, then triggers a Lakeflow job on Serverless GPU Compute to run Meta’s SAM3 segmentation model frame by frame and retain matching moments. Those clips preserve original timestamps and are sent through the Databricks Foundation Model API for summaries that can be written to tables or passed into downstream workflows. In one example, 26 minutes of traffic footage became one minute and 55 seconds of relevant video. The model-agnostic pipeline uses MLflow signatures to support interchangeable or custom models, while event-driven execution and independent serverless GPUs allow concurrent processing without cluster management or idle GPU costs.


### [A Decision Framework for ETL Migration to Databricks](https://yomu.fyi/post/a-decision-framework-for-etl-migration-to-databricks.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Rafael Aielo
- Published: Jun 26, 2026

An ETL migration with hundreds of stored procedures, schedulers, scattered permissions, and a warehouse renewal deadline needs workload-by-workload decisions rather than a single rewrite strategy. The framework assigns work among Databricks SQL, Spark Declarative Pipelines (SDP), and PySpark or Spark SQL notebooks. SQL tasks suit single statements, Unity Catalog-governed stored procedures handle procedural logic, SDP manages dependencies, retries, quality constraints, and batch-plus-streaming, while notebooks cover complex logic, ML feature engineering, integrations, and large or tightly controlled Spark workloads. It recommends four phases—assessment, quick wins, modernization, and optimization—using profiling, side-by-side validation, and parallel runs before retiring legacy systems. Migration tools can automate 60–80% of initial conversion, but architecture choices remain essential: the goal is consolidating orchestration, metadata, lineage, permissions, and validation rather than reproducing technical debt.


### [How the English Office for Students leverages Databricks to enhance higher education standards and drive better student outcomes](https://yomu.fyi/post/how-the-english-office-for-students-leverages-databricks-to-enhance-hi.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Kacey Hertan
- Published: Jun 26, 2026

The Office for Students, which regulates more than 400 higher education providers in England, modernised its data and analytics environment after a legacy platform could no longer handle growing volumes, varied sources, or emerging analytical demands. Its data spans millions of student records collected over 15 to 20 years, and a workflow processing about 300 million records took eight hours. Moving to Databricks consolidated structured, qualitative, and near-live data with analytics and AI workflows, while Unity Catalog added lineage, access controls, and security patterns for governed use. Genie Code reduced a student segmentation analysis from at least two weeks for two analysts to half a day, and a provider-registration triage proof of concept flags missing submissions earlier. The organisation frames AI as decision support rather than decision-making, keeping humans responsible for regulatory judgments.


### [From test bench to lakehouse: how AVL modernizes measurement data analytics with Impulse](https://yomu.fyi/post/from-test-bench-to-lakehouse-how-avl-modernizes-measurement-data-analy.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Dr. Thomas Bonfert, Jonathan Bräuer, Fabian Ade, Maxim Hammer, Florian Gorzitzke, David Crescence, Christa Simon, Jörg Zimmermann, Hannes Schneider
- Published: Jun 25, 2026

AVL’s Lakehouse for Measurement Data addresses the scale, reproducibility, and governance limitations of desktop tools and isolated scripts used for automotive measurement analysis. Built on Databricks, the platform ingests ASAM MDF4 and other files into a Medallion Architecture, applies configurable DQX quality rules in a hierarchical Silver model, and uses Impulse to compile declarative Python TSAL expressions into distributed Spark execution. Engineers can select channels, create virtual signals with alias resolution, unit conversion, time alignment, and interpolation, define events, and compute duration- or distance-weighted aggregations in about 10 lines of Python. Impulse supports Gold-layer reporting, ad-hoc Spark DataFrames, and ML feature matrices, with Unity Catalog governance and Workflow orchestration. AVL reports reducing analysis time from days to minutes, processing many recordings per run, lowering infrastructure costs versus on-premises solutions, and enabling self-service, reproducible, standardized analysis.


[Newer posts](https://yomu.fyi/topic/databricks/page/2.md) · [Older posts](https://yomu.fyi/topic/databricks/page/4.md)