Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
- 542 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
How Imperial College London is accelerating dementia research with a modern data platform
Imperial College London’s Care Research and Technology Centre (CR&T) needed to scale Minder, a platform that combines in-home sensors, sleep monitors, and electronic health records to support dementia care and research. Growing data volumes caused competing workloads, tightly coupled storage and compute, and limited access for researchers and clinicians. The re-architected platform validates IoT data through Kubernetes, stores it in Delta Lake on Azure Data Lake Storage using bronze, silver, and anonymized gold layers, and adds Databricks analytics with Unity Catalog governance while retaining FHIR-based EHR interoperability and Kubeflow for model deployment. Research-to-production workflows, reusable pipelines, dashboards, and timestamped data improve collaboration, accessibility, and reproducibility. During migration, uptime remained 100%; new sources integrated in as little as one month, model development fell to about one month, and insights reached 581 people living with dementia in five months.
Contextual Policies in Omnigent: Using session state to better govern AI agents
Omnigent, an open-source meta-harness for AI agents, introduces contextual policies to make agent controls safer and less disruptive than per-action allow, deny, or approval rules. Policies receive session events, maintain private state such as tools used, documents read, accumulated risk, initial intent, and model spend, then allow, deny, transform, or escalate the next action. Omnigent’s server intercepts tool calls from supported agents and applies these policies consistently, while examples include Google Drive restrictions based on documents created or marked confidential, risk thresholds that require approval for later email or file-sharing actions, and budget thresholds that pause or redirect work to a cheaper model. Intent-based authorization limits tools according to the user’s opening request, applying least privilege across supported harnesses and custom agents. The project is described as open source and alpha, with the server providing one interception layer for agents using different harnesses.
Barracuda makes security logs conversational with Genie
Barracuda Managed XDR needed to make security-log investigations accessible beyond analysts who knew each vendor’s schema and could write SQL. Its AI-Powered Log Search integrates Databricks Genie into the XDR dashboard: Genie translates natural-language questions into SQL, runs them against customer logs, and returns tabular results with the generated query. Security-specific metadata in Unity Catalog helps map requests across network, server, cloud, email, and endpoint sources, while multi-turn context supports iterative filtering. Secure views enforce organization-level filtering before execution; scoped service principals and SQL validation add further controls against cross-tenant access. Barracuda reports reducing routine investigation time from hours to minutes and estimates that self-service could free 83 to 100 SOC analyst hours monthly, or 1,000 to 1,200 annually.
Automatic Upgrades: best practice features for your lakehouse tables
Automatic Upgrades gives Unity Catalog managed tables a hands-off path to adopting best-practice table features without manually auditing thousands of tables or running ALTER TABLE repeatedly. It observes table access over a 100-day window, verifies that every accessing Databricks client supports the feature, skips inactive or externally accessed tables, and applies eligible changes through a lightweight background job. New tables inherit a feature when it becomes a schema default, while explicit table properties take precedence. Supported capabilities include Row Tracking, Automatic Liquid Clustering, Deletion Vectors, Column Mapping, Parquet V2, Catalog Commits, and Checkpoint V2, with benefits spanning query speed, storage and write costs, interoperability, and reliability. Changes are visible in DESCRIBE HISTORY and Catalog Explorer, reversible per table, and not re-enabled after a user disables one.
Reimagining Data Modeling on the Lakehouse: Introducing Vibe Data Modeling
Vibe Data Modeling is presented as a way to build a governed Silver-layer data model from a plain-English description of a business, addressing the months or years often required to hand-build one or adapt a generic industry template. The single-notebook implementation uses four widgets and a multi-model LLM agent to organize organizations into divisions, domains, subdomains, products, and attributes, then generate a logical model and deploy it to Unity Catalog. Its pipeline advances through four generate-and-validate stages, applying 251 rules, deterministic structural gates, two architect reviews, and a retry loop that changes strategy when checks fail. The authoritative model.json also produces schemas, Delta tables, informational foreign keys, classification tags, metric views, an RDFS ontology, DBML, and synthetic sample data, while plain-English refinements create auditable, reversible versions.
Scaling Security Alert Triage With Specialized Agents on Databricks
Databricks describes a security-alert triage system designed to review low-severity alerts that historically received less attention because of their volume and low fidelity. An initial single-agent prompt escalated 50% of alerts, so the team built a fleet of 17 source-specific agents plus a Threat Intelligence agent, using Structured Streaming to ingest alerts, enrich evidence, and route each alert. The pipeline combines deterministic benign-signal filtering, historical and behavioral context, specialized prompt functions, optional tools, structured dispositions, and cost controls before sending escalations to human analysts. Those analysts provide ground truth through MLflow traces and labels, supporting ongoing evaluation across escalate, monitor, and close decisions. After triaging more than 18,000 alerts, the system reached a 3.2% escalation rate, a 10.5-second median triage time, and more than 6,500 analyst hours saved in 30 days; escalated low-severity alerts were roughly 10 times more likely to be true positives than HIGH and MEDIUM alerts.
OpenAI and Databricks at DAIS 2026: Making enterprise AI real
At Data + AI Summit 2026, Databricks and OpenAI presented a partnership centered on combining OpenAI’s frontier models and agents with Databricks’ enterprise context and control. The post describes GPT models and Codex running natively on Databricks, with Unity AI Gateway governing interactions, budgets, routing, auditing, and visibility, while Agent Tools provide governed enterprise data access through MCPs. Examples include OpenAI’s marketing data foundation on Databricks, which cut storage costs by $400,000 per month, and a Hertz application built in 11 business days with GPT-5.5 and Databricks that lifted lead conversion from 60–65% to 75–80%. The sessions argue that deployment, security, evaluation, monitoring, context, and sharing comprise most of the work around enterprise agents. A joint virtual event scheduled for August will address shipping agentic applications at scale, including a Stellantis case study.
The 3 questions to answer to take AI from experimentation to impact
The post argues that enterprises moving AI from experimentation to impact should answer three questions: whether employees and governance are ready, whether tools are accessible, and whether workers have the capabilities to use them. It recommends secure, governed AI agents that let employees experiment safely, apply consistent oversight across workloads, and build skills without compromising business security; the text notes that fewer than half of companies have formal governance for autonomous workloads. AI should appear inside natural workflows, including single chat interfaces and embedded intelligence dashboards, with access to company data, automated identity management, consistent governance, and business logic across engagements. Finally, agents should provide contextually accurate, actionable intelligence and automation, challenge users’ thinking, suggest next steps, and take action rather than only answer questions.
Inside the infrastructure strategies propelling AI leaders
The post examines why enterprise AI efforts can become too expensive, slow, and difficult to scale, citing survey findings that 67% of organizations with disconnected data environments identify storage, movement, and duplication as their largest recurring AI cost. It presents three infrastructure considerations: delivering infrastructure at agentic speeds, streamlining data, and adopting systems built for AI scale. Examples include instant temporary environments with secure rollback and restoration, unified operational and analytical data stored separately from compute in low-cost cloud storage, and elastic scaling that can move from high concurrency to zero in seconds. The conclusion is that open, AI-ready, purpose-built databases can reduce pipeline complexity and costs, support experimentation, and let organizations align spending with unpredictable workloads while enabling faster AI innovation.
How we keep GPUs reliable across Databricks AI
Databricks AI describes GPU reliability as a fleet-scale problem involving crashed jobs, silent slowdowns, and numerical corruption during distributed training. Its approach combines stress testing with demanding reinforcement learning, agentic coding, and document intelligence workloads, plus gpu-monitor, a multi-stage service that checks nodes during bootstrap, continuously while active, and periodically between workloads. The checks cover compute burn-in, peer connectivity, NCCL correctness and bandwidth, RDMA, ECC and HBM memory, PCIe integrity, throttling, thermal conditions, XID errors, and fabric health. A production incident showed why cumulative InfiniBand port downtime matters: a single outage exceeded the lower-level NCCL_IB_TIMEOUT, killing a collective before the longer PyTorch watchdog timeout. Nodes that fail checks are quarantined, while inter-node probes sweep message sizes from 8 bytes to 2 GiB and apply latency or BusBW criteria; the system is continuously expanded as new failure modes emerge.
Celebrating the Winners of the 2026 Built-On Databricks Startup Challenge
The 2026 Built-On Databricks Startup Challenge recognized early-stage startups building B2B applications on Databricks, with winners presented at the 2026 Data + AI Summit. VisionHeight won the Grand Prize for an agentic threat-intelligence platform that maps adversary infrastructure across the Internet while it is being constructed, aiming to give defenders earlier warning. Linkup took second place with a production-grade Web Search API that independently crawls and indexes the open web at the fact level, delivering sourced results in about two seconds. Intelo placed third with five coordinated AI Agent Teams covering retail merchandising and planning, while Clarecast, Gemini Sports, and LakeFusion received Honorable Mentions. Judges assessed market potential, founding-team caliber, and innovative Databricks use; the announced Startup Program also offers qualifying startups up to $200,000 in combined Databricks and Neon credits.
Granular Usage Attribution for dbt Pipelines with Query Tags
Databricks Query Tags give dbt pipelines granular usage attribution when identical query-history rows obscure model ownership, cost, and performance. The dbt-databricks adapter version 1.11+ automatically injects model, materialization, dbt-core, and adapter-version metadata, while profile-level and model-level configurations add custom dimensions such as team, cost center, project, and environment. These tags are stored as a MAP<STRING, STRING> in system.query.history, where SQL can extract them for cost attribution, performance debugging, monitoring, and dashboards. In the reference project, four mart tables accounted for 92% of compute time, whereas staging views and metric views were near-instantaneous. The accompanying open-source project demonstrates Genie exploration, a self-monitoring AI/BI dashboard, metric-view tagging, and deployment with Databricks bundles, while recommending consistent profile hierarchies, environment tags, and limited custom metadata.
Beyond dashboards: Introducing Decision Execution Platforms
Databricks Forward Deployed Engineering introduces Decision Execution Platforms (DEPs), an enterprise analytics category intended to connect KPI signals, executive decisions, operational execution, and measured outcomes. The proposal addresses workflows in which dashboards reveal problems but meetings, decks, spreadsheets, and messaging threads leave implementation fragmented and impact measurement disconnected. DEPs run the four-stage loop on governed Databricks infrastructure: agents recommend actions, alternatives, predicted impact, and reasoning; approved choices execute through systems of record; and results persist in a Decision Log for continuous learning. Their architecture combines a foundation of Lakebase, Genie, Unity Catalog, Lakehouse, Agent Bricks, and MLflow with an SDK of reusable primitives and a Databricks Apps executive surface. A retailer case used a DEP to unify fulfillment data and enable simulated, controlled rerouting, with scaling aimed at measurable bottom-line and customer-satisfaction outcomes.
Forecasting at the speed of modern retail
Retail and CPG forecasting now spans hundreds of thousands, sometimes far more, time series across fragmented channels, promotions, and short-lived products, making legacy methods and manual exception management difficult. Multi-model forecasting addresses this complexity by evaluating a range of techniques against actual data and selecting the best-performing model for each series, but enterprise-scale experiments require scarce forecasting and distributed-systems expertise. Released in 2024, Databricks’ open-source Many Model Forecasting (MMF) integrates more than 35 statistical, deep-learning, and foundation time-series models and runs on distributed Databricks compute. MMF Agent, built on Genie Code, guides users through data quality, series classification, compute configuration, forecasting, post-processing, and model selection, while Unity Catalog helps it use organizational data context. The workflow is intended to reduce setup from days to hours, improve targeting and accuracy, and make rigorous forecasting more accessible while remaining customizable for technical teams.
Celebrating the Winners of the 2026 Built-On Databricks Startup Challenge - Cloned
The 2026 Built-On Databricks Startup Challenge recognized early-stage startups building B2B applications on the Databricks platform, with winners announced at the Data + AI Summit in San Francisco on June 16. VisionHeight won the grand prize for an agentic threat-intelligence platform that maps adversary infrastructure across the Internet while it is being built; Linkup placed second with a production-grade Web Search API delivering results in about two seconds, and Intelo placed third with an agentic workforce for retail merchandising and planning. Clarecast, Gemini Sports, and LakeFusion received honorable mentions for predictive intelligence, football squad planning, and an AI-native data foundation combining MDM, PIM, and LakeGraph. Judges assessed market potential, founding-team caliber, and innovative use of Databricks, while the startup-program offer provides qualifying startups up to $200,000 in credits across Databricks and Neon.
From monolith to Lakebase to LTAP: rethinking the database from storage up
The post examines why monolithic OLTP databases become fragile and difficult to scale when compute, the write-ahead log (WAL), and data files share one machine. Using Postgres as its primary example, it contrasts local WAL durability and physical cloning with Lakebase, whose stateless Postgres compute externalizes those components into SafeKeeper and PageServer services. SafeKeeper replicates log records across a quorum through Paxos-based network replication, while PageServer applies the WAL and materializes data in cloud object storage. The resulting separation is presented as a way to improve durability, independently scale reads and writes, isolate workloads, and support high availability and branching without physical database clones. LTAP extends the design by making transactional tables directly queryable for analytics from a single governed copy, avoiding CDC or mirroring and allowing transactional and analytical engines to scale independently.
How Databricks is turning video into searchable, actionable intelligence
Databricks presents video analysis as a data engineering problem for organizations with terabytes of footage that is difficult and expensive to review manually. An app accepts a video and a natural-language prompt, then triggers a Lakeflow job on Serverless GPU Compute to run Meta’s SAM3 segmentation model frame by frame and retain matching moments. Those clips preserve original timestamps and are sent through the Databricks Foundation Model API for summaries that can be written to tables or passed into downstream workflows. In one example, 26 minutes of traffic footage became one minute and 55 seconds of relevant video. The model-agnostic pipeline uses MLflow signatures to support interchangeable or custom models, while event-driven execution and independent serverless GPUs allow concurrent processing without cluster management or idle GPU costs.
A Decision Framework for ETL Migration to Databricks
An ETL migration with hundreds of stored procedures, schedulers, scattered permissions, and a warehouse renewal deadline needs workload-by-workload decisions rather than a single rewrite strategy. The framework assigns work among Databricks SQL, Spark Declarative Pipelines (SDP), and PySpark or Spark SQL notebooks. SQL tasks suit single statements, Unity Catalog-governed stored procedures handle procedural logic, SDP manages dependencies, retries, quality constraints, and batch-plus-streaming, while notebooks cover complex logic, ML feature engineering, integrations, and large or tightly controlled Spark workloads. It recommends four phases—assessment, quick wins, modernization, and optimization—using profiling, side-by-side validation, and parallel runs before retiring legacy systems. Migration tools can automate 60–80% of initial conversion, but architecture choices remain essential: the goal is consolidating orchestration, metadata, lineage, permissions, and validation rather than reproducing technical debt.
How the English Office for Students leverages Databricks to enhance higher education standards and drive better student outcomes
The Office for Students, which regulates more than 400 higher education providers in England, modernised its data and analytics environment after a legacy platform could no longer handle growing volumes, varied sources, or emerging analytical demands. Its data spans millions of student records collected over 15 to 20 years, and a workflow processing about 300 million records took eight hours. Moving to Databricks consolidated structured, qualitative, and near-live data with analytics and AI workflows, while Unity Catalog added lineage, access controls, and security patterns for governed use. Genie Code reduced a student segmentation analysis from at least two weeks for two analysts to half a day, and a provider-registration triage proof of concept flags missing submissions earlier. The organisation frames AI as decision support rather than decision-making, keeping humans responsible for regulatory judgments.
From test bench to lakehouse: how AVL modernizes measurement data analytics with Impulse
AVL’s Lakehouse for Measurement Data addresses the scale, reproducibility, and governance limitations of desktop tools and isolated scripts used for automotive measurement analysis. Built on Databricks, the platform ingests ASAM MDF4 and other files into a Medallion Architecture, applies configurable DQX quality rules in a hierarchical Silver model, and uses Impulse to compile declarative Python TSAL expressions into distributed Spark execution. Engineers can select channels, create virtual signals with alias resolution, unit conversion, time alignment, and interpolation, define events, and compute duration- or distance-weighted aggregations in about 10 lines of Python. Impulse supports Gold-layer reporting, ad-hoc Spark DataFrames, and ML feature matrices, with Unity Catalog governance and Workflow orchestration. AVL reports reducing analysis time from days to minutes, processing many recordings per run, lowering infrastructure costs versus on-premises solutions, and enabling self-service, reproducible, standardized analysis.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.