Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
- 574 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
Southern Company’s SCOUT: Completing the Storm Intelligence Story
Southern Company’s SCOUT is a real-time storm operations application designed to make outage intelligence accessible beyond control-center desks during active events. Built on the Databricks lakehouse, it combines Delta Lake storage, Databricks Lakehouse warehouses, Unity Catalog governance, and collaborative notebooks, querying a dedicated warehouse every minute for current operational data. The mobile-friendly application unifies outage, customer, GIS, weather, terrain, crew, and reliability information, helping teams assess restoration times, assign appropriately equipped crews, coordinate mutual assistance, and communicate with customers. SCOUT is used across Southern Company’s operating businesses and had 1,139 adopted employees, including more than 250 users on a peak storm day; the source also describes exploration of AI crew recommendations and autonomous-drone integration, not a completed deployment.
Expanding Genie Agents: Deep analysis, file reasoning, and more
Genie Agents updates expand expert-curated business agents beyond quick factual lookups to deeper analysis across structured and unstructured data. Agent mode can create and refine research plans, issue iterative queries, and return reports with findings, visualizations, and citations; Agent mode APIs extend this workflow to custom applications, chatbots, scheduled reports, and internal tools through streaming Server-Sent Events. Agents can analyze PDFs, documents, slide decks, and images in Unity Catalog volumes alongside governed tables, with permissions and content search indexing applied to retrieval. Genie Code helps authors create baseline agents, diagnose conversation or benchmark failures, and identify knowledge gaps from user feedback, while authors retain review control over proposed changes. Shared chats keep follow-up messages and edited visualizations visible to authorized users, supporting collaboration and agent-quality review.
Announcing the Databricks Big Book of AgentOps
The Big Book of AgentOps presents AgentOps as an operating discipline for building, deploying and improving production AI agents, combining architecture, evaluation, observability, governance, security and cost management. It addresses the operational risks of tool calls, enterprise-data retrieval, API actions, retries and multi-step execution, where quality, permissions and spend can become difficult to control. The six-chapter book covers four agent architectures, four deployment patterns, a seven-phase lifecycle, DevOps practices, operationalization planning and stakeholder alignment, with Unity Catalog, Unity Gateway and MLflow central to its Databricks-based deployment guidance. Its practices include building evaluation datasets from real traces, calibrating automated judges with subject-matter-expert feedback, designing tracing and access controls into systems, and using RACI matrices to clarify ownership. The announcement cites customer examples including FactSet’s 44% accuracy improvement, DXC Technology’s 30% total-cost-of-ownership reduction, and ICE’s 77% syntactic accuracy and 96% execution matches.
How the FDA is building a secure, AI-ready data foundation on Databricks for Government
The FDA built HALO (Harmonized AI and Lifecycle Operations for Data) as a secure, governed, AI-ready enterprise data platform for modernizing siloed systems without interrupting regulatory work. Its move to Databricks on AWS GovCloud, following FedRAMP High authorization sponsorship, added Unity Catalog as a governance layer across a multi-tenant architecture, while Terraform-based security patterns, PrivateLink, customer-managed keys, and the compliance security profile support regulated workloads. The agency migrated more than 5,000 users and 8,000 jobs and pipelines with zero downtime, refactoring over 1,000 pipelines and 4,000 notebooks. After onboarding eight centers and 30 programs, FDA reported query responses improving over 30%, compute costs falling over 20%, and provisioning and sharing time dropping over 75%. HALO also supports responsible AI use cases such as MARS, with humans retaining decision authority.
How we eliminated $1 million a year of wasted AI agent spend in one hour
Databricks engineers investigated whether silent failures in MCP tool calls were driving hidden AI-agent costs as usage increased. Unity Gateway emitted OpenTelemetry traces for every invocation, recording tool names, arguments, errors, token counts, latency, and session IDs in a table that Genie One could query with plain-English questions. In a 24-hour sample, the analysis identified seven Jira and Google Drive/Docs bugs responsible for an estimated $499,000 annually in wasted tokens and 12,023 hours of agent wait time, or $1.2 million in lost productivity. The most costly patterns included under-specified interfaces, incompatible argument shapes, and cryptic or misleading errors that triggered retries and recovery loops. Engineers fixed the issues with a coding agent, completing the find-quantify-fix cycle in about an hour; the post recommends designing tools to tolerate reasonable LLM inputs and continuously tracing failures.
Collaboration makes us all stronger
An external researcher, Mehmet Ince, reported a memory-safety flaw in PostGIS’s address_standardizer extension, where caller-controlled grammar data indexed a fixed-size array without bounds checking. Because the extension was available to ordinary tenant roles on managed Postgres services, Databricks treated the issue as a platform exposure rather than a third-party problem, while noting that its microVM architecture meant the exploit did not produce cross-customer impact on Lakebase Postgres and Neon. Detection alerted the Neon team during testing, enabling rapid validation and downstream patching while a durable fix was developed and deployed to protect tenants without customer action. The upstream fix initially missed cases; Mehmet supplied the remaining pieces, and the complete correction was returned to PostGIS, with his bounty donated and matched for the project.
Operationalizing Genie Ontology in Your Data Stack
Genie Ontology is presented as a way to give enterprise AI agents shared business context beyond a semantic model, including definitions, relationships, business rules, authoritative sources, and permissions. It combines Unity Catalog Semantics—Metric Views, Pages, and Domains—with context inferred from governed tables, queries, dashboards, notebooks, and other supported assets. The guidance recommends six progressive layers, beginning with clean gold data and resolved golden records, then metadata, semantic modeling, enterprise context, governance, and evaluation. Critical implementation details include declaring informational primary and foreign keys, defining canonical measures in Metric Views, adding synonyms and example queries, and using permissions plus human-reviewed automation. Rather than waiting for complete coverage, it advises starting with one high-value domain and metric, then using feedback, telemetry, benchmarks, and drift reviews to strengthen trust over time.
The new Brickbuilder Partner Network tiers for ISVs and Data Providers are here
The Databricks Partner Network has launched Bronze, Silver, Gold, and Platinum tiers for ISVs and data providers, with placement based on customer impact, technical excellence, strategic alignment, readiness and enablement, and joint go-to-market activity. Technical Excellence is evaluated through the Partner Well-Architected Framework (PWAF), which provides prescriptive, AI-ready guidance for secure, reliable integrations and serves as the standard for validated architecture. For customers, the tiers signal that a partner solution has met a high bar for architecture, security, compliance, and joint customer success, reducing vetting effort. For partners, each tier defines requirements and a roadmap tied to greater visibility, co-selling, and go-to-market support, while Databricks plans to refine the program and expand benefits over the coming months.
How Discovery Bank delivers hyper-personalized banking at scale: behavioral AI, governed data, and real-time decisioning
Discovery Bank’s hyper-personalized banking model addresses how to make interactions relevant to individual behavior while meeting financial-services demands for scale, speed, security, and governance. The bank combines demographic, transactional, digital-engagement, savings, borrowing, credit-risk, rewards, and lifestyle-related data on the Databricks Data and AI Platform, producing governed reusable features, indicators, scores, forecasts, and recommendations. Behavioral models and a reusable next-best-action decisioning layer support personalization, fraud detection, servicing, and banker assistance; TRUST alerts assess transactions against client-specific norms and can escalate from explanation to account locking. A four-layer architecture adds control services and specialized generative AI and agents to trusted data and analytical foundations, with governance remaining in the loop. Reported results include a 40% uplift in client engagement impact, 20x faster pipeline development and data processing, 5x faster data-product creation, more than 300 models built per day, and return on investment above 500%.
Autoscaling Lakebase Postgres
Lakebase Postgres addresses the waste and uncertainty of choosing a fixed database instance size before workload demand is known. Its decoupled architecture separates stateless compute from durable storage, allowing compute to start, stop, move, or resize without moving the database. The autoscaling algorithm combines CPU load, memory pressure, and a time-aware estimate of the Postgres working set, using modified HyperLogLog registers that store timestamps and estimate windows from one to 60 minutes. Scaling occurs live through autoscaler-agent, vm-monitor, Kubernetes scheduling, and NeonVM, with checks preventing unsafe downscales and scheduler approval for upscales. The system scales in both directions without dropping connections and operates across 100-millisecond, five-second, and 20-second timescales, including more than 32,000 resizes per month.
What is an AI Copilot?
An AI copilot is an assistant embedded in a software application that uses workflow context to offer suggestions, generated content, or approved actions while keeping the human in control. Unlike a standalone chatbot, it combines an LLM with contextual grounding, retrieval-augmented generation (RAG), access to enterprise data, and action layers that can call APIs, execute code, trigger workflows, or update records. The explainer surveys code, productivity, data and analytics, customer-facing, and domain-specific copilots, including uses such as SQL generation, dashboard creation, email drafting, and case summarization. Reported benefits include faster routine work, broader data access, more consistent outputs, and less context switching; a cited GitHub study found Copilot users completed a controlled coding task 55% faster. It also stresses hallucinations, privacy, bias, overreliance, integration complexity, cost, and the requirement for human review, governance, and accountability.
Beyond answers: New Genie One features to turn insights into action
Genie One’s latest updates target the manual work that follows an AI-generated answer, including accuracy checks, report formatting, sharing, and repeated workflows. The release adds a native macOS desktop app in beta with a global launcher, persistent conversations across web and desktop, and access to documents, skills, scheduled tasks, Genie Agents, and Databricks Apps. Genie One can create editable documents with live visualizations, comments, version history, sharing, PDF export, and external-document editing through MCP writes. Users can also turn conversations into reusable, shareable Genie Agents or distribute read-only chat links containing questions, answers, visualizations, and context. Additional capabilities include ontology snippets, private uploads of supported files, and governed MCP actions, with Unity Catalog, source authentication, and Unity Gateway governing access across assets and connected tools.
How Trackunit turns construction data into decisions with AI
Construction data is fragmented across systems, organizations, equipment types, and formats, limiting the operational context available for decision-making. Trackunit addresses this data-intelligence problem with IrisX, an operating data platform built on Databricks that connects equipment, machine, operator, site, and operational data. Its three-part model—connect, distill, and amplify—combines data engineering, analytics, and AI on an open platform, then delivers governed, construction-specific intelligence through applications, workflows, and AI interfaces. The platform supports natural-language analysis of equipment signals and contextual factors such as maintenance history, operating conditions, location, contracts, project needs, and asset availability. Examples described include battery-management guidance for OEMs, missed-invoice detection for rental companies, and asset redeployment for contractors, with reported value of approximately $3 million and $2 million in two examples.
How Indra unified EV charging data on Databricks
Indra Renewable Technologies consolidated a fragmented EV-charging data estate spanning Cosmos DB, Synapse, Azure Data Lake Storage, Azure Functions, Power BI, and other services onto Databricks. The migration began with a parallel Synapse prototype and expanded to Delta tables, a consolidated fleet pipeline, and greater use of Databricks for operational workloads. A medallion architecture stores raw fleet data in bronze, transforms it with PySpark in silver, and publishes gold tables for reporting and client delivery; one pipeline now serves three clients and improved performance by 60% to 70%. Unity Catalog, serverless SQL warehouses, AI/BI dashboards, and Genie Agents support governed self-service analytics, while reported results include 80% to 90% business cost savings, 90% storage savings, and query latency falling from 24.4 seconds to 3.5 seconds.
Fast, fault-tolerant PyTorch training on AI Runtime
At scale, training jobs face frequent GPU interruptions, while remote-storage input pipelines can leave accelerators idle. Distributed Checkpoint (DCP) has each rank write a shard plus a .metadata file, allowing parallel saves and reloads onto a different number of GPUs; DCP is presented as useful for DDP as well as sharded training. Asynchronous saves stage data locally and upload in the background; cited measurements report 1.8x savings for a 2.8B-parameter DDP model and 58x for a 20B-parameter FSPD model on 32 H100s, though the comparison excludes torch.save network storage time. UCVolumeDataset and the accompanying DataLoader cache files on local NVMe and prefetch disjoint rank/worker slices while computation runs. Correct recovery also requires checkpointing data position plus seeds and RNG state so resumed sampling remains consistent.
One Write Path, Multiple Entries: How Rippling uses Unity Catalog-managed Apache Iceberg™
Rippling describes an architecture in which Databricks writes some tables while Trino, Snowflake, and other lakehouse engines read them, making catalog ownership a design decision rather than a metadata detail. Its rule is that the catalog follows the writer: AWS-native workloads use Glue, Databricks workloads use Unity Catalog, and Snowflake-produced shared data uses Horizon Catalog with Snowflake-managed Iceberg tables. For selected Databricks outputs, Unity Catalog-managed Apache Iceberg stores table data and metadata in Rippling-owned S3, exposes access through the Iceberg REST Catalog API or federation, and uses credential vending for scoped storage access. This lets downstream engines read Parquet directly without duplicate materialization or producer-side query execution. Databricks also provides Predictive Optimization for maintenance, while the trade-off is continued Unity Catalog dependence for metadata, permissions, lineage, and lifecycle.
Building for the AI Era: Lakebase, Streaming, and Lakehouse Innovations at VLDB 2026
Databricks previews its VLDB 2026 keynote, four accepted papers, a demo, and a sponsor talk covering database architecture, streaming, and Lakehouse optimization. Reynold Xin will introduce Lakebase, which separates serverless PostgreSQL compute from storage, and LTAP, a transactional-analytical model intended for AI-agent workloads; Lakebase stores data and write-ahead logs in cloud object storage and supports sub-second cold starts and copy-on-write branching. The papers describe a decade of Spark Structured Streaming evolution, including microbatch pipelining that improved throughput by up to 3x, stateful APIs, and fine-grained access control, alongside AutoLiquid’s CLUSTER BY AUTO and Ultron’s history-based query optimization. AutoLiquid outperformed customer-selected clustering keys on over 95% of evaluated workloads, while Ultron improved median join latency by 25% in production workloads. The program also includes an Enzyme demonstration of incremental materialized-view maintenance and a sponsor presentation on LakehouseRT and the Reyden engine.
What QSR reports miss about the decisions matter the most
Traditional QSR reporting can show that a limited-time offer (LTO) missed systemwide expectations, but not whether the cause was weak demand, lower franchise participation, ingredient shortages, inconsistent execution, slower service, or menu cannibalization. The post presents Lovelytics’ QSR Executive Performance Control Tower, an AI blueprint/solution built on Databricks that connects corporate performance, supply chain and operations, customer and digital, and financial signals. It is designed to distinguish demand from operational and participation issues, link changes to likely drivers and financial impact, and give corporate and franchise leaders a daily briefing on what changed, why it matters, and where to focus. The stated advantage is faster, more actionable decision-making based on a governed, reusable intelligence layer rather than disconnected reports.
Enhancing Agent Retrieval with Structured Chart Extraction
Enterprise retrieval agents often struggle with questions that require reading values, labels, or patterns in charts, because text-only indexes and captions omit fine-grained visual data. The post evaluates structured chart extraction as a way to improve retrieval and answering, comparing caption-only figures with figures enriched by chart JSON from ai_parse_document. Its pipeline uses ai_prep_search to create retrieval chunks, a 300-million-parameter text embedding model for indexing, and Genie for retrieval and answer generation. Evaluation covers 310 chart-heavy questions from ViDoRe V3 and 114 visually grounded questions in the synthetic Chart-RAG dataset, with answer correctness, Hit Rate@10, and nDCG@10 measured across three runs. Chart JSON improves retrieval and answer quality; adding the top three retrieved images raises correctness further, reaching 75.9% on ViDoRe V3 and 75.1% on Chart-RAG, exceeding four multimodal embedding baselines.
Vertical Advantage: Transforming Industries with Lakebase and Agentic AI
The post presents partner-built, industry-specific solutions on Databricks Lakebase for financial services, manufacturing and energy, retail and consumer sectors, travel and hospitality, healthcare and life sciences, communications and media, and the public sector. Lakebase is described as a serverless Postgres transactional database on the Databricks Data and AI Platform, combining operational and analytical workloads under unified governance and providing operational serving, agent memory, audit trails, and transactional runtime capabilities. Examples include regulation-change monitoring, AI-native claims operations, KYC and personalization accelerators, near-real-time fraud detection, telecom customer-lifecycle decisioning, and public-sector emergency Situation Reports, using components such as Unity Catalog, Lakeflow, Genie, Mosaic AI agents, and Agent Bricks. The stated outcome is production-ready acceleration from insight to action, including faster compliance response, claims adjudication in minutes, low-latency decisions, and reduced manual effort, with partners positioned to help scope pilots.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.