Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
- 564 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
Real-Time Retail Intelligence: Building E-Commerce Recommendations with Lakebase and AI Search on Databricks
The post presents a reference architecture for building a production-grade e-commerce recommendation system on Databricks, based on a real-world implementation for a leading fashion e-commerce platform in Asia serving over 1 million monthly active users across 100,000+ SKUs. The system ingests approximately 1,000 events per second via Lakeflow Connect’s Zerobus into Unity Catalog Delta tables within a medallion architecture. Serving uses two paths: Path A for pre-computed batch recommendations via key-value lookup from Lakebase online tables, and Path B for real-time session-aware scoring using AI Search and a LightGBM model in a 3-stage funnel. The architecture unifies ingestion, feature management, model training, vector retrieval, and serving on a single governed platform, with cold-start handling for new users and products. Expected outcomes include low 2-digit millisecond responses, daily-fresh recommendations, and unified governance via Unity Catalog.
Lakebase Postgres branch-based restores for fast recovery at scale
Traditional Postgres point-in-time recovery is slow because compute and storage are coupled, requiring provisioning a new instance, restoring a snapshot from object storage to disk, and replaying WAL, leading to hours of downtime for large databases. In a survey of developers running 1TB+ production Postgres, 59% had a critical production failure and 30% were down for over three hours. Lakebase Postgres decouples compute and storage using safekeepers, a pageserver, and object storage, storing database history as an immutable timeline. A restore becomes a branch at a point in history, a metadata operation that does not copy data, making it near-instant regardless of database size. This eliminates data copies and WAL replay, so restoring a 100 TB database is as fast as restoring 10 GB, and agents can use restores as ordinary tool calls for versioning and undo.
Connecting customer context to measurable ROI with agentic marketing
Customer journeys have become non-linear while identity signals decay, and marketing value is shifting from reach frequency to a prediction economy that rewards anticipating needs. AI agents can recommend or take the next-best action for each customer within human-defined guardrails, with each action measured incrementally against what would have happened without it. Identity resolution must be continuous, connecting known and pseudonymous signals across channels, to provide reliable context for agent decisions. The context required includes customer behavior, business objectives, decision history, and control guardrails, with a single transaction carrying up to 50 metadata parameters. A practical starting point is focusing on one meaningful decision, such as reactivating lapsed customers, and expanding as trust and results build.
IP Functions are Generally Available, bringing high-performance network analytics to the Lakehouse
The announcement introduces the general availability of IP Functions, a feature that brings high-performance network analytics to the Lakehouse by enabling SQL-based processing of IP addresses. The post explains that IP data is warehousing data, as firewalls, load balancers, VPNs, CDN edges, DNS resolvers, Kubernetes clusters, and application servers emit streams of records keyed on IP addresses, generating tens of billions of events daily for large enterprises. These streams support valuable analytics such as threat detection, fraud investigation, and network governance. The post notes that network analytics used to be painful because IP addresses were deceptively hard to handle in SQL, with no regex support. The feature aims to simplify this by providing native IP functions for the Lakehouse.
Read Restrictions and Catalog Labels: Unifying governance across engines and catalogs
The Apache Iceberg community has adopted two additions to the Iceberg REST Catalog — read restrictions and catalog labels — each addressing distinct governance challenges. Read restrictions standardize delegated enforcement: a catalog evaluates policy for a principal and returns column-projection actions and row-filter expressions from a bounded vocabulary for a trusted engine to enforce. Catalog labels enable portable governance by allowing catalogs to exchange opaque key-value metadata at table and column level, so a consuming catalog applies native policies without per-user calls to the source. Read restrictions suit engine-to-catalog access where the source catalog trusts the engine and policies fit the standard vocabulary, whereas catalog labels fit federation between heterogeneous catalogs needing scalable governance and independent audit trails. These building blocks, along with centralized enforcement via cross-engine ABAC, advance the vision of unified governance across the open lakehouse.
How to scale agentic applications without creating AI sprawl
As agentic applications move from answering questions to taking actions, enterprises risk AI sprawl from duplicated integrations, inconsistent policies, and fragmented context. Databricks asserts scaling requires three pillars: choice across models and tools, context grounded in governed enterprise data and business semantics, and control through consistent permissions, policies, observability, and cost management. Its platform provides Agent Bricks for unified agent fleet building and governance, Omnigent as a common layer above agent harnesses, and Unity Gateway as a centralized control plane for models, agents, and tools. Context is shared via Unity Catalog for data and AI asset governance and Genie Ontology for common business concepts, letting teams reuse governed context across applications. Control uses Unity Gateway for access policies, budgets, and traces and Databricks Sandbox for isolated execution, with MLflow enabling evaluation and workflow inspection to maintain reliability as agents scale.
A practical guide to cost optimization with Lakebase Postgres
Lakebase uses separated storage and compute to reduce costs: branches share storage, autoscaling matches compute to demand, and scale to zero suspends compute after inactivity. Read replicas and high availability add compute without duplicating storage. For synced tables from Lakehouse, sync only the active working set via materialized views and choose the appropriate sync mode—snapshot for volatile data, triggered for incremental, continuous for real-time—and binpack tables into one pipeline to share compute. Right-size initial compute by focusing on the working set, and set PITR retention and snapshot schedules based on actual recovery needs. Snapshot storage is priced at $0.090/GB-month and PITR storage at $0.200/GB-month, both cheaper than regular branch storage. Compute and storage usage can be tracked through system.billing.usage, with separate SKUs for compute, branch storage, PITR, and snapshots.
Data visualization best practices: A practical guide to clear, honest, effective charts
The post examines why business intelligence (BI) dashboards lose user trust, even when data is accurate, focusing on how inconsistent metric definitions and opaque data sources cause users to doubt results. Richard Tomlinson argues that AI tools accelerate chart creation but also amplify trust debt if underlying data governance and semantics are weak, since AI can generate many inconsistent charts quickly. He recommends governing data and meaning, preserving provenance, and designing for inspection so visualizations invite investigation rather than presenting unexplained answers. Tomlinson stresses that dashboards should be self-explanatory, traceable, interactive, and context-aware, and that the critical metric for modern BI is ‘time to trust’ not just time to answer. Self-service fails when verifying an AI-generated answer takes longer than asking an analyst, so good visualization must enable inspection and challenge.
How Databricks rolls out frontier models to 12,000 employees on Day 1
Databricks provides 12,000 employees Day 1 access to new frontier models through a pipeline using Unity Gateway for governance, cost management, and observability. New models are immediately made available on an experimental basis with per-user budgets to control costs and mitigate risk from models that may not be on the efficiency frontier. Budgets include monthly maximum, daily runaway limit, quality frontier budget, and experimental budget, with configurations pushed via Unity Gateway CLI running on employee laptops. After three days, Databricks evaluates models using benchmark data, user-reported quality, and cost tracking via OpenTelemetry traces to decide whether to promote the model. Opus 5.5 was promoted as higher quality and lower cost than predecessors, while GPT-6 Sol was included in the smart router due to its cost advantage over GPT-5.6 Sol.
Your data, your storage, your rules: a 2026 guide to storing Unity Catalog managed tables
Unity Catalog managed tables let you control where your data physically resides while keeping it in cloud storage you own, such as S3, ADLS, or GCS, unlike platforms that store tables in provider-controlled storage. You set managed storage locations at the metastore, catalog, or schema level, with the most specific level winning, and can change where new tables land using ALTER CATALOG or ALTER SCHEMA commands. Managed tables use open formats including Iceberg and Delta, and external engines such as Apache Spark, Flink, and Trino can read and write through Iceberg REST Catalog and Unity Catalog open APIs. When converting an external table to managed, Databricks copies the data and transaction log into the managed storage location you have set. This approach enables a single source of truth across analytics and AI workloads while satisfying compliance, cost allocation, and data residency requirements.
Manufacturing data and AI: Connecting the product value chain
Manufacturing data is fragmented across systems such as MES, SCADA, QMS, ERP, and PLM, so answering a cross-stage question like whether a scrap spike stems from a supplier lot, machine setup, or process drift requires manual tickets, exports, and reconciliation. Databricks connects the product value chain by pooling or federating these sources using Lakehouse Federation and zero-copy Open Sharing, with Unity Catalog governing both mirrored and federated data under a single permission model and lineage. Lakeflow orchestrates the refinement of raw data into trusted Bronze, Silver, and Gold tables, while Genie and Agent Bricks enable natural-language queries grounded in governed business semantics. A quality engineer can thus ask about a defect’s history across plants and get answers that join MES, supplier, and QMS records without migrating every system. The result is that traceability becomes a query, transforming weeks of manual investigation into minutes of analysis across the entire value chain.
Lakebase Search: State-of-the-art full text and vector search for Postgres
Traditional OLTP systems like Postgres face challenges meeting the low-latency, high-accuracy search demands of AI agents, often requiring standalone search engines with ETL pipelines. Lakebase introduces lakebase_vector and lakebase_text, two extensions that bring scalable approximate nearest neighbor search and BM25 full-text search directly into Lakebase Postgres. lakebase_vector overcomes pgvector’s memory-bound HNSW index by using hierarchical IVF clustering and binary quantization, achieving 71-millisecond P99 latency at 97 percent recall on the 100-million-vector LAION benchmark. The extensions decouple storage from compute for pay-per-use scaling, enable fast parallel index builds, and support native hybrid search that fuses vector similarity with keyword relevance in a single SQL query. Early adopters like Conexiom reduced infrastructure costs by three times while achieving five times higher throughput, and the service is generally available on AWS and Azure.
How Databricks rolls out frontier models to 14,000 employees on Day 1
Databricks provides employees Day 1 access to new AI models using Unity Gateway for governance and cost management. The company uses a three-step pipeline: immediate experimental availability, per-user budget constraints, and data-driven promotion decisions. Unity Gateway CLI distributes model configurations to employee laptops, tagging new models as experimental. Per-user budgets include a monthly maximum, daily runaway limit, quality frontier budget, and experimental budget to control costs. Within three days of the Opus 5.5 and GPT-6 Sol launches, Databricks gathered benchmark, user feedback, and cost data to confirm they were on the efficiency frontier and promoted them, with Opus 5.5 showing a 29% cost reduction per session over Opus 4.8 and GPT-6 Sol showing a 48% reduction over GPT-5.6 Sol.
How to roll out Genie One: A step-by-step enterprise playbook
Genie One is an AI coworker that lets business users ask questions in plain language and receive cited answers grounded in governed data. The post outlines a phased rollout playbook: Phase 0 foundations include choosing one team, defining metric views for 10-20 numbers, creating Pages for concepts, and establishing governance with Unity Catalog. Phase 1 runs a 4-6 week pilot with 5-10 users, scoring Genie against 25-50 real questions and logging every incorrect answer to fix root causes, with exit criteria including accuracy holding steady and users reaching for Genie over their old workflow. Phase 2 expands to 2-3 adjacent domains with shared ontology definitions and a lightweight review workflow. Phase 3 aims for organization-wide adoption with federated ownership, MCP server governance, and embedding Genie in Slack, Teams, Excel, or Google Sheets, with a realistic timeline of 1-2 weeks for foundations to 1-2 quarters for full rollout.
The 24 most commonly misunderstood marketing data terms
Marketing and data engineering teams often interpret the same data terms differently, leading to campaign rework and misaligned expectations. The guide covers 24 commonly misunderstood terms organized around five practical questions about campaign execution, including where data comes from, who a customer record represents, and what makes an audience ready. For example, ‘customer’ might mean a person to marketing but an entity at a specific grain like account or household to engineering, while ‘real time’ can mean soon enough not to feel broken to marketing but sub-second streaming with always-on architecture to engineering. The post provides paired definitions for each term as conversation starters to surface assumptions and agree on meanings before work begins. It also recommends documenting business definitions in Unity Catalog, connecting agreed meanings to datasets or workflows, and using them for AI agents to produce consistent results.
From Data to Dialogue: How S&P Global Energy Made Its Structured Data Estate Conversational with Databricks Genie Agents and MCP
S&P Global Energy aimed to make its structured data estate—spanning LNG, Chemicals, Crude Oil, and other commodities—conversationally accessible via AI agents. The team evaluated approaches and chose Databricks Genie Agents, where subject matter experts curate one agent per dataset group without writing code. Each Genie Agent is automatically exposed as a managed MCP server, providing a governed, zero-deployment integration point. A FastMCP-based proxy composes multiple Genie MCP servers into composite endpoints for cross-domain questions. This architecture shortened time to market from months to days, with governance via Unity Catalog and an open MCP standard enabling both internal and external consumers.
Running open-Jev in SQL on Databricks
System One decision models such as Jev produce well-calibrated decisions from discrete options and are fast and cheap for large-scale applications, with potential use cases ranging from analyzing customer support transcripts to applying complex decisions at scale. Open-weight versions like SemIf-OpenJev are available, and Databricks enables serving them on governed data using serverless GPUs via AI Runtime, which provides on-demand GPU compute without managing infrastructure. The workflow is packaged in an importable notebook that downloads the model files, logs and registers the model using Express Deployments, and creates a GPU Model Serving endpoint automatically. The endpoint can be called from SQL via ai_query, which supports arbitrary custom model APIs, allowing structured request formats directly from SQL to return selected classifications and option probabilities as structured columns. The example classifies hotel reviews as good or bad, and users can import the notebook, select Serverless GPU, and run all to get started.
Deploy and manage coding agents at scale with the Unity Gateway CLI
Organizations face a challenge as new frontier models appear roughly every five days, making it difficult to balance flexibility with governance. Databricks introduces the Unity Gateway CLI, which lets admins centrally configure approved coding agents, models, MCP servers, skills, and spending policies. Developers launch any approved agent with a single ug command, and the CLI handles authentication and applies the published settings. Features include Smart Routing that selects cost-appropriate models per task, budget-aware defaults that switch to lower-cost options as spending thresholds are reached, and tracing to capture tool calls and skill invocations for waste analysis. In internal use, the system processed over 61 billion input tokens across roughly 360,000 requests, and tracing helped identify bugs that saved an estimated 1.2 million per year.
How I built agent-based security reviews on Databricks
Security reviews at Databricks relied on automation but still consumed expert time on predictable work, leaving less room for genuinely novel, high-risk architectures. The author built an agent-based layer on Databricks using Unity Catalog for governed tables containing standards, request data, and decisions; Databricks-hosted foundation models (Claude Haiku, Sonnet, Opus) for classification and reasoning; and Lakeflow Jobs for orchestrating notebook-based workflows on serverless compute. A conversational intake app built with Databricks Apps turns plain-language descriptions into structured requests and provides preliminary risk indications before a formal request is created. Seven focused agents each have a bounded responsibility—intake, risk assessment, requirements, specialized reviews, validation, workflow, and learning—ensuring behavior remains inspectable and testable. Routine requests that once waited days in the queue can now be completed in minutes, and security engineers spend more time on designs requiring genuine expertise.
How Concurrence governs clinical AI at a trillion-token scale with Unity Gateway
Concurrence processes 100.8 billion input tokens and 11.2 million LLM calls every 30 days, at an annualized rate of 1.2 trillion tokens. To govern these workflows, the company consolidates on Databricks with Lakebase for operational state, Unity Catalog for governance, and Unity Gateway for centralized AI access. Patient data is recorded as immutable events in Delta tables, from which a current patient state—called the world model—is computed while preserving provenance. All agents are tested against simulated patients before deployment; simulation traffic is approximately seven times production traffic, using immutable event history to replay states. Concurrence routes all coding-agent traffic through Unity Gateway, attributing each request to the engineer, and enforces compliance-gated model routing where HIPAA requirements are met first.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.