Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
- 574 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
How Hamlet uses Databricks Lakebase Postgres to test changes against production data
Hamlet, an AI platform indexing meetings from over 7,200 local government bodies, uses Databricks Lakebase Postgres to give each pull request its own branch of production data. This approach lets engineers and AI coding agents like Claude and Codex test data pipeline changes against the exact data that caused a bug without any risk to production. Previously, the small team spent 15 to 30 minutes per pull request staging data and up to several hours weekly cloning data locally, a significant productivity drain. With Lakebase data branching, each preview environment connects to an isolated copy-on-write database where only branch changes consume separate storage, and branches can have an expiration time-to-live for automatic cleanup. A person still approves every merge into production, combining developer speed with human oversight for data pipeline development.
Introducing Agent Memory on Databricks
Databricks announces Agent Memory on its Agent Bricks platform, now in Beta. Agent Memory improves agent quality by storing critical facts, preferences, and workflows that agents should remember, reducing both cost and latency. It is built on the MemCurrent architecture from Databricks Research, which combines context logging plugins, on-demand tool access, and a background dreaming pipeline to consolidate memories, avoiding the cost of replaying full histories or building bespoke memory systems, all served on Lakebase for low latency. In internal benchmarks, agents with memory tripled pass rates on real enterprise tasks while using 31% fewer tokens compared to stateless agents. Agent Memory joins other Agent Bricks capabilities including hosting, Sandbox, Unity Gateway, pre-built MCP servers, and tracing and evaluation, providing a complete platform for building production agents at scale.
MemCurrent: Designing Persistent Memory for Production Agents
Enterprise agents operating at scale must balance answer quality against cost and latency, a trade-off that memory can address if designed carefully. MemCurrent is a persistent memory architecture that stores both episodic memory (raw session traces) and derived memory (distilled learnings in a structured file system indexed by memory.md). It accesses memory through three patterns: plugins for automatic context logging and injection at session start, tools the agent calls explicitly during a session, and an offline dreaming pipeline that consolidates memory asynchronously. On an internal benchmark built from 114 real-world test cases across 29 users, MemCurrent triples the agent’s pass rate compared to no memory while reducing token consumption by 31% and latency by 26%. It also matches full-context accuracy on open-source benchmarks at roughly 12x fewer input tokens, placing it on the cost–accuracy Pareto frontier.
CustomerLake enters Beta: Leading enterprises going live with the Agentic CDP
Databricks announced CustomerLake’s Campaign Agents moving to Beta and Profile Agents entering Private Preview, expanding access to the Agentic CDP. The platform embeds customer data platform capabilities directly into Databricks, allowing marketers to build audiences using governed data without recreating context in separate tools. Early adopters including HP, Getnet by Santander, AB InBev, and Wine.com reported significant improvements: HP achieved audience building in hours versus months and a potential 40% reduction in total cost of ownership. Campaign Agents uses Databricks’ Genie AI to enable natural language audience creation, while Profile Agents unifies scattered customer data into Customer 360 profiles using deterministic, probabilistic, and LLM-based matching. Philadelphia Union cut audience turnaround time by 90%, and Databricks’ own Data + AI Summit campaign using CustomerLake delivered a 10% conversion lift.
How HP Is Building an AI-Ready CDP with Databricks CustomerLake
HP is building an AI-ready go-to-market data foundation on Databricks, unifying customer data and third-party signals in a governed Marketing Lakehouse after consolidating hundreds of data repositories. The company estimates that consolidating CDP capabilities with CustomerLake could reduce total cost of ownership by up to 40% compared to its former Composable CDP approach. HP expanded available marketing data with 90% net-new signals across media, brand, intent, and technographic sources. In the first session of using CustomerLake, HP configured data models, published attributes, and became ready to build audiences in two hours, a process that previously could take two to three months. HP is also bringing identity resolution into its Databricks environment to create a governed foundation for measurement, audience activation, and model improvement, and using AI/BI Genie to bridge the gap between technical readiness and business usability.
How to get the most out of running dbt on Databricks
Tens of thousands of organizations use dbt, and where it runs directly affects pipeline cost, speed, and operational complexity. Running dbt on Databricks provides an open, unified platform, yet many teams underutilize its capabilities. The post details practices to reduce costs, such as applying appropriate incremental strategies and replacing full recomputes with materialized views, which delivered a 17x speedup in a benchmark. It also covers automating data layout with Liquid Clustering and Predictive Optimization, and attributing every dollar to specific models via query tags. For operational simplicity, it recommends bringing ingestion into dbt with streaming tables, standardizing metrics through Unity Catalog Semantics, and orchestrating pipelines natively with Lakeflow Jobs.
What makes a catalog open? Unity Catalog Advances the Open Lakehouse
An open catalog requires storage you own, full features on that storage, and the ability to move tables without copying data. Unity Catalog in Databricks’ Open Lakehouse meets all three by keeping managed tables in customer-owned cloud storage. It supports REGISTER and UNREGISTER from the Apache Iceberg REST catalog spec, enabling tables to enter or leave without data movement. Databricks contributed UNREGISTER to the Iceberg REST catalog spec, making this an open standard. On other platforms, customer-owned storage often lacks features like fail-safe recovery or automatic compaction, pushing users toward provider-controlled storage.
Your agents are making decisions on stale data
AI agents cannot compensate for stale data like humans, turning sync lags into machine-speed errors such as approving fraud. Traditional operational data stores (ODS) rely on CDC pipelines and separate copies, which add latency, break on schema changes, and remain ungoverned. LTAP (Lakehouse Transactional Analytical Processing) unifies storage so operational writes land directly in a single governed foundation, eliminating replication entirely. It uses two independent engines—a row-oriented engine for operational reads and a columnar engine for analytical scans—both accessing the same copy without duplication. This removes the sync step, reduces infrastructure complexity, and ensures agents always query current data without degrading transactional performance.
The best AI database solutions for 2026
AI databases extend traditional databases with vector search, full-text search, and hybrid search to support production AI applications like RAG and AI agents. This guide evaluates 10 AI database solutions across seven criteria: search capabilities, serverless scaling, elasticity, Postgres compatibility, high concurrency, low latency, unified governance, and analytics integration. Solutions are grouped into purpose-built vector databases, such as Pinecone, Milvus, Weaviate, and Qdrant, and relational databases with AI search, like PostgreSQL with pgvector and Databricks Lakebase. The choice depends on whether retrieval is the primary requirement or if AI search must coexist with transactional and analytical workloads. Dedicated vector databases suit large-scale retrieval, pgvector fits existing Postgres teams, and Lakebase unifies AI search, transactions, and analytics on the Databricks platform.
Databricks Is Doubling Down On New York City As A Premier AI Talent Hub
Databricks is doubling its New York City office space at 5 Bryant Park to approximately 120,000 square feet across five floors, with plans to grow headcount significantly. The expansion includes a dedicated Engineering and Research floor for focused R&D, an Executive Briefing Center, and room for future growth. The NYC hub is home to AI research teams led by the Chief AI Scientist, infrastructure teams building Unity Gateway for cost and model governance, and applications teams developing CustomerLake for enterprise marketing. Databricks cites industry proximity to Fortune 500 companies in finance, media, and healthcare, access to talent from Columbia, NYU, and Princeton, and New York’s time zone connectivity to Europe and India as strategic advantages. The company is investing in New York as a premier AI talent hub where engineers build products alongside enterprises rather than in isolation.
Transactions are Generally Available, advancing warehousing on the Lakehouse
Databricks made multi-statement transactions generally available on Delta Lake and Apache Iceberg tables, providing atomic guarantees for complex, multi-step workflows directly on the lakehouse. Previously, organizations had to rely on separate warehouse systems for these transactional guarantees, adding expensive data movement, duplicate copies, fragmented governance, and custom consistency workarounds. Row-level concurrency enables concurrent transactions to write to the same table as long as they do not update the same row, delivering significantly stronger performance than traditional cloud and legacy warehouses. The transactions are composable with stored procedures, temporary tables, and streaming tables, and they respect Unity Catalog attribute-based access controls including row-level security and column masking on both reads and writes. Fivetran used transactions to remove consistency-repair code for its slowly changing dimension updates using SCD Type 2, and Accolade processes roughly 1,700 transactions daily for claims and liabilities on the governed lakehouse.
Stripe Data Pipeline now syncs real-time Stripe data to Lakebase Postgres
Stripe Data Pipeline now supports real-time sync to Postgres, with Databricks Lakebase as a supported destination. Stripe streams real-time API data into Lakebase and manages the sync end to end, eliminating the need for custom integrations with APIs or webhooks that require handlers, retry logic, and reconciliation processes for missed or out-of-order events. One Stripe account can feed two Databricks destinations: Unity Catalog for analytics and AI workloads, and Lakebase for applications. With real-time sync, payment, customer, subscription, refund, and payout data updates arrive in Lakebase within seconds, enabling applications to control product and feature access based on subscription status, keep billing and account details up to date without calling the Stripe API on every page load, and provide real-time financial visibility with dashboards showing daily sales volume, refunds, and payouts as they happen. The feature is available in public preview and free to get started.
Closing the Binary Blind Spot: Domain-Knowledge Injection for LLM Malware Triage in npm Packages
Supply chain attacks on the npm registry are growing, and packages that bundle a binary payload are rare but disproportionately hard for LLMs to triage. The authors built a scanner that combines static analysis with LLM reasoning and introduced capability profiles, which encode malicious techniques as weighted conditions injected into the prompt before the model decides. They tested nine models across seven providers on five adversarial payload types, including credential stealers and reverse shells, at varying obfuscation levels. Without profiles, the budget-tier models failed to reach a malicious verdict on 50 percent of tests, with failures following distinct model-specific patterns rather than random error. With profiles enabled, the failure rate dropped to 4 percent, and 91 percent of prior failures recovered to a detection.
Geospatial queries are up to 38X faster with Lakehouse//RT
Databricks announced that Spatial SQL and spatial joins now run on Lakehouse//RT, delivering up to 38X faster performance under high concurrency compared to traditional cloud data warehouses. The platform supports GEOMETRY and GEOGRAPHY data types, over 100 spatial functions, and high-performance spatial joins, all within the open lakehouse using Apache Spark and Parquet files. Organizations in insurance, agriculture, energy, telecom, retail, and logistics can consolidate geospatial workloads onto a single governed platform, eliminating separate stacks like PostGIS. Benchmarks using SpatialBench show Lakehouse//RT is 3X faster on average and up to 20X faster for single-query concurrency, with no code changes required. The service is available in Beta, allowing users to run existing spatial queries directly on Delta and Iceberg tables without copying data.
Reducing the noise: How Atlassian built an OCSF security lakehouse on Databricks
Atlassian faced a security data pipeline ingesting 44 billion events daily from 20 years of growth and acquisitions, each with proprietary logging formats. To unify this fragmented telemetry, the team standardized on the Open Cybersecurity Schema Framework (OCSF) and built a security lakehouse on Databricks. They embedded the entire OCSF schema into Go structs (TypeGo), guaranteeing compile-time correctness and reducing AI hallucinations during automated transformation. Table optimization with partitioning and Z-ORDER cut query times by 95%, and Graviton clusters with Photon reduced cost and runtime by two-thirds. The result was a self-service platform where SOC analysts can onboard new data sources in days, and detection teams moved from one-to-one alerts to signal-based clustering using standardized OCSF values.
How our AI Research team uses AI Runtime
Databricks’ AI Research team uses AI Runtime (AIR) to manage hundreds of parallel research workstreams for products like Genie, Smart Routing, and AI Functions, training models from a few billion to nearly a trillion parameters. AIR performs preflight checks on GPU nodes to ensure health and communication before training, and monitors node health during execution to detect issues like NCCL failures or thermal throttling. The AIR CLI and custom Docker containers enable fast experimentation by launching jobs in seconds without rebuilding environments, supporting both supervised fine-tuning and reinforcement learning. MLflow integration automatically tracks experiments, while Unity Catalog FUSE mounting gives governed access to research data across workspaces without changing training code. The team has scaled to agent-led training, where AI agents propose changes, launch trials, and evaluate results, increasing training job volume by 8x across AI Functions research workloads.
How FactSet cut model training and deployment from days to hours with Databricks AI Runtime
FactSet’s Text-to-Formula (TTF) system converts natural-language investment queries from professionals into correct FactSet Query Language (FQL) formulas. Initially built on retrieval-augmented generation (RAG), the approach hit a quality ceiling as queries became more specialized and formulas more complex. FactSet adopted fine-tuned open models within a Compound AI architecture, reducing end-to-end latency by about 60 percent and improving quality for complex formulas. Fine-tuning runs on Databricks AI Runtime with serverless GPU infrastructure, where engineers choose compute based on prompt length and output needs, and the pipeline integrates with MLflow, the Lakehouse, and Lakeflow Jobs for a unified workflow. This setup cut model setup, training, and deployment from days to hours, enabling the team to retrain every two to three weeks as new data and models become available.
Reducing Qwen training time by 6x on AI Runtime
The post describes systematic performance optimization to reduce fine-tuning time for the Qwen 3.8 27B model on AI Runtime from 7 hours 44 minutes to 1 hour 17 minutes, a 6x improvement. The baseline showed 98% GPU utilization, but torch traces revealed the CPU was stalling the GPU with too many small operations. Installing manually fused kernels such as flash-linear-attention and causal-conv1d reduced the number of GPU operations by 15x and improved training speed by 3.7x. Further gains came from switching from 2D padded batches to 1D packed tensors with correct position_ids, which reduced the number of batches by 36% and eliminated dynamic shape recompilation. The post emphasizes measuring what matters beyond GPU utilization and leveraging pre-built community kernels and efficient batching.
Unlock powerful insights and forecasts with Predictive AI Functions
Data teams frequently field questions about future sales, metric movements, and anomalous data, but answering them typically requires data science teams to build and manage models, creating a bottleneck. Predictive AI Functions, now in Beta, remove that bottleneck by providing SQL-native functions for forecasting, driver analysis, and anomaly detection, backed by state-of-the-art ML models maintained centrally by Databricks. The launch includes ai_forecast for time series projection, ai_top_drivers for segment driver analysis, ai_detect_anomalies for outlier detection, and ai_predict_value and ai_predict_class for regression and classification. These functions can be called directly in SQL, invoked through Genie with plain-language questions, or embedded in dashboards, delivering consistent, high-quality answers without infrastructure management. Benchmarking on fev-bench shows ai_forecast outperforms forecasting functions from leading cloud data warehouses, achieving industry-leading accuracy out of the box with no tuning required.
OLTP vs. OLAP: Choosing Between Transactional and Analytical Databases
OLTP handles high-volume, low-latency transactions on current operational data for online banking, e-commerce, and inventory, while OLAP handles complex analytical queries on large historical datasets for financial analysis and business intelligence. OLTP databases ensure data integrity through ACID compliance and use normalized row-oriented tables; OLAP systems use denormalized dimensional models and columnar storage for efficient scans and aggregations. Data pipelines such as ETL or CDC tools like Debezium connect OLTP and OLAP systems, but latency and pipeline complexity rise with analytical freshness requirements. For mixed workloads, HTAP or lakehouse architectures can reduce synchronization overhead and analytical lag. The choice between OLTP and OLAP depends on the workload’s read-write pattern, latency needs, and data volume, with TPC-C and TPC-H as reference benchmarks.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.