Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
- 567 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
Building an A/B testing analysis framework for mobile gaming on Databricks
HARDlight needed a more consistent way to analyse mobile game A/B tests as experimentation expanded, because manual workflows and differing statistical approaches slowed decisions and weakened trust. It built a Databricks-native framework that standardises experiment definitions, telemetry, modelling, and governed outputs, with notebooks materialising effect estimates, uncertainties, and segment-level impacts into a unified analytics model. Databricks AI/BI then presents daily refreshed, progressively detailed dashboards, beginning with an LLM-generated summary and extending through KPIs, diagnostics, revenue, engagement, monetisation, and segment analysis. At experiment completion, a frozen snapshot preserves outcomes, decisions, and recommended actions. The framework reduced manual effort by more than eight hours weekly, saves roughly one day per experiment, and enabled a targeted two-times increase in monthly testing capacity without added headcount.
Tevogen Bio’s Journey to Streamlining Life
Tevogen Bio faced drug development costing upwards of $3 billion and requiring 10–12 years; selecting targets for its single-HLA-restricted SARS-COV2 product took 18–24 months of wet-lab work. Tevogen.AI partnered with Microsoft and Databricks to build and curate a multi-terabyte protein-sequence library and parallelize processing for machine-learning training. Databricks Data + AI Platform, Medallion Architecture, Unity Catalog, and distributed computing cut processing from 50 days to 24 hours under governed, fine-grained access controls. The dataset reached 24 million proteins, 16 billion datapoints, ~700 million unique peptides, and ~37 million cross-matched expert articles. An MLOps framework automated training, inference, monitoring, and retention; the alpha PredicTcell model used XGBoost and ESM models and reported 93–97% recall and 38–43% accuracy, while beta PredicTcell and alpha AdapTcell work began.
What is a Cloud
A cloud-based database management system stores, manages, and queries data on third-party cloud infrastructure instead of local or on-premises servers. It retains capabilities such as query optimization, security, transactions, and support for relational SQL, NoSQL, data warehouse, and in-memory workloads, while operating across distributed nodes. Organizations can self-manage database software on cloud virtual machines or use DBaaS, where the provider manages the software and infrastructure; connections typically use SQL, APIs, or web consoles. Virtualized resources can scale, and providers may handle patching, backups, maintenance, and failover, while replication across availability zones supports recovery and availability. The comparison with on-premises systems covers ownership, scalability, cost models, maintenance, accessibility, control, compliance, total cost, and emerging capabilities including AI integration, vector search, autonomous operation, multicloud, hybrid, and serverless architectures.
Fizz accelerates ecommerce analytics with Databricks SQL
Fizz.hu, an ecommerce marketplace backed by OTP Group, outgrew a Microsoft SQL Server and Power BI setup that relied on daily batch loads and separate virtual-machine Python workloads. The company migrated about 50 tables and several stored procedures to Databricks SQL, recreated core views, and moved to a lakehouse architecture using an MVP-first approach rather than redesigning everything. Completed in three months, the migration switched off SQL Server while keeping Power BI reports running on the new engine. Nightly processing fell from three to four hours to roughly 90 minutes, reports became consistently available by 4:30 a.m., Power BI refreshes dropped about 50%, and gigabyte-scale exports finish in minutes. Databricks now unifies SQL, Python, machine learning, governance work, and Hungarian-language Genie self-service, with about 20 users reclaiming roughly 20% of an analyst’s time.
Zero downtime patching lakebase part-1 prewarming
Lakebase addresses the workload disruption caused by planned PostgreSQL maintenance, where a restart clears in-memory buffer and local file caches. Although the database can return in about one second at P99, the source reports a roughly 70% pgbench TPS reduction while caches warm, potentially causing timeouts. Lakebase uses stateless Postgres compute replicas and disaggregated shared storage to prewarm a replacement compute before a scheduled restart. The control plane transfers cached-page information, loads pages from shared storage, and follows relevant WAL from Safekeepers, while the current primary continues serving traffic. After prewarming, Lakebase promotes the new compute with PostgreSQL’s pg_promote without restarting the server; tests on 10 GB pgbench workloads found nearly instant throughput recovery, especially for read-only queries.
Protect Performance and Reduce Surprise Costs with Default Warehouse
Databricks SQL’s generally available Default Warehouse lets administrators choose the SQL warehouse automatically used for ad hoc experiences, addressing manual selection and fallback behavior such as “Last Selected” or alphabetical ordering. A workspace-level setting applies across SQL Editor, Catalog Explorer, AI/BI Dashboards, Alerts, and Genie Spaces, while users can override it or retain “Last Selected”; admin APIs and a Terraform resource support per-user assignment, auditing, and control. The feature supports serverless, pro, and classic SQL warehouses and respects existing governance and access controls, with the setting opt-in and new assets automatically using it. In customer results, Catalog Explorer queries sent to smaller warehouses rose from 77% to 96%, and users running ad hoc SQL across multiple warehouses fell by 15%, while future coverage is planned for MLflow, Lakeflow CLI, and DBSQL MCP.
The next era of the open lakehouse: Apache Iceberg™ v3 in Public Preview on Databricks
Databricks has placed support for Apache Iceberg v3 into Public Preview, bringing new table-format capabilities to its open lakehouse. Unity Catalog Managed Iceberg v3 tables support Row Lineage, Deletion Vectors, and the VARIANT type, while interoperability extends these features to foreign Iceberg tables. Row IDs and modification sequence numbers identify changed rows, and lightweight delete files avoid immediate Parquet rewrites, with data manipulation reported as up to 10x faster than copy-on-write. VARIANT stores evolving semi-structured payloads beside relational columns for direct SQL querying, while Unity Catalog provides cross-catalog governance and Delta-Iceberg interoperability without data duplication. Iceberg v3 is available on Databricks Runtime 18.0+ with Unity Catalog enabled; Databricks is also advancing Iceberg v4 metadata proposals.
What is Agentic Analytics?
Agentic analytics is an approach in which autonomous AI agents continuously monitor data, detect patterns and anomalies, generate contextual insights, and take actions with minimal human intervention. Unlike traditional business intelligence, which relies on people to query dashboards and interpret reports, these systems operate as goal-directed decision loops that can recommend or trigger alerts, forecasts, workflows, and system changes within defined boundaries. Their architecture combines data access, agent orchestration, LLM-based reasoning, a semantic layer, action integrations, and feedback signals. The source identifies faster insight, lower reporting overhead, reduced analytics backlogs, and potentially better forecasting as benefits, while stressing governance, auditability, security, monitoring, and human oversight to control errors and unintended outcomes.
Database Branching in Postgres: Git
Shared staging databases often drift as schemas diverge, test data accumulates, and developers avoid creating isolated environments because traditional pg_dump copies are slow and expensive. Databricks Lakebase Postgres addresses this with database branching, which creates fully isolated environments from a parent’s exact schema and data at a point in time. Branches use copy-on-write: they share underlying versioned storage until a branch changes, then record only new data, while each branch has independent compute and a separate connection string. Idle branches scale compute to zero, and storage cost tracks changes rather than the full database size. This supports per-developer, per-pull-request, per-test-run, and recovery workflows, including point-in-time branching within the restore window, with the source describing branch creation through the console, CLI, or API in under a minute.
Memory scaling for AI agents
Memory scaling is proposed as an agent’s performance improving when it accumulates past conversations, feedback, interaction trajectories, and business context in persistent external memory. Unlike model-parameter or inference-time scaling, it relies on selective retrieval to supply relevant information without filling long context windows. In MemAlign experiments on Databricks Genie Spaces, adding labeled examples raised test scores from near zero to 70%, about 5% above an expert-curated baseline, while average reasoning steps fell from about 20 to 5; filtered user logs produced similarly strong gains, exceeding 50% after 62 records. Production use requires storage and retrieval for episodic and semantic memories with organizational and user-level scoping, plus controls that preserve permissions and sensitivity labels.
coSTAR: How we ship AI agents at Databricks fast, without breaking things
Databricks describes coSTAR, a methodology for testing and iteratively improving long-running, stochastic AI agents without relying on manual review for every change. It addresses nondeterministic outputs, slow executions, cascading failures, and subjective quality by treating scenarios as fixtures, MLflow traces as a reusable harness, and judges as an automated test suite. Its two coupled STAR loops align judges with human experts using a Golden Set, then use those judges to refine the agent until it passes the scenarios. Agentic judges selectively explore traces with tools instead of receiving entire, potentially thousands-line executions, while persisted traces allow rescoring without rerunning agents. The approach can also reuse judges for production monitoring, but remains limited by manually authored scenarios, judge-calibration costs, coverage gaps, overfitting, and difficult root-cause attribution.
Business Analytics Tools: A Complete Guide for Data
Business analytics tools have evolved beyond static reporting as organizations seek explanations, forecasts, and immediate actions. This guide categorizes visualization and dashboard platforms, self-service analytics, statistical tools, spreadsheets, and SQL-based query tools, then examines how AI adds natural-language querying, anomaly detection, summaries, and forecasting. It argues that inconsistent, duplicated, or differently governed data can undermine even sophisticated dashboards. Lakehouse architecture combines data-lake scalability with warehouse governance, performance, and SQL compatibility, enabling tools such as Tableau, Power BI, and Looker to use broader, fresher datasets alongside analytics and AI workloads. Reported examples include 94% faster BI queries at Anker Innovations, 98% faster query performance at AnyClip, and consolidation across 120+ global analysts at JLL.
MLOps Frameworks: A Complete Guide to Tools and Platforms for Production ML
The guide explains how MLOps frameworks close the gap between notebook experimentation and reliable, scalable machine-learning production by applying automation, version control, and continuous delivery across the lifecycle. It describes experiment tracking, model registries, workflow orchestration, feature stores, deployment and serving, and monitoring as the capabilities needed to make training reproducible, prevent training-serving skew, automate pipelines, and detect degradation after release. MLflow is presented as a modular open-source option with Tracking, Model Registry, Models, and Projects, supporting run history, lifecycle stages, standardized packaging, and reproducible execution across environments. Kubeflow suits Kubernetes-standardized organizations and large distributed GPU workloads, while Metaflow, cloud-native services, and unified platforms such as Databricks are positioned as alternatives according to developer experience, cloud alignment, or operational preferences.
What is Machine Learning? How It Works, Types and Use Cases
Machine learning (ML) is a subset of artificial intelligence that learns statistical patterns from data to make predictions or decisions without hard-coded rules. The guide distinguishes ML from generative AI, which creates new content, and from deep learning, a specialized ML approach using multilayered neural networks that learns features automatically and typically requires more data and computing power. It describes a workflow of preparing and splitting data, training models by comparing predictions with actual outcomes, and adjusting parameters to reduce error, illustrated with image classification and customer-churn scoring. It also covers supervised, unsupervised, and reinforcement learning, applications including computer vision and natural language processing, newer architectures such as transformers and Mamba, and MLOps practices for deployment, monitoring, and retraining.
Building the future of security with NAB with Lakewatch
National Australia Bank (NAB) is working with Databricks as one of five design partners for Lakewatch, an Open Security Lakehouse intended for modern security operations. NAB, part of Australia’s critical infrastructure, ingests more than 30TB of security data daily while confronting AI-enabled threats, expanding digital services, and growing third-party integrations. The partnership brings NAB’s threat-informed cyber-defense experience to a platform designed to unify signals across environments, apply advanced analytics and AI-driven workflows, generate higher-confidence insights, and automate response actions. Lakewatch is presented as an open, composable, AI-native foundation built natively on the Lakehouse for organizations facing constantly evolving threats, supporting enterprise-wide correlation, proactive threat hunting, detection, monitoring, and response at scale.
Databricks Announces Lakewatch: New, Agentic SIEM
Databricks announces Lakewatch, an agentic SIEM intended to help organizations defend against increasingly sophisticated, AI-driven attackers. Launching in Private Preview, it unifies security, IT, and business data in a governed lakehouse built on Unity Catalog, while supporting open OCSF formats and storage in customer-controlled cloud object stores. Lakeflow Connect ingests and normalizes major security sources, and Detection-as-Code lets teams define YAML rules with SQL queries or Python notebooks, backtest them against historical data, and deploy through CI/CD. Genie Code and Genie Spaces provide AI-assisted ingestion, detection authoring, rule refinement, investigation, and natural-language threat hunting, while custom models can support anomaly detection, behavioral analytics, and entity risk scoring. The announcement also introduces an Open Security Lakehouse Ecosystem and acquisitions of Antimatter and SiftD.ai; Lakewatch is available in Private Preview, with Adobe and Dropbox among the named customers.
How agentic software development will change databases
AI agents are changing software development from a slow, linear process into rapid evolutionary experimentation, creating many small, ephemeral applications and databases. The post identifies three resulting database requirements: near-zero marginal cost with elastic growth, native branching for rapid variants, and open ecosystems that agents can use reliably. Lakebase addresses branching through O(1) metadata copy-on-write at the storage layer, so agents can branch production or test state without physical data copying and pay only for experiment compute. Its decoupled, serverless compute can scale in sub-second intervals and down to zero, while standard open Postgres page formats in cloud object storage keep data accessible to external engines and tools. Lakebase telemetry reports about 10 branches per project on average, some nested beyond 500 iterations, and roughly four times as many databases created by agents as by humans.
Announcing the 2026 Built
Databricks has announced its third annual Built-On Databricks Startup Challenge, inviting early-stage startups that use Databricks as a core part of customer-facing, B2B applications to compete for more than $1M in prizes. Eligible companies must have been founded within the last five years, be incorporated in one of the listed countries, and be no later than Series B if funded. The competition offers a potential Databricks Ventures investment of up to $600,000, up to $500,000 in Databricks product credits across finalists, and access to Databricks resources, workshops, mentorship, and feedback. Judging considers Databricks usage, market potential, and founding team strength and experience; applications close May 22, 2026, with judging in late May and winner announcements during the week of June 15, 2026, at the Databricks Data + AI Summit in San Francisco.
Announcing General Availability and Open Sourcing of Unity Catalog Business Semantics
Databricks announces the general availability of Unity Catalog Business Semantics, a unified semantic foundation for consistent business definitions across BI dashboards, notebooks, SQL applications, and AI agents. Built into Unity Catalog, it applies existing governance, security, lineage, and access controls to centrally defined Metric Views, whose metadata includes display names, formats, and synonyms. Metric Views are authored declaratively in SQL, compiled and executed deterministically at query time, and can use materializations with automatic pre-aggregation, incremental refresh, intelligent query rewriting, and transparent routing; materialization remains in Preview. The core Metric View implementation is being open sourced in Apache Spark OSS for an upcoming release, with Unity Catalog OSS v0.5 support coming soon, while a point-and-click authoring UI is in Public Preview.
How Addepar scales investment workflows with Databricks AI agents
Addepar, a global technology and data platform, manages nearly $9 trillion in assets and moved from older systems to a unified data intelligence platform on Databricks running on AWS. It ingests hundreds of disparate data feeds, standardizes and enriches them, and delivers results through products, APIs and data sharing. Since adoption, pipeline costs fell 60% versus legacy infrastructure, saving more than $2 million, while new pipelines and integrations became five times faster. Addison, a native AI experience embedded in the platform, combines governed portfolio Q&A, proactive insights and action-oriented workflows, using live data and Addepar’s core calculations engine. Addepar is adopting Databricks Agent Bricks, including Supervisor Agent flows with Genie-powered analytics, while Foundation Model Serving and managed MLflow support governed model serving, traceability, evaluation and iteration for human-supervised agentic workflows.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.