Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
- 569 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
Databricks is a Visionary the Gartner® Magic Quadrant™ for Analytics and Business Intelligence Platforms
Databricks debuted as a Visionary in the 2026 Gartner Magic Quadrant for Analytics and Business Intelligence Platforms, achieving the highest debut performance in the report’s 20+ year history. The company also received one of the highest Completeness of Vision scores among all vendors evaluated. Databricks attributes this recognition to its ability to deliver on three major trends reshaping businesses: data and AI platform convergence, agentic analytics, and semantics at the core. The Databricks lakehouse integrates BI directly, eliminating data silos and the need for extracts, so agents can reason over complete, governed context. With Genie and AI/BI, the platform automates complex workflows and provides a unified semantic layer through Unity Catalog Business Semantics and Genie Ontology.
What's New in Lakeflow Designer - October 2026 Roundup
Lakeflow Designer, a visual no-code data preparation tool in Databricks, added three new operators for cleaning, deduplication, and quality checks, addressing customer requests for more no-code capabilities. The Prepare operator standardizes data by changing case, converting types, replacing values, and applying formulas; Unique removes duplicates across selected columns with ordering control; Guardrails enforces quality rules with configurable failure handling. Existing operators gained multi-table joins with separate outputs for matched, left unmatched, and right unmatched rows, branching in Filter, and dynamic column selection in Select. Output options expanded to materialized views in Unity Catalog that refresh automatically, merge into existing tables, and direct file writes to CSV, Excel, or JSON with dynamic worksheets. Canvas improvements include a redesigned interface with better contrast and larger connection targets for large workflows, a searchable table of contents, visual version history, and live cursor presence.
Closing the AI context gap: what it takes to make enterprise AI trustworthy
The post argues that model intelligence is no longer the bottleneck for enterprise AI; the real gap is business context — the unique knowledge about an organization’s processes, customers, and products. Without proper grounding, agents resort to brute force, iterating through guesses and burning tokens while producing unreliable answers. While building a well-grounded agent for a single domain is now easier, scaling context across hundreds of agents in an enterprise remains unsolved due to fragmentation, fast-changing definitions, and differing team-specific interpretations of the same concepts. Genie Ontology addresses this by providing a unified, governed context layer that combines intentionally curated knowledge with automatically learned observations from system usage. This enables every agent connected to Databricks to work from the same consistent business understanding, supporting accurate answers without each agent rediscovering the business from scratch.
The data lakehouse is a better data warehouse: 2026 benchmarks and proof
Data warehouses have been the standard for structured SQL analytics for two decades but cannot serve modern AI, ML, streaming, and unstructured data workloads. The data lakehouse now matches warehouse SQL performance while serving all these workloads from a single governed foundation. Databricks Lakehouse improved production SQL workload performance by 77% from a 2022 baseline, with BI queries improving 14% over a recent five-month period. Customer migrations show 90% faster queries and 30–40% cost reduction at Lumen Technologies, and 80–90% faster analytics at Trek Bicycle. The lakehouse eliminates redundant data copies, supports open formats like Delta Lake and Iceberg, and extends governance to AI agents through Unity Catalog and Unity Gateway.
How Flo Health built safer, more useful open models with Databricks AI Runtime
Flo Health, a women’s health app with 81 million monthly active users, aimed to improve generative AI responses for medical safety and usefulness while operating at scale. Relying on general-purpose models created cost, dependency, and domain-expertise limitations, so Flo built a post-training pipeline with Databricks AI Runtime. The team defined 26 AI judges across three categories: 12 medical-safety judges acting as hard constraints, 6 micro judges for nuanced behaviors, and 8 usefulness criteria. Using these judges as a reward signal, they generated over 15,000 synthetic questions from privacy-first principles and applied reinforcement fine-tuning with supervised fine-tuning (SFT) and direct preference optimization (DPO). After fine-tuning on 32 NVIDIA H100 GPUs using PyTorch and Fully Sharded Data Parallel (FSDP), medical safety scores rose from 34% to 61%, usefulness from 42% to 76%, and the combined metric improved nearly threefold to 47%.
Brickster Voices: Meet Clara Kurian
Clara Kurian, Databricks’ first university recruiter for APJ, leads the company’s early career hiring in India. Databricks has invested over $250 million in India, treating it as a strategic hub rather than a satellite office, with workforce growth of 270% since 2022 and leadership expansion of 350% across all business units. Clara designed the first structured India Intern Program with 51 interns, aiming to give them a genuine experience of the company’s pace, people, and culture through Speed Mentoring with engineering leadership and activities like Sneaker Art and Pickleball socials. She advises early career candidates to build strong fundamentals, curiosity, and a builder mindset, noting that the field moves fast and thrives on continuous learning. Candidates stand out by showing genuine curiosity, authenticity, and a willingness to figure out unknown problems, as the technical bar is almost a given.
From post-training to production: How TinyFish runs its web-native model on Databricks AI Runtime
TinyFish builds infrastructure for enterprises that need agents to work across the live web and is developing its own specialized web-native model. The company uses Databricks AI Runtime to manage GPU compute for post-training with supervised fine-tuning and moving toward Direct Preference Optimization, MLflow for evaluation, and Model Serving for production deployment. TinyFish evaluates its model by measuring full agent task completion rather than individual responses, achieving a 77.6% success rate on its internal go/no-go benchmark where 1,691 of 2,178 tasks were completed successfully. Over a 15-day period, the web-native model handled 294,781 agent runs across 6.4 million tool steps at an estimated serving cost of $0.088 per run or $0.0040 per tool step. By consolidating training, evaluation, serving, and operational data on Databricks, TinyFish reduces infrastructure overhead and maintains a continuous development cycle from agent behavior to deployment.
How California built its Behavioral Health Public County Profile on Databricks
The California Department of Health Care Services built the Behavioral Health Public County Profile on Databricks, a public dashboard consolidating behavioral health data from all 58 counties for nearly 40 million Californians. Launched on October 1, 2026, following the 2024 passage of Proposition 1, the dashboard supports the new Behavioral Health Services Act (BHSA) tying funding to statewide goals for outcomes, equity, and transparency. The platform uses Spark Declarative Pipelines and Lakeflow Connect for unified data ingestion, Semantic Views for standardized metrics across counties, and serverless compute to process data at scale without infrastructure management. It operates on a FedRAMP-authorized environment meeting federal security standards for health data, with data quality and lineage ensuring every number can be traced to its source. The County Profile enables public comparison of county plans, funding, service delivery, and performance measures, creating a feedback loop between policy intent and measurable outcomes for democratic oversight.
Solving defense supply chain visibility through governed data sharing
The Department of War cannot obtain cost transparency across its supply chain of over 200,000 suppliers due to reliance on manual data calls and fragmented spreadsheets. The proposed solution is a federated data exchange where each contractor publishes validated cost and pricing evidence from existing records to named government recipients using open-source sharing protocols. Databricks provides the platform, enforcing governance, access controls, CUI markings, and audit lineage without storing raw records centrally or requiring contractors to adopt Databricks. Small and mid-tier suppliers submit through validated intake forms that produce the same governed data contracts as large primes, requiring no new accounting systems. The reporting obligation already exists through CSDR, FlexFile, and DD Form 1921, so a pilot exchange can launch in months by defining two data contracts on a single program.
Lakebase and Agentic SDLC: Branching Databases for Coding Agents
Coding agents working in parallel on shared databases create schema conflicts, test interference, and data exposure risks. Lakebase Postgres solves this with database branching, which creates an isolated copy of an entire database in under a second using copy-on-write. Each agent gets its own database branch via a post-checkout Git hook tied to a worktree, and each pull request gets an ephemeral branch through CI tooling. Schema changes are tracked in code with migration tools such as Drizzle or Alembic and promoted to the parent branch during deployment, not by merging database branches. Idle branches scale to zero, so running many concurrent agents incurs no compute cost for unused branches.
Introducing Funke: Native HL7v2 Parsing on Databricks
HL7v2 is the dominant healthcare messaging standard but is difficult to parse due to its nested, delimiter-encoded structure. Existing workarounds like converting to FHIR or using third-party engines add cost and latency. Funke, the successor to the open-source Smolder library, is a new open-source Python and PySpark library under the Databricks License that parses HL7v2 messages directly into native Spark types, preserving the full hierarchy losslessly. It deploys as a Databricks Asset Bundle with a Declarative Pipeline that uses Auto Loader to ingest raw messages into a Unity Catalog volume, then parses them into structured silver tables following the medallion pattern. The library supports all HL7 message types and versions, includes accessor helpers for extracting fields by segment and component, and ships with a demo using synthetic ADT events to illustrate end-to-end ingestion and gold table creation, including a live census and bed utilization dashboard.
Biomedical Imaging's Real Bottleneck Is the Data, Not the Model
Medical imaging’s real bottleneck is not model architecture but data access. Hospitals, academia, medtech, and pharma all struggle to extract, de-identify, and query imaging data locked inside PACS and vendor archives. The approach proposed is a data lakehouse model on Databricks using a medallion architecture that ingests raw DICOM into Bronze, extracts de-identified metadata into Silver, and curates queryable Delta tables in Gold. This foundation unifies imaging with multi-omics, clinical-trial, and real-world data under a single governance model. The algorithms exist; the missing piece is a governed, queryable, and linkable data layer that makes the model the easy 10% again.
Set Budgets and Alerts for Cloud Data Warehouse Costs
Reactive cost management for cloud data warehouses fails because usage is unattributed, budgets are absent, and alerts arrive too late. Databricks’ proactive framework addresses this through five pillars: cost attribution via warehouse and query tags, layered budgets with email alerts, real-time dashboards, and optimizations. SQL Serverless warehouses eliminate idle compute costs, as Lumen Technologies demonstrated with a 30-40% reduction. Query tags (Public Preview) enable per-query attribution, linking spend to specific dashboards or dbt models. The post provides step-by-step instructions for tagging, budget creation in the Account Console, and tuning auto-stop and statement timeouts to reduce waste.
Announcing Workday Data Connect federation in Unity Catalog
Databricks announced the public beta of a Workday Data Connect federation connector for Unity Catalog, enabling zero-copy queries of Workday HR and finance data directly from the Databricks Data and AI Platform without ingestion pipelines. Workday shares tables via an Iceberg REST catalog; Databricks creates a foreign catalog in Unity Catalog and reads data from Workday’s managed object storage, with Unity Catalog governing read-only access through table-level permissions, lineage, and auditing. Teams can query the federated data using Databricks SQL, AI/BI, notebooks, or Genie for natural-language questions about workforce and financial trends. The connector requires Databricks Runtime 19 or above and a workspace admin to enable it from the Previews page. The release builds on the Databricks-Workday partnership announced at Workday Rising 2025 and complements existing Lakeflow Connect and Live Data Query options.
How to Repoint dbt ETL Pipelines to Databricks
Repointing a dbt project to Databricks involves swapping the adapter and profile, handling SQL dialect differences, and keeping transformation logic unchanged. The dbt-databricks adapter bridges the project to Databricks Lakehouse, which uses a three-level namespace (catalog.schema.table) and supports open formats like Delta Lake and Apache Iceberg. Common dialect gaps include REGEXP_SUBSTR, DIV0, and OBJECT_CONSTRUCT, which can be fixed with Databricks-native functions or dbt_utils macros; the post walks through each error and its fix. After compilation and validation using audit-helper to compare outputs model by model, the project can be deployed as a dbt task in Lakeflow Jobs with built-in governance via Unity Catalog. The process is incremental: one-time setup costs apply, then each additional model requires only dialect fixes, making it a low-risk starting point for warehouse migration.
Now GA: Building permission-aware Databricks Apps with on-behalf-of-user authorization
Databricks Apps now generally supports on-behalf-of-user (OBO) authorization, enabling apps to act using the signed-in user’s identity so Unity Catalog enforces existing data permissions including row filters and column masks. Developers choose between app authorization for app-owned operations and user authorization for user-specific data access, with API scopes such as sql:restricted-query limiting what the app can do on the user’s behalf. The forwarded user access token is short-lived and per-request, passed via the x-forwarded-access-token HTTP header to the SQL connector, and the app must never store it. Workspace administrators can set an allowlist of API scopes, providing two levels of control: the app declares minimum scopes and the workspace defines the maximum, with account administrators able to add scopes outside the allowlist. The post recommends keeping app-owned and user-authorized clients separate in code and failing closed when a user token is missing to avoid returning results with incorrect permissions.
The lakehouse is a better data warehouse: 2026 benchmarks and proof
The data warehouse served structured SQL workloads for two decades, but the lakehouse now supports BI, data engineering, ML, streaming, and AI agents from a single governed platform. Databricks reports a 77% improvement in its production SQL workload mix since 2022, with customer migrations showing 90% faster queries (Lumen) and 80-90% faster analytics (Trek Bicycle). Lakehouse//RT delivers sub-second analytical queries at high concurrency directly on governed lakehouse tables, with Databricks reporting sub-100-ms latency at 12,000 queries per second, collapsing the traditional architecture of separate serving databases. The lakehouse eliminates data copying between systems, handles unstructured data, and runs on open formats like Delta Lake and Apache Iceberg, governed by Unity Catalog. Customer cost reductions range from 25% to 75%, with AXA Japan reducing warehouse costs by 60% and ETL costs by 70%, driven by eliminating redundant systems and data copies.
Meta’s ads MCP server comes to Databricks: Put your customer intelligence to work in advertising campaigns
Meta’s ads MCP server, now in Databricks Marketplace, connects Meta Ads Manager tools with enterprise data through Genie for natural language campaign management. The server uses the open MCP standard to expose over 25 tools across the campaign lifecycle including creating campaigns, adjusting budgets, defining audiences, and pulling performance data. Marketers can prompt Genie One to use business data like churn scores and gross margins, for example identifying high-value customers at risk of leaving and launching a retention campaign with an $8,000 daily budget. Governance is enforced server-side: Databricks administrators control access via Unity Catalog while Meta advertisers set rules in Business Settings that block actions like budget increases over 20%, with every tool call checked before execution. The integration builds on the Meta Conversions API partnership, letting agents combine signal diagnostics with Databricks models to put customer intelligence directly into campaign operations and execution.
Scaling and Operating a Large dbt Project on Databricks: IFCO's Data Team on Performance, Visibility, and Debugging
IFCO’s data platform team, with Databricks Forward Deployed Engineering, optimized a large dbt project on Databricks to handle billions of daily tracking events for reusable packaging assets. The pipeline faces scale, unpredictable late-arriving data, and the need to reconcile history when late events change previously reported asset timelines. The team tuned incremental model settings to match Delta Lake write behaviors, including liquid clustering, deliberate merge vs. delete+insert strategies, and bounding writes to recent ingestion windows. Diagnosing performance from actual query plans rather than compiled SQL revealed that the consolidation model was scanning far more rows than necessary. These changes cut the core semantic layer job’s daily runtime by over 60 percent, retired a costly nightly full refresh, and reduced daily compute cost by 58 percent.
NEAREST BY Join: Scaling Vector Search in Databricks Runtime
Batch vector search workloads, such as entity resolution and semantic tagging, require high throughput across millions of queries against billions of vectors, unlike real-time serving. Databricks built NEAREST BY, a first-class top-k ranking join syntax, directly into its runtime engine rather than federating to an external endpoint. The feature is implemented as a logical join that the optimizer lowers onto standard relational operators, enabling fault tolerance and elastic parallelism. A fused Photon operator collapses the plan’s middle section with a blocked GEMM kernel designed to approach peak arithmetic intensity for vector scoring. An optional IVF index, stored as an ordinary liquid-clustered Delta table, allows approximate queries to probe only relevant clusters while preserving the same primitives and correctness guarantees.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.