Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
- 542 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
Database modeling: a practical guide to techniques and best practices
Database modeling defines the structure, relationships and constraints that keep systems coherent across transactional, analytical and flexible workloads. The guide presents a three-phase design process: conceptual modeling captures entities and relationships, logical modeling adds attributes, keys, cardinality and normalization, and physical design translates that model into tables, indexes, constraints, partitioning and other platform-specific choices. It compares relational, document/NoSQL, dimensional, hierarchical and network models, linking each to data shape, workload and consistency requirements. It recommends explicit keys, consistent naming, appropriate normalization, query validation and version-controlled DDL, while warning against premature physical design, missing foreign keys, unclear NULL handling and poorly balanced schemas. It concludes that Databricks Lakebase combines a transactional database engine with the Databricks Lakehouse Architecture so one model can support operational and analytical workloads.
MLOps vs DevOps: A Practical Guide for Data Scientists and IT Teams
The guide compares DevOps and MLOps, framing both as practices for reliable production systems but distinguishing traditional application code from machine-learning systems involving data and models. DevOps uses source control, automated testing, staging, and CI/CD, while MLOps governs the “holy trinity” of code, data, and models through separate versioning, quality checks, lineage, and access controls. Its central operational difference is Continuous Training: MLOps pipelines can validate model quality, monitor drift, and trigger retraining when data distributions shift, alongside testing model-serving infrastructure and promoting models through a registry. The guide also assigns responsibilities across data scientists, ML engineers, data engineers, and IT operations, and recommends dedicated MLOps for business-critical, frequently retrained models, DevOps for applications without ML, and split pipelines for hybrid products.
Top Data Warehouse Tools For Modern Data Analytics
Data warehouse tools are presented as a strategic choice for analytics and ML teams facing fragmented estates of warehouses, lakes, and standalone ML systems. The guide proposes evaluating platforms across query performance, scalability, data integration, BI connectivity, total cost of ownership, and governance and security, with attention to MPP, columnar storage, and decoupled compute and storage. It contrasts traditional warehouses, data lakes, and lakehouses: warehouses favor structured SQL analytics, lakes offer native-format flexibility but weaker quality and performance guarantees, and lakehouses combine these capabilities using open formats such as Delta Lake and Apache Iceberg. The stated conclusion is that a lakehouse can provide a single governed foundation for SQL, BI, streaming, ML, and AI workloads, while teams should select tools according to workload, scale, budget, and future AI needs.
Agentic AI vs Generative AI: Comparing Autonomy, Workflows, and Use Cases
Agentic AI and generative AI differ primarily in whether a system autonomously pursues a multi-step goal or produces content in response to a prompt. The post defines agentic systems through a perceive-plan-act cycle: agents maintain memory and state, decompose goals, call tools or sub-agents, evaluate conditions, and recover from errors, while generative AI typically performs bounded, reactive inference. It presents workflow examples such as sales follow-up and market-intelligence summarization, showing how APIs connect agents to CRMs, databases, communication platforms, and news services while LLMs provide text generation or reasoning at individual steps. RAG can ground generative outputs in external knowledge, but agentic deployments add operational concerns around repeated inference loops, human oversight, provenance logging, and access controls. The conclusion recommends choosing by task structure: generative AI for single-turn creation or summarization, agentic AI for autonomous coordination, and both together for complex enterprise workflows.
Unlocking SAP business context in Databricks with semantic metadata Delta Sharing
SAP Business Data Cloud now offers generally available semantic metadata synchronization with Databricks Unity Catalog for mounted SAP BDC Delta Shares. When a table is accessed, Unity Catalog receives table-level business-friendly display names, descriptions, context, and governance tags, while SAP BDC remains the single source of truth and subsequent changes are reflected. The capability builds on BDC Connect and Delta Sharing, allowing governed SAP data products to be discovered and combined with other enterprise sources without recreating business context or governance separately. Column descriptions, table relationships such as primary and foreign keys, and PersonalData namespace tags give Databricks AI Assistant and AI/BI Genie explicit context for natural-language questions and join-ready queries. The stated result is more understandable, discoverable, and AI-ready SAP data, with automated classification signals supporting compliance, access control, and responsible AI.
The marketing activation gap has a fix: Databricks and Stitch partner to turn data infrastructure into marketing performance
Databricks and Stitch are partnering to connect enterprise data infrastructure with marketing execution, addressing a gap that leaves campaigns dependent on stale segments, delayed data, and disconnected tools. The partnership positions Stitch as a marketing implementation layer that structures Databricks data for real-time segmentation, personalization, AI-driven decisioning, and self-service analytics while building applications and agents directly on the platform. Its work spans campaign-ready architecture, full-stack marketing applications, Genie-based access for nontechnical users, AI-powered campaign operations, and migrations from legacy platforms. Examples include real-time transaction data reaching customer marketing at a convenience-store brand, measurable campaign results within weeks for a medical testing company, and a global QSR rebuilding campaign workflows on Databricks as AI tools improve.
Alert fatigue is a business risk
Enterprise security operations centers may receive tens of thousands of alerts daily, making prioritization necessary and leaving lower-priority signals uninvestigated. Alert fatigue is presented as a data architecture problem: fragmented endpoint, network, identity, and cloud telemetry, combined with proprietary SIEM collection-and-discard practices, limits correlation and overwhelms analysts. Lakewatch proposes an open lakehouse foundation that unifies security, IT, and business telemetry, applies automated OCSF normalization, and uses Agent Bricks for data wrangling and alert triage. Databricks Genie is positioned as a natural-language AI security agent whose autonomous agents can hunt, summarize, and neutralize threats, while Unity Catalog logs queries and actions for audit and forensic purposes. Lakewatch is currently available in Private Preview.
Backstage with Lakebase
Backstage with Lakebase describes a proof of concept that moves Backstage’s stateful internal developer portal from standard Postgres to Databricks Lakebase, a serverless Postgres surface in the Databricks Data + AI Platform. Connection required an app-config.yaml change, PgSearchEngine instead of the default in-memory search, and short-lived OAuth JWT credentials generated with databricks postgres generate-database-credential; Knex migrations then ran cleanly. Using copy-on-write branches, the team cloned a roughly 63 MB catalog in 1.09 seconds and recovered 32 deleted entities in 3.78 seconds, while observing that point-in-time recovery snaps backward to the nearest WAL record. The post argues that cheap, isolated database state can replace much mocking and staging iteration with real-data development, branch-based testing, schema validation in CI, and automated cleanup, while noting that VS Code/Cursor synchronization tooling remains underway.
Shipping faster isn't learning faster
Product organizations can ship features in days while taking weeks to understand their behavioral impact, because fragmented analytics stacks depend on analysts, BI expertise, or SQL skills. The post argues that this architectural bottleneck slows the insight-to-ship cycle, causing roadmap decisions to rely on instinct, anecdotes, and lagging indicators. It presents Databricks AI/BI Genie as a conversational interface to event-level behavioral data, with experiment integration, cohort analysis, and product-specific growth-metric definitions. According to the post, product leaders can ask questions in plain language without filing analyst requests, while governed data access supports faster follow-up and feature-impact decisions. It reports that Genie users across 3,300+ Databricks customers cited a 49% productivity gain, 41% faster speed to market, and 5x faster ad-hoc analysis, though these are reported customer results.
Why your OEE dashboard is lying to you
Manufacturing OEE dashboards can look healthy while omitting recurring downtime that supervisors recognize, because relevant facts are split across SCADA, MES logs, maintenance tickets, and shift reports. OEE combines Availability, Performance, and Quality, but extracting those inputs often requires SQL or an analyst, delaying root-cause analysis after throughput drops. Databricks Genie is presented as a conversational AI layer over a unified data platform that leaves MES and SCADA in place while allowing business leaders to ask plain-language questions such as OEE by line against maintenance windows. Its semantic-layer awareness maps terms to actual fields, while governed access and logged, source-cited answers support different visibility levels and traceability. The proposed shift is from static reporting to faster operational questioning about production status, forecast risk, line performance, and quality signals.
The turbine that tried to tell you it was failing
Gas turbines produce millions of daily readings—vibration, temperature, pressure, flow rates, and electrical output—but maintenance teams may learn about warning signals only when an unplanned outage occurs. The post argues that predictive maintenance has struggled less because of model capability than because decision-makers lack fluid access to model findings and operational context. Databricks Genie provides a conversational interface to sensor, maintenance, dispatch, cost, and regulatory data, allowing questions about elevated vibration against maintenance baselines and timing maintenance around outage cycles. It is presented as decision support, not automation, with time-series analysis, maintenance history, integrated costs, and compliance context helping asset managers investigate fleets and act with more confidence.
Predicting readmissions isn't enough. Acting in time is.
Readmission risk models can identify patients likely to return within 30 days, but prediction alone does not ensure timely intervention. In large health systems, risk information may remain in population-health dashboards or EHR flags without reaching the care coordinator with enough context to create an effective post-discharge plan. The post presents Databricks Genie as a natural-language interface for governed clinical and outcomes data, allowing leaders to query readmission rates and compare periods while combining EHR, operational, financial, risk-score, intervention, and outcome information. Genie operates within Unity Catalog governance, with access controls, audit logging, and de-identification policies, and is described as supporting EHR integration and clinical taxonomy awareness; the intended result is to shorten the path from prediction to action.
Clinical trials run longer than they have to. That's a patient problem.
Clinical trial operations can lose weeks because site-level performance data is reviewed with a 2–4 week lag: enrollment velocity is monthly, while protocol deviation rates are assessed quarterly. The post presents Databricks Genie as a natural-language interface to unified trial data, allowing clinical operations leaders to query enrollment, screen-failure, protocol-deviation, query-response, and data-entry metrics across sites. Its stated capabilities include automatic site comparison, integration across CTMS, EDC, safety databases, and site-performance data, plus protocol-aware reasoning and traceability to source records for GCP documentation. The example query identifies Phase II oncology sites with screen-failure rates above 40% over 60 days and compares enrollment pace with activation targets, positioning earlier detection as a way to reduce timeline impact and speed treatment access.
Network quality is a revenue problem, not a technical one
Telecommunications network operations centers monitor extensive performance telemetry, but the source describes a gap between technical network events and the customers, contracts, and revenue they affect. A degraded tower serving 12,000 postpaid customers averaging $85 per month presents a different business problem from one affecting 12,000 prepaid customers with high churn propensity, while enterprise SLA exposure can remain hidden from the NOC. Databricks Genie connects network elements, serving areas, customer records, contract terms, SLA thresholds, and churn-risk scores so leaders can query commercial impact in seconds. It also supports proactive alerts based on defined commercial thresholds, enabling restoration and investment decisions to account for customer impact rather than technical severity alone.
Shelf availability starts with better demand visibility
Retail out-of-stock rates in grocery and general merchandise typically run between 7% and 10%, leaving roughly one in ten sought-after items unavailable at a given moment. The revenue impact is real, but repeated shelf gaps can cause customers to build shopping habits elsewhere. Modern retail replenishment requires real-time synthesis of POS velocity by store and SKU, distribution-center inventory, on-order quantities and delivery windows, supplier fill-rate histories, promotional calendars, and other demand signals. Databricks Genie is presented as a conversational interface to the full inventory and demand environment, allowing leaders to ask which high-velocity SKUs are projected to stock out within 72 hours and receive current on-order positions in seconds rather than hours. Its stated differentiators include store-SKU granularity, shared promotional, event, and weather data, supplier performance context, and cross-distribution-center rebalancing; the post says Genie is available today.
When predicting the next hit requires more than intuition
Content investment decisions in entertainment commit substantial funds despite limited information, because leaders often combine executive instinct, competitive benchmarks, and historical performance data. Although companies hold viewing and subscriber signals such as episode completion, skip patterns, genre performance by demographic, acquisition, and retention, those insights are frequently buried in dashboards and spreadsheets. The post presents Databricks Genie as a natural-language interface that lets content leaders query governed performance data directly, including comparisons linking content types with 90-day retention, subscriber demographics, lifetime value, acquisition source, and historical greenlights. It argues that current, conversational access can complement creative judgment by reducing dependence on delayed analyst recommendations, while external market data can add competitive context.
Approximate answers, exact decisions: New sketch functions for analytics
Databricks now supports four sketch function families, built on Apache DataSketches, for replacing expensive exact percentiles, distinct counts, set operations, top-K rankings, and metric aggregations with bounded-memory approximations. KLL sketches answer quantiles, Theta sketches support unions, intersections, and differences, approximate top-K sketches track frequent items, and Tuple sketches combine distinct counting with aggregated metrics such as customer revenue. The sketches can be built during ETL, stored as columns in Delta tables, and merged on read, allowing dashboards and streaming pipelines to avoid rescanning raw data; SQL, DataFrame, and Structured Streaming pipelines are supported. The stated trade-off is configurable 1–2% relative error, with the source citing a 1000x speedup for applicable workloads. Exact methods remain appropriate for financial auditing, compliance reporting, and other precision-required uses.
Companies winning with AI built the data layer first
Trinity Industries’ experience is presented as a case for treating the data layer, rather than models or dashboards, as the foundation of enterprise AI. The railcar manufacturer migrated 95% of its enterprise data to a single Databricks lakehouse, adopted Medallion architecture, moved transformations upstream, and retired legacy dashboards containing nearly 600 measures. That consolidation supports streaming ETA predictions, procurement agents, and Genie conversational analytics: the ETA model is reported as 50% more accurate than industry ETAs, agents helped increase on-time material delivery by 15%, and Genie handles more than 1,000 questions monthly. The migration took close to a year, followed by six to eight months of additional work, but the account argues that trusted, standardized data enables experimentation, automation, and broader employee access to analysis.
Rethinking SQL ETL for modern data platforms
SQL ETL is often distributed across a warehouse, transformation framework, orchestrator, monitoring, lineage, and data-quality systems, creating operational fragmentation as data teams scale. The post proposes a unified platform model that combines execution, orchestration, observability, governance, and optimization while supporting analytics engineers, warehouse engineers, and analysts through dbt, scripts and stored procedures, Materialized Views, declarative pipelines, and no-code tools. Databricks implements this model with shared execution, governance, and observability, using serverless infrastructure and AI-driven optimization alongside open table formats and ANSI SQL. The stated benefits include reduced operational overhead, support for batch and real-time workloads, and incremental modernization rather than disruptive rewrites. Reported examples include HP’s 32% cloud savings and 36% lower combined job runtime after moving to serverless compute, and Adobe reducing query execution from eight minutes to three seconds with Materialized Views.
Stripe data now available on Databricks via Databricks Marketplace
Stripe Data Pipeline is now available on Databricks Marketplace through Delta Sharing, allowing payment and business data to be analyzed in a Databricks workspace without ETL. Stripe data—including transaction records, customer histories, subscriptions, refunds, and payouts—remains in Stripe’s infrastructure and is queried directly through Unity Catalog, where it can be joined with other tables. The approach replaces polling integrations and custom jobs, avoiding per-call charges, connector licensing fees, data duplication, and maintenance code while providing row- and column-level access controls, audit trails, and compliance features. The shared data is positioned for AI-native uses such as continuous transaction monitoring, churn scoring, LLM-powered retention workflows, Genie queries, and finance analytics apps. Getting started involves visiting the Stripe Data Pipeline listing on Databricks Marketplace.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.