Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
- 542 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
Announcing Lakebase Change Data Feed (CDF)
Lakebase Change Data Feed (CDF) is available in Public Preview to reduce the manual effort of moving data from operational databases into downstream systems. The feed is enabled once for all tables in a project, stored and governed in Unity Catalog Managed Tables, and readable by engines, models, and agents without separate extraction pipelines. From one shared feed, teams can build streaming pipelines with SDP, create materialized views with DBSQL, or compute and store embeddings with Agent Bricks, while consumers remain isolated from the primary operational workload. The announcement positions Lakebase as the native Bronze layer in a medallion architecture, complementing Synced Tables and providing governance and lineage across the data lifecycle.
Building a FHIR-native health data platform on Databricks Lakebase
Healthcare organizations often keep a FHIR server, analytics warehouse, and ETL pipelines separate, replicating clinical data and fragmenting governance. The proposed architecture standardizes incoming HL7v2, C-CDA, X12, and proprietary data into FHIR using Health Samurai converters, terminology normalization, MDM/MPI deduplication, and Implementation Guide validation. Aidbox, Health Samurai's FHIR server and database, runs on Databricks Lakebase, while Moonlink synchronizes operational and analytical formats without ETL. This exposes one governed dataset through Spark, SQL, ML, AI/BI, FHIR API, SMART on FHIR, and SQL on FHIR ViewDefinitions. The architecture supports use cases including EHR optimization, value-based care, member engagement, and compliance capabilities, with insights connected to clinician and billing workflows through SMART on FHIR and CDS Hooks.
AI readiness in telecommunications
Telecommunications companies are adopting AI for customer experience, network operations, and cost reduction, yet initiatives often stall before production because fragmented, ungoverned, semantically opaque data creates data debt. The post argues that AI readiness depends on a semantic layer unifying datasets and business definitions, governance, and catalog metadata across systems such as Oracle, Snowflake, Salesforce, ServiceNow, and Databricks. It presents Unity Catalog as the proposed foundation, using Delta Sharing, Lakeflow Connectors, and Lakehouse Federation to exchange, ingest, or query data without uniformly replicating it, while privilege-aware metadata and audit logging support compliance. Metric Views, lineage, tags, and glossaries give agents authoritative meanings for measures and terms such as revenue, ARPU, active user, and FTTH. The conclusion is that trustworthy operational AI requires a governed, unified data foundation and organizational commitment, not simply more capable models.
Pharma launch analytics: How to compress the first 90 days and win the three years that follow
Pharmaceutical launch analytics depends on compressing the time between data signals and commercial decisions, because early choices shape a trajectory measured over 12-to-36 months. The source frames the first 90 days as three phases: weeks 1–4 validate feeds, set NBRx and patient-start benchmarks, and identify coverage gaps; weeks 5–8 support tactical adjustments through AI-generated narratives, adoption cohorts, and access-barrier escalation; weeks 9–12 recalibrate against benchmarks, shift promotional spend, and record decisions. Databricks Genie lets commercial leaders question unified Rx, specialty-pharmacy, payer-coverage, field-activity, and patient-services data in natural language at prescriber, territory, and regional granularity, with governance and benchmark context. The stated operating benefit is a decision cycle under seven days, enabling teams to detect suppression early, reallocate resources, and respond to access barriers while the launch remains correctable.
Scaling for MHHS: 50x cost-efficient margin data engineering at Octopus Energy
Octopus Energy’s move to Market-wide Half-Hourly Settlement (MHHS) increases settlement data volume 48x for its more than 8 million customers, exposing the cost limits of a monthly, single-grain margin pipeline. The rebuild separates margin processing into half-hourly settlement cost, half-hourly revenue for smart tariffs, and daily revenue for standard tariffs, all backed by a unified multi-grain consumption layer. A “Job of Jobs” Databricks workflow coordinates the streams, while Change Data Feed where available and microbatch dbt elsewhere enable incremental processing; Spark work uses pruning, broadcast joins, partition tuning, and Adaptive Query Execution. Rows processed per run fell from 25 billion to 300 million, and freshness improved from weekly to daily. Projected cost per settlement date fell from $23.63 to $0.48, with approximately $1 million in annualised cost avoidance, excluding upstream savings.
Accelerating LLM Inference with Prompt Caching for Open‑Source Models on Databricks
Repeatedly processing identical system or instruction prompts for LLM requests wastes compute, increases latency and costs, and limits throughput. Databricks extends its built-in prompt-caching capability from proprietary models to open-weights models used through Foundation Model APIs, including batch inference, pay-per-token, provisioned throughput, and higher-level services such as Agent Bricks, Genie, and AI Functions. Supported models include GPT-OSS 20B and 120B, Gemma 3 12B, fine-tuned Llama 3.1 8B via PEFT serving, and Llama 3.1 8B and 3.3 70B. By implicitly reusing KV caches for identical prompts, the system skips prefill on cache hits; caches are isolated, held only in volatile memory, and never persisted. In a production GPT-OSS batch-inference pipeline, per-replica input-token throughput rose 2.5x and P50 latency fell 3x despite a 30% cache hit ratio, with no customer configuration required.
Observability for any agent, anywhere: Production-ready tracing with OpenTelemetry & Unity Catalog on Databricks
Databricks supports writing OpenTelemetry (OTel) traces directly to Unity Catalog, where real-time telemetry is stored in Delta tables for governed analytics and retention. AI traces capture prompts, tool calls, responses, latency, and execution paths, enabling debugging, evaluation, and monitoring, while lakehouse storage also allows SQL queries, dashboards, joins with business data, and PII controls. The managed, serverless ingestion layer uses Zerobus Ingest to accept OTLP over gRPC from collectors and REST integrations, streaming spans, logs, and metrics to Unity Catalog without intermediate message buses. A LangGraph support manager assistant demonstrates instrumentation with mlflow.langchain.autolog() and an @MLflow.trace root span, while Genie is invoked through MCP for data-driven questions. The resulting traces can be searched in MLflow, evaluated at scale, and monitored continuously, with FAQ details stating 200 QPS starting throughput, no storage limit, and Unity Catalog governance options for access control, masking, and filtering.
How World Bank Group uses Databricks to eradicate poverty through shared knowledge
The World Bank Group is building a unified data and AI platform on Databricks to make tens of millions of repository documents and structured operational data findable for teams pursuing shared prosperity. Legacy databases and disconnected document repositories forced researchers to search manually, slowing decisions and limiting the ability to surface lessons learned across its global portfolio. The implementation migrates operational data, applies Unity Catalog governance, uses Databricks Volumes for unstructured content, Genie for natural-language structured queries, a metrics layer for deterministic reporting, and vector search for retrieval-augmented generation over project documents. Because each Genie serves a specific metrics domain, an agentic layer combines intent and domain classification with query decomposition, routing requests among domain agents, RAG, and visualization components before assembling one response; a multi-region feedback prototype was deployed in approximately two and a half days.
Using observability data to prevent incidents
Engineering teams often remain reactive not because they lack telemetry, but because metrics, logs, traces, error budgets, and SLO burn rates are difficult to query at decision speed. The post frames reliability intelligence as trend analysis that identifies accumulating risk before a user-facing incident, contrasting it with observability focused on current system state. It proposes unified telemetry access and natural-language self-service, using Databricks Genie to examine p99 latency, error-budget burn, deployment frequency, incident rates, dependencies, and capacity across services. The Trade Desk’s Genie Router handles questions across more than 10,000 tables, while the post says ad-hoc analysis runs five times faster with Genie and can support mitigation before a page fires.
How Databricks Genie democratizes data access in financial services
Financial services organizations have built sophisticated lakehouses, streaming pipelines, model-serving infrastructure, and self-service BI, but access remains concentrated among technical teams. Business leaders still often rely on analysts because they may lack SQL skills, BI training, or analyst access, creating the “last mile” of data democratization. Databricks Genie addresses this gap through a conversational AI interface that converts plain-English questions into governed SQL queries executed against the Databricks Lakehouse without an analyst in the loop. It operates within Unity Catalog access policies, restricts users to authorized data, makes queries read-only, and logs interactions for audit purposes, while its semantic layer maps organizational terminology such as NIM, LTV, and NII to the organization’s meanings. The stated outcome is faster, auditable answers for business questions and usage data that can inform data-product priorities.
How security teams can report cyber risk to boards
Boards are seeking visibility into cyber risk, but technical reports often fail to connect security posture with business impact or financial exposure. The post explains that compliance and cyber risk leaders can use Databricks Genie to query vulnerability posture, asset criticality, threat intelligence, control data, and historical incident costs in a governed environment. It recommends probabilistic financial modeling, including Monte Carlo simulation, to run randomized attack scenarios and produce loss distributions; Value-at-Risk framing can make those results familiar to directors. This approach replaces qualitative red/amber/green reporting with expected-loss ranges, supports investment prioritization, and enables trend analysis and board-ready answers, while the suggested cadence combines quarterly strategic briefings, monthly operational reviews, and incident-triggered updates.
Transforming industries with conversational AI: Partner solutions built on Databricks Genie
Databricks presents a first cohort of partner-built industry solutions that use Genie to reduce the “Analyst Bottleneck,” where leaders wait for custom SQL or dashboard updates. Genie provides a conversational analytics layer grounded in Unity Catalog metadata and business semantics, using specialized AI agents to return governed, secure, traceable answers to natural-language questions. The featured accelerators span communications, media and entertainment, financial services, healthcare and life sciences, manufacturing and energy, public sector, and retail, with examples including churn prediction, next-best-action recommendations, billing anomaly investigation, KYC monitoring, actuarial analysis, and commercial insights. Across the examples, solutions combine domain expertise with Databricks platform capabilities such as machine learning, Agent Bricks, Lakebase, Serverless SQL, and auditable governance to support self-service analysis and operational decisions.
From emissions reporting to decarbonization decisions
Energy companies have built sophisticated infrastructure for Scope 1, 2, and 3 emissions reporting across complex asset portfolios, but that infrastructure is often oriented toward historical disclosure rather than forward-looking action. Sustainability leaders still need timely answers about which assets warrant intervention, what operational choices drive carbon intensity, and whether current performance is tracking toward commitments. Databricks Genie addresses this gap by letting users query emissions, operational, and financial data in natural language, linking results to dispatch decisions, fuel purchases, and asset utilization. It supports multi-scope analysis, traceability to source data, and scenario questions such as retiring an asset or increasing renewable PPA volume. The stated outcome is a shift from compliance reporting toward data-driven decarbonization decisions, with Genie available today for energy-sector use.
You’ve built the media products, now make them personalized
Media companies may have launched streaming services, digital editions, and mobile apps, yet still face a digital product intelligence gap: product teams must answer behavioral data questions quickly enough to personalize experiences and improve engagement. Databricks Genie gives nontechnical product leaders a conversational interface that translates natural-language questions into SQL queries, visualizations, and actionable insights over governed enterprise data, without requiring code or analyst handoffs. The agent queries governed Delta Lake tables fed by streaming clickstreams, views, and session signals, combining event-level behavior, A/B tests, audience segments, and cross-platform data in conversational answers. The source says internal benchmark accuracy improved from 32% to over 90% through multi-LLM orchestration, specialized knowledge search, and parallel reasoning, and presents Genie as reliable enough for production personalization decisions.
From "What Happened?" to "What Will Happen?"
Databricks Genie makes descriptive analytics accessible in natural language, but predictive questions still require specialized data science workflows and carefully prepared datasets. This post presents a multi-agent supervisor deployed as a Databricks App, combining Genie, TabPFN, and Agent Bricks to turn business questions into predictions. The orchestrator asks Genie to use governed Lakehouse data, schemas, relationships, and semantics to generate labeled training data through SQL, then sends it to TabPFN, which predicts in a single forward pass without feature preprocessing, model selection, or hyperparameter tuning. The resulting conversational experience supports descriptive and predictive analytics with Unity Catalog lineage and access control, while an MLflow GenAI evaluation harness monitors reliability and regressions. Its central limitation is that predictions depend on Genie producing a meaningful dataset with a clear label, so missing signals, joins, outcomes, or agent omissions can make results unreliable.
Unlock seamless and cost-effective marketing campaigns with Lakebase
Retail marketing teams often store customer segments in OLTP databases, creating underused capacity between campaigns and synchronization work whenever new segments are requested. Lakebase, Databricks’ implementation of a lakebase architecture, separates storage from compute and uses elastic, serverless Postgres that can scale to zero when idle and up for bursty campaign traffic. The integration with the Lakehouse uses managed Synced Tables for customer segments and Lakehouse Sync for continuous CDC-based replication of operational data into Unity Catalog Delta tables. The SAP Engagement Cloud example configures a Lakebase Autoscaling project, native Postgres credentials, an ISRG Root X1 certificate, and snapshot synchronization for datasets with more than 10% updated. The post concludes that this design lowers idle and sizing costs, reduces pipeline maintenance, and supports low-latency point lookups, while warning that Lakebase is not optimized for large scans or classic OLAP.
Governing AI agents at scale with Unity Catalog
With AI agents proliferating across organizational functions, governance must answer which agents access sensitive data and how they act. Traditional approaches either scatter logs and permissions across systems or restrict deployment, while agents require controls over autonomous tool use and runtime behavior. Unity Catalog and Unity AI Gateway address this through four pillars: delegated access, data-centric governance, cost intelligence, and open interoperability. The design passes user identity through agents with on-behalf-of tokens, evaluates MCP tool calls with Service Policies, applies inline guardrails, and records model, access, trace, and usage data in lakehouse tables. It also connects data quality, classification, spend, and outcomes, while supporting multiple frameworks and model providers through shared governance.
How telecom CFOs can make smarter network capex decisions with AI
Telecom CFOs face multi-year, multi-billion-dollar network capex decisions involving spectrum, fiber, and 5G densification amid technology uncertainty and competitive dynamics. The post argues that operators can improve allocation by unifying network quality, customer churn, ARPU, billing, and investment history in a queryable environment. Databricks Genie provides a governed natural-language interface over that enterprise data, allowing finance leaders without SQL or data engineering backgrounds to ask cross-domain questions at geographic granularity and receive answers from systems of record. Example analyses compare ARPU and churn after 5G densification with markets still on the deployment roadmap, while scenario modeling can test accelerated deployment against historical returns. The stated outcome is better evidence for strategic judgment, shifting conversations from generic industry benchmarks toward comparable investments in the operator’s own network.
How Databricks Genie improves retail personalization
Retail personalization often stalls when CX leaders must wait for analysts to answer questions about segment behavior, loyalty performance, channel preference, or churn risk, narrowing the window for action. Databricks Genie addresses this access gap by letting business users query unified structured and unstructured enterprise data in plain English instead of SQL. Its retail differentiators include identity-resolved queries across channels and devices, lifecycle-stage awareness, campaign response and control-group data integration, and privacy controls enforced through Unity Catalog. The source says Genie reduces routine analyst requests rather than replacing data science teams, while enabling merchandisers, category managers, loyalty marketers, and CX leaders to self-serve operational questions. It cites 7-Eleven’s use of Databricks SQL, Unity Catalog, and AI/BI Genie to launch, refine, and measure personalized offers within a secure, unified platform.
Databricks for Good and Virtue Foundation: Partnering to Connect Medical Volunteers to Critical Health Services in 72 Countries
Virtue Foundation’s VF Match platform connects medical professionals with volunteer opportunities across 72 low and low-middle income countries, but scaling its initial LLM proof of concept into production required a robust data pipeline. The Databricks-based Foundational Data Refresh combines Overture Maps and Bright Data, processes more than 25 million web pages with targeted GPT extraction steps, and uses Spark, Photon, and Lakeflow Jobs to orchestrate checkpointed, modular workloads. Splink resolves duplicate facilities and nonprofits across messy sources; Photon reduced the slowest entity-resolution partition from 30 minutes to approximately two minutes, a 15x improvement. A prototype VF Agent adds natural-language access through LangGraph, Databricks Model Serving, AI Search, and Genie, while the resulting system provides an up-to-date global view of healthcare infrastructure and volunteer needs.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.