Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
- 564 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
How MakeMyTrip achieved millisecond personalization at scale with Databricks
MakeMyTrip needed sub-second personalization for “last-searched” hotels across its consumer and corporate travel businesses, serving millions of daily users. Spark Structured Streaming’s micro-batch mode consistently delivered one to two seconds, while adopting Flink would have added a second engine, duplicated logic and increased operational overhead. The team instead adopted Databricks Real-Time Mode (RTM), using continuous data flow, concurrent pipeline scheduling and streaming shuffle to process events with millisecond-level latency on Spark. Its unified pipeline merges B2C and B2B Kafka streams, enriches events with each user’s “Last N” hotels from Aerospike, and writes results to Redis for serving in under 50ms. After tuning partitions, backpressure and rate limits, the production deployment reduced P50 latency from approximately 1.23 seconds to 44ms, P99 latency from over one minute to about 500ms, and increased click-through rate by 7%.
How FSIs eliminate silos between clients, operations, and finance
Financial institutions often split portfolio, operations, actuarial, and finance work across separate systems, even when teams depend on the same portfolios, positions, liabilities, and transactions. Databricks presents a unified, governed platform combining Databricks SQL, Unity Catalog Metric Views, Lakebase, Genie, Databricks Apps, and Lakeflow Designer to connect analysis with operational action. A worked workflow follows an actuary's duration analysis, a portfolio manager's scenario modeling and mandate changes, operations' IBOR/ABOR reconciliation, and finance's approval of adjustments and ledger entries. Governance features including row-level security, column masking, RBAC, ABAC, and shared business semantics keep access and terminology consistent, while Lakebase records corrections in auditable, timestamped tables. The result described is a closed-loop process in which strategy, risk, and accounting use the same governed data rather than reconciling separate systems.
How Zalando built a unified data foundation for AI and analytics on Databricks
Zalando describes how it is building a unified data foundation for analytics and AI across a microservices-driven ecosystem serving more than 50 million active customers and 7,000 brands and partners. Its architecture combines private domain-owned catalogs for autonomous development with a centrally governed shared catalog that exposes cross-team data through Dynamic Views, enabling identity-based policies, GDPR checks, column classification, and auditability. Databricks Metric Views provide a governed semantic layer where business logic and KPI definitions are centralized and reusable across SQL, dashboards, notebooks, and AI agents. Genie uses those metrics plus enriched metadata to answer natural-language questions more reliably, although poorly curated or unaggregated data can increase costs; Zalando plans broader enterprise adoption through curated spaces and additional orchestration.
Powering clinical research innovation: How TriNetX uses Databricks to accelerate drug development
TriNetX operates a federated real-world health data network spanning more than 230 healthcare organizations and needed infrastructure to make growing volumes of RWD easier to use for clinical research. Databricks now provides the centralized lakehouse architecture, consolidating electronic health record data while supporting custom datasets, consulting, ML models, proprietary algorithms, APIs and self-service access. TriNetX is extending this foundation with a beta Query Assistant for natural-language analysis and a Support Assistant prototype built with Databricks’ Agent Bricks, requiring no programming expertise for the former. In 2025, pharmaceutical clients reduced protocol amendments by up to 50%, while an AI-enhanced site-identification effort achieved a 63% site acceptance rate and nine-day average response time. Other models suggest enrollment conversion could rise from 33% to 85%, with expanded APIs and Genie exploration planned for 2026.
What is AIOps?
AIOps applies AI and machine learning to IT operations to analyze logs, metrics, traces, events, alerts, and related system data. It detects anomalies, correlates events, uses topology and dependencies to identify likely root causes, and supports guided or automated responses. The guide positions AIOps between observability and action, reducing noise and adding context without replacing observability, DevOps, or human judgment. Its workflow includes data ingestion, normalization, anomaly detection, event correlation, root-cause analysis, automation and orchestration, and collaboration workflows, with human-in-the-loop review recommended for higher-risk production changes. It contrasts domain-centric and domain-agnostic approaches and recommends evaluating coverage, depth, integrations, setup effort, and organizational fit before expanding automation.
The Web Search Your Agent Inherited Isn't Good Enough
An agent that keeps a company’s market view current must combine Databricks-hosted internal account records with external web signals such as funding rounds, leadership changes, product launches, and hiring surges. The initial implementation duplicates enrichment logic across Claude Code, Codex, and a direct model API, with inconsistent search results and no shared cost, governance, or audit trail. Omnigent consolidates the agent definition, model, tools, policies, limits, and model-call governance above those harnesses, while letting the engineer select a common search provider. Nimble’s Search API supplies live web data, while its Web Search Agents orchestrate multi-source retrieval and extraction, cross-check sources, return citations, and adapt to difficult sites and recurring retrieval paths. In Nimble’s testing, adding its search increased LLM benchmark accuracy from 46% to 71% and halved web search costs; the proposed Databricks setup then connects internal records and external context under one governance surface.
A multi agent approach audience intelligence
Advertisers often lose strategic intent when campaign briefs become data queries, while planners and data teams also miss insights hidden across large customer datasets. An AI-powered audience generation solution on the Databricks Data + AI Platform addresses this gap by combining Genie, custom tool-calling agents, Agent Bricks, Unity Catalog, and a Databricks App. Planners describe audiences in natural language; Genie produces governed SQL segments, an Affinity Agent calculates statistically validated lift against the full population, and a Supervisor Agent routes or chains these capabilities. The resulting workflow saves audiences in Unity Catalog for auditability and downstream activation, and is described as compressing planning and iteration cycles from days or weeks to minutes while enabling faster response to market shifts and pacing issues.
"Regex for Rows": Simplifying Pattern Detection in SQL with MATCH_RECOGNIZE
SQL's unordered-set model makes it difficult to detect ordered event sequences such as consecutive login failures followed by a success, especially when timing matters. MATCH_RECOGNIZE, now available in Databricks compute including Lakehouse Real-Time, expresses row sequences through regex-like states while supporting time anchoring and row-relative checks. The post shows its use for credential-stuffing anomalies, V-shaped stock reversals, checkout abandonment, and equipment-failure signals: FIRST can anchor a failure window, the end-of-partition marker $ can represent inactivity, and PREV/NEXT can compare sensor readings. By replacing chains of common table expressions, gaps-and-islands scaffolding, self-joins, and window-function logic, the clause is presented as easier to validate, maintain, and update for event-sequence analytics.
How energy teams turn theft detection into governed action with Genie and AI business processes
Energy theft detection often stops at an ML risk score, leaving revenue-protection teams to manually interpret cases, prepare investigations, dispatch crews, recover losses, and report results. The described Databricks workflow turns that score into a governed operational loop: Lakeflow prepares features, models run through Model Serving and Unity Gateway, and a Databricks App uses Lakebase to store live case state. AI-generated summaries explain why accounts were flagged, while dispatch-ready reports provide next steps, evidence checklists, and safety notes. Genie One answers leadership questions using metric definitions in Unity Catalog, and an Agent Bricks Multi-Agent Supervisor can assemble a traceable month-end board pack. Unity Catalog lineage, PII labels, access controls, model guardrails, and Unity Gateway routing support regulated use under frameworks including Ofgem’s and ELEXON’s requirements, while the same pattern is presented as applicable to other ML-driven interventions.
How Databricks’ marketers use data 3x more with Genie, an AI analytics assistant
Databricks marketers struggled to get trusted answers from fragmented marketing data quickly, prompting the creation of Marge, a marketing implementation of Genie Agents. The team first unified campaign, sales, web, event and advertising information in a governed Marketing Lakehouse, then grounded the assistant in business definitions, verified logic and example queries. Marge translates natural-language questions into analytical queries, uses Unity Catalog for lineage and role-based access, and can ask clarifying questions or perform multi-step analysis. The reported results include threefold greater data use in decisions, more than 85% adoption, 50% quarter-over-quarter growth, over 5,000 answered questions and a 25% reduction in flagged incorrect responses. Databricks attributes the rollout to continuous feedback, benchmark evaluation, focused use cases and embedding Genie in existing workflows rather than treating conversational analytics as a one-time deployment.
Managed Postgres: What Lakebase Actually Takes Off Your Plate
Managed Postgres varies from OS patching alone to operating scaling, failover, backups, recovery, and related database tooling. Lakebase Postgres applies this model to serverless infrastructure, offering automatic scaling, scale-to-zero, PostgreSQL compatibility, automatic snapshots, point-in-time recovery, branching, built-in PgBouncer, and support for extensions including pgvector and PostGIS. Within a region, patching, scaling, failover, and backups run automatically, while encryption, access control, audit visibility, connection pooling, and lakehouse integrations are also addressed. The discussion also stresses checking PostgreSQL versions, configuration, extensions, and migration methods before moving an existing database. Cross-region disaster recovery remains the main exception: it is in Private Preview for AWS, with periodic replication, manual failover, and customer-managed recovery procedures.
Announcing On-Demand State Repartitioning for Apache Spark™ Structured Streaming on Databricks
Stateful Apache Spark Structured Streaming queries historically locked their state-store partition count into the checkpoint, so changing spark.sql.shuffle.partitions after a workload grew had no effect without rebuilding and losing accumulated state. Databricks Runtime 18 and later introduces on-demand state repartitioning, a Public Preview feature that preserves the checkpoint while moving state to a new partition layout. Operators set spark.sql.streaming.stateStore.partitions, stop the query, and restart it; the query finishes any pending microbatch, physically rehashes and redistributes state, then resumes with the requested count. The feature applies to aggregations, stream-stream joins, deduplication, sessionization, and transformWithState, provided the RocksDB state store is used. Repartitioning pauses processing and its duration is reported in controlBatch.REPARTITION; Coveo reports a 40% reduction in related Amazon S3 API costs.
Health Plans: Your BI Tells You MLR Moved. Can Your AI Tell You Why?
Health plan finance teams can see when medical loss ratio (MLR) moves above budget, but the variance alone does not explain which business areas, costs, populations, or providers caused it. Conversational AI is presented as a way for finance leaders to ask follow-up questions across claims, membership, provider, contract, clinical, quality, risk, and financial data without repeatedly commissioning reports. Its usefulness depends on explicit payer context: MLR combines premiums with medical-care components such as pharmacy rebates, IBNR, recoveries, risk adjustment transfers, and provider settlements, while cohort definitions and metric logic vary. Abacus supplies normalized, connected healthcare data and payer-specific business context; Databricks provides governed enterprise data and AI capabilities for conversational analytical experiences. Together, they are described as creating a reusable foundation for tracing executive signals into operational questions across MLR and other payer domains, while leaving action judgment to the organization.
Unify your marketing data with Lakeflow Connect
Lakeflow Connect addresses the fragmentation of Customer 360 data across ad platforms, analytics tools, CRMs, and support systems. Its fully managed native connectors ingest sources through a UI or API, incrementally landing governed tables in Unity Catalog without infrastructure to provision, manage, or scale. Connectors for Google Ads, Meta Ads, TikTok Ads, Reddit Ads, and LinkedIn Ads provide pre-built or custom reports with spend, impressions, clicks, and conversions, while OAuth flows simplify setup for some platforms. The post also describes HubSpot, Salesforce, and other journey-wide connectors, plus the Ad-Genie accelerator, which transforms bronze data into silver and gold tables for AI/BI dashboards and Genie agents. Customer examples report reduced engineering overhead and faster ingestion.
Unifying governance across engines and catalogs in the Open Lakehouse
Apache Iceberg’s REST Catalog additions—read restrictions and catalog labels—address different cross-platform governance problems: delegating enforcement to trusted engines and carrying governance context across federated catalogs. Read restrictions let a catalog evaluate policy for a principal and return bounded column-projection actions and row-filter expressions that a trusted engine must enforce, while leaving trust establishment outside the protocol. Catalog labels exchange opaque key-value metadata at table and column level, allowing a consuming catalog to map classifications locally, preserve native policy expressiveness, and maintain its own audit trail. Together with centralized enforcement through Cross-Engine ABAC, they provide distinct options for direct access and catalog federation, although read restrictions cannot express policies outside their standardized vocabulary.
Improving Lakebase Postgres compute cache
Lakebase Postgres uses disaggregated storage backed by object storage, so compute-side caching must deliver low latency and high throughput while avoiding inefficient memory use. Its original design paired a conservatively sized PostgreSQL shared buffer with a local file cache (LFC), but the roughly 1 GB shared-buffer cap pushed many hits to slower NVMe storage and could create double buffering. For fixed-size computes with CU >= 80, the new configuration disables the LFC and sets shared buffers to 75% of DRAM, using explicit 2 MB HugeTLB pages to control page-table and TLB overhead. Benchmark tests found huge pages reduced tail read latency by up to ~40% and CPU use by up to ~30%, while production examples reported doubled throughput, 5× fewer storage reads, or 5× lower CPU use. Work is underway to dynamically resize shared buffers and huge pages for autoscaling computes.
Five AI Questions We're Hearing from Financial Services Leaders
Financial services leaders are shifting from asking whether AI works to asking whether it can earn trust, amid governance and audit concerns. The post frames five operational questions around moving finance from reporting to action, using AI in financial crime without losing control, supporting bankers and investment teams, using real-time data, and managing expanding costs. Examples include governed GL-to-Risk reconciliation with cited evidence, human maker-checker oversight and an audit trail; AML agents that draft SARs while humans file; and multi-agent workflows for client and portfolio analysis. The post also cites Coinbase's sub-100ms P99 fraud-detection benchmark and warns that token costs manageable in pilots may grow in production, making model routing and spend visibility relevant. Databricks presents these topics through live demos, talks and meetings at Sibos 2026 in Miami, scheduled for September 28–October 1, including a session on governing AI in financial-crime operations.
A practical approach to end-to-end Solvency II reporting in Databricks
Solvency II reporting connects data, actuarial reserving, capital calculation, quantitative reporting templates (QRTs), the ORSA, approvals, and disclosure, but fragmented systems make readiness and cross-process questions difficult to manage. The described Databricks implementation provides a governed workflow and control tower around existing actuarial and capital suites, automating ingestion checks, data-quality review, reconciliation, orchestration, reporting, and audit trails. It uses MLflow-managed reserving models alongside engines such as Prophet, RAFM, and Igloo, while narrow-scope AI agents can investigate QRT mismatches, suggest remediation, route questions, and support human review. An LLM can draft an ORSA from current numbers, while scenario analysis assesses doubling a cyber book over 12 months; reviewers retain decision authority. The workflow is presented as easier to monitor and investigate.
Beyond embedding: How to secure AI/BI Dashboards for every viewer
Embedding a Databricks AI/BI Dashboard in a customer-facing application is straightforward; authorization determines which rows each viewer may see. The pattern supports external partners and internal Databricks teams with one dashboard and shared dataset, while applying different regional access and masking rules. An entitlements table stores partner IDs and IdP-synchronized group names alongside regions and PII-mask flags; a secured view filters on the signed __aibi_external_value from the embed token. Direct SQL access is separately protected by Unity Catalog row filters and column masks, which evaluate the querying user's identity and groups. The design provides a default-deny, pooled logical multi-tenancy model that centralizes rules while covering both application and workspace query paths.
Introducing Consort: Test-driven development on a branching database
Consort is introduced as an open-source agentic development framework for test-driven development on branching Lakebase Postgres databases. It addresses the limitations of mocks, shared staging databases, and risky schema-change ceremonies by bringing integration testing into the inner loop. Its build lane runs red/green/refactor cycles against isolated copy-on-write branches of real data, while versioned migrations keep code and schema changes together and pull-request checks test migrations against fresh branches. Consort organizes work through role-based agents and a deterministic conductor, preserving feature artifacts in dedicated directories and supplying scoped context packages. A VS Code plugin pairs Git and Lakebase branches, while an observability dashboard exposes agent turns and human approval gates; the project is community-supported rather than a bundled platform feature with an SLA.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.