---
title: "Databricks"
description: "Data and AI platform for data engineering, analytics, machine learning, and generative AI."
---

# Databricks
> Data and AI platform for data engineering, analytics, machine learning, and generative AI.

## Articles

### [Unlocking the Archives: Turning Unstructured Documents into a Searchable Database for Groundwater Discovery](https://yomu.fyi/post/unlocking-the-archives-turning-unstructured-documents-into-a-searchabl.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Andres David Blandon Restrepo, Mofeed Nagib
- Published: May 11, 2026
- Type: Problem & solution

MapAid and the Sudan Association for Archiving Knowledge (SUDAAK) needed to make nearly 700 scanned geological reports searchable and recover well data for MapAid’s WellMapr groundwater models. Databricks built a Databricks-only pipeline that renders pages as images, uses multimodal AI Functions for page classification and schema-constrained JSON, and applies targeted sampling to cut classification processing by more than 70%. Water-relevant documents receive page-by-page OCR through the Foundation Model API, with well and borehole identifiers linking information across pages before structured records capture locations, depths, static water levels, and yields. In its first full run, 654 documents and 5,570 pages were classified in under three hours; 95% received excellent or good evaluations, about half contained water data, and 299 well or borehole records were extracted.


### [Predictive quality starts where defect detection stops](https://yomu.fyi/post/predictive-quality-starts-where-defect-detection-stops.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Caitlin Gordon
- Published: May 11, 2026
- Type: Explainer

Manufacturing quality teams often receive defect-rate reports after the conditions behind them have changed, because inspection, supplier, and environmental data are disconnected. Predictive quality combines production, inspection, and supplier data with machine learning to forecast defects before final inspection, shifting quality management from reactive documentation to proactive intervention. Databricks Genie is presented as a natural-language interface for querying those sources together, including questions about first-pass yield, supplier lots, root-cause contributors, and process conditions; its answers include citations and can surface unusual patterns. The described capabilities include contextual understanding of terms such as NCR, CAPA, and CPK threshold, multi-source reasoning, and traceable outputs tied to records. The intended outcome is faster analysis and earlier action to reduce scrap before its cost is incurred.


### [Retail markdown optimization: from reactive markdowns to proactive](https://yomu.fyi/post/retail-markdown-optimization-from-reactive-markdowns-to-proactive.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Sarah Duffy
- Published: May 11, 2026
- Type: Problem & solution

Retail markdown optimization addresses the gap between changing demand signals and delayed merchandising action. The post defines it as strategically reducing prices on slow-moving or end-of-life inventory, using demand forecasts, sell-through rates, weeks of supply, and price-elasticity models to choose which SKUs to mark down, when to start, how deep to discount, and where to act. It presents Databricks Genie as a natural-language interface across e-commerce, store, and wholesale data, with supplier, margin, and historical-pattern context; one example is Coop’s Microsoft Teams assistant, AskCap, which reported a 30% retention rate among internal users. Earlier detection of sell-through deceleration can give merchants more time to protect margin, adjust open-to-buy, and reallocate capital, while Genie supports decisions rather than making them.


### [Using MemAlign to Improve Evaluation of Traditional Machine Learning in Genie Code](https://yomu.fyi/post/using-memalign-to-improve-evaluation-of-traditional-machine-learning-i.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Stepan Nosov, Pavle Martinović, Tejas Sundaresan, Alkis Polyzotis, Nemanja Petrovic
- Published: May 8, 2026
- Type: Benchmark

Genie Code’s traditional machine learning workflows are difficult to evaluate because notebook quality depends on code, best practices, and adaptation to data and task. The team built an evaluation pipeline with nine LLM judges, each scoring applicable dimensions—including data exploration, imputation, training, MLflow logging, and cell organization—from 1 to 3, using test cases across classification, regression, and forecasting. On 50 human-graded notebooks, initial judge alignment was weak in several areas, including model training (MAE 0.680) and model use (0.562), with positivity bias contributing to missed technical flaws. Applying MLflow’s MemAlign, which combines generalized semantic guidelines with retrieved episodic examples, reduced judge error by 74–89% on the most important dimensions; removing episodic memory weakened most gains, while data exploration showed a nonsignificant regression.


### [How Superhuman and Databricks built a 200K QPS inference platform together](https://yomu.fyi/post/how-superhuman-and-databricks-built-a-200k-qps-inference-platform-toge.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Myke Troianovskyi, Christoph Stüber, Wai Wu, Arjun DCunha, Amine El Helou, Tian Ouyang, Jarek Odzga, Alex Coleman
- Published: May 8, 2026
- Type: Problem & solution

Superhuman replaced a DIY vLLM-based serving stack for its grammar-correction model with Databricks model serving, targeting peak traffic above 200,000 QPS, sub-second P99 latency, four-nines reliability, and no quality regression. The platform combines an Endpoint Discovery Service with power-of-two-choices load balancing, concurrency-based autoscaling, and asymmetric scale policies to reduce hotspots and handle rapid traffic ramps. Lazy-loading container images changed startup from several minutes to a few seconds by fetching metadata first and retrieving 4MB blocks on demand. Runtime work on H100 GPUs raised per-pod throughput from 750 to 1,200 QPS: FP8 weight quantization contributed up to 30%, while a multiprocessing server added 20%, alongside C++ and asynchronous scheduling optimizations. The teams validated changes through shadow testing and evaluation harnesses, and Superhuman is migrating additional models and workflows to the platform.


### [Addressing HR's widening capacity gap with AI](https://yomu.fyi/post/addressing-hr-s-widening-capacity-gap-with-ai.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Paurul Yadav, Soumya Dash, Bryan Smith
- Published: May 8, 2026
- Type: Problem & solution

HR teams face a widening capacity gap as strategic expectations, complex employee issues, workforce volatility, skills shortages, and demands for personalized support collide with largely unchanged headcount and tools. The article presents AI transformation as an incremental journey: first establish a secure Employee 360 from structured and unstructured enterprise data, then build reusable workforce insights, augment workflows with human oversight, and progress toward broader transformation. It emphasizes data governance, including access controls, auditing, quality, standardization, and reliable interpretations, while noting that trust has limited AI’s business impact so far. MathCo and Databricks support this roadmap through NucliOS, whose Data Studio, AI Studio, and Decision Studio environments connect governed data, explainable models, feedback loops, and decision applications; Databricks supplies the lakehouse foundation, lineage, quality checks, and privacy-compliant access.


### [MCP Marketplace brings real-time intelligence to agentic applications](https://yomu.fyi/post/mcp-marketplace-brings-real-time-intelligence-to-agentic-applications.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Roman Ostrovski, Harish Gaur, Antoine Amend
- Published: May 8, 2026
- Type: Announcement

The MCP Marketplace connects agentic applications with real-time external intelligence alongside enterprise data. The problem appears in use cases such as loan approval, where historical internal records omit market conditions, updated credit signals, property changes, and competitor activity, making manual research a bottleneck. Databricks Marketplace provides governed access to MCP servers from You.com, Moody’s, and Cotality, while Unity Catalog authenticates connections and tracks access and lineage; Lakebase stores state, decisions, and audit trails across multi-step workflows. Examples show agents combining internal data with web research, credit ratings and sector outlooks, or property-resolution and mortgage signals before surfacing decisions for human review, including a commercial-loan flow with recorded sources, timestamps, and approver.


### [Pushing the Frontier for Data Agents with Genie](https://yomu.fyi/post/pushing-the-frontier-for-data-agents-with-genie.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: The Databricks AI Research Team
- Published: May 8, 2026
- Type: Benchmark

Genie is Databricks’ data agent for complex questions across structured enterprise assets—tables, dashboards, and notebooks—and unstructured sources including workspace files, Google Drive, and Sharepoint. Unlike coding agents operating in static environments, it must discover relevant assets at enterprise scale, determine authoritative knowledge from potentially contradictory sources, and handle questions without verifiable tests or guaranteed answers. It combines specialized knowledge search using semantic context and metadata, parallel thinking across sampled trajectories, and Multi-LLM orchestration with optimized prompts for distinct sub-agents. On an internal benchmark of real-world data-analysis tasks, these techniques raised accuracy from 32% to over 90% against a leading coding agent while reducing cost and latency; specialized search alone improved table-discovery performance by up to 40%, while parallel thinking added latency and token costs before further optimization.


### [Energy trading analytics in a real-time market](https://yomu.fyi/post/energy-trading-analytics-in-a-real-time-market.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Caitlin Gordon
- Published: May 8, 2026
- Type: Problem & solution

Energy trading markets settle every 15 minutes, while many analytics environments still rely on nightly batch pipelines, creating latency between changing conditions and trading decisions. The post presents Databricks Genie as a conversational interface that lets traders and portfolio managers query actual market data, weather integrations, historical position data, and current books. Its example combines a node’s 90-day natural-gas basis distribution with a Northeast temperature anomaly, while the described system can query real-time and historical data together. Genie also includes settlements, curtailments, and ancillary-service data, with governed access so users see only authorized information. The conclusion is that faster data access can remove information latency and compound margin without replacing trading expertise; the source states that Genie is available today.


### [First-party audience data is the ad sales relationship now](https://yomu.fyi/post/first-party-audience-data-is-the-ad-sales-relationship-now.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Elena Tesser
- Published: May 8, 2026
- Type: Problem & solution

The post argues that media companies face a stronger need to turn first-party audience data into an advertising sales advantage as third-party cookies decline and buyer attribution expectations rise. Advertisers are buying audiences rather than inventory, so sales teams need granular audience insights, validated campaign performance, clear attribution, and revenue context to compete in RFPs. Databricks Genie is presented as a natural-language interface to governed first-party data, allowing leaders to query defined audience segments such as users who completed at least 75% of automotive content in 90 days, segmented by income quartile and household ownership. The same analytical environment connects demographic composition, content and behavioral signals, post-campaign measurement, attribution, and revenue pacing against budget and prior year.


### [Operating room utilization is hiding in your scheduling data](https://yomu.fyi/post/operating-room-utilization-is-hiding-in-your-scheduling-data.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Adam Crown
- Published: May 8, 2026
- Type: Explainer

Operating room utilization measures in-room surgical minutes against allocated block-time minutes, yet most US health systems reportedly run at 65–75% versus an 80% industry target. Daily performance reports arrive the next morning, after schedules are set and opportunities to release unused blocks, redeploy staff, or backfill add-on cases may have passed. The post presents Databricks Genie as a natural-language interface for querying scheduling, utilization, and outcomes data without a data analyst request. Its proposed analytical environment combines scheduling data, actual case logs, block-release records, contribution margin, and staffing costs, with breakdowns by surgeon, service line, facility, and day of week. Genie surfaces specific intervention targets, although the post says it provides data access rather than automating OR management.


### [Why telecom churn prediction misses the intervention window](https://yomu.fyi/post/why-telecom-churn-prediction-misses-the-intervention-window.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Elena Tesser
- Published: May 8, 2026
- Type: Problem & solution

Telecom churn programs often intervene after customers have already shifted behavior, contacted support, or decided to leave, even though earlier signals exist in operational data. The post frames the gap as organizational: propensity models may be sophisticated, but retention leaders need timely, specific answers about high-value customers, likely triggers, and historically effective interventions. Databricks Genie is presented as a natural-language interface over customer behavioral and commercial data, able to surface targets such as premium postpaid customers with usage declines above 20%, recent support contacts, and contracts ending within 90 days. Its described capabilities combine usage, support, billing, network experience, competitive tenure, and intervention history while supporting segment and individual analysis. The proposed operating model prioritizes interventions by customer lifetime value and aims to act early enough for retention efforts to change outcomes.


### [Growth analytics is what comes after growth hacking](https://yomu.fyi/post/growth-analytics-is-what-comes-after-growth-hacking.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Madelyn Mullen
- Published: May 8, 2026
- Type: Explainer

Growth analytics is presented as the next stage after growth hacking, arguing that competitive acquisition now depends on precise unit economics, cohort quality, and rapid conversion optimization rather than isolated tactics. It distinguishes product analytics, which explains feature use and user flows, from growth analytics, which connects acquisition sources and costs with activation, revenue, retention, and cohort behavior. The stated bottleneck is fragmented tooling: answering questions such as 90-day LTV by channel correlated with seven-day activation can require manually joining multiple systems and delay weekly budget decisions. Databricks Genie is described as a governed natural-language interface over unified acquisition, behavioral, and billing data, supporting attribution-to-LTV analysis, payback modeling, and paid-versus-organic comparisons. The text reports a 50% relative acquisition-rate lift, from 8% to 12%, for customers using Genie, and shorter onboarding insight cycles from months to weeks.


### [Real-world evidence for medical affairs: who can actually use it?](https://yomu.fyi/post/real-world-evidence-for-medical-affairs-who-can-actually-use-it.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Adam Crown
- Published: May 8, 2026
- Type: Explainer

Real-World Evidence (RWE) is increasingly sought by payers, providers, and regulators, but the post distinguishes it from abundant Real-World Data (RWD), which must undergo rigorous study design, analysis, and interpretation to become credible evidence. It identifies four Medical Affairs use cases: regulatory submissions and commitments, payer formulary discussions, rapid HCP scientific exchange, and internal pipeline or portfolio decisions. The stated operational problem is that teams with claims, EHR, registry, and other assets often lack the fluency or capacity to answer complex questions within competitive and regulatory timelines. Databricks Genie is presented as a natural-language interface that can query unified RWE assets, including a treatment-initiation and 12-month-persistence example that reportedly surfaces in seconds rather than requiring several days of data-science work. The post also describes governance, logged and attributable requests, treatment-pathway awareness, and access controls for MSL support.


### [Wealth advisor productivity starts with the client conversation](https://yomu.fyi/post/wealth-advisor-productivity-starts-with-the-client-conversation.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Kim Hatton
- Published: May 8, 2026
- Type: Explainer

Wealth management client reviews are intended to turn portfolio information into advisory conversations, but substantial meeting time is spent confirming allocations, benchmark performance, and major positions. Preparing for these meetings requires advisors to combine portfolio, tax, estate, and goal-related information from multiple systems for each client, often under time pressure. Databricks Genie is presented as a conversational way to query client portfolio data in real time, including a tax-loss-harvesting question that combines unrealized losses with an estimated current-year tax rate. Its described capabilities include unified holdings, transaction, tax-lot, and profile data; household-level analysis; authorization controls; and external market context, with the stated aim of reducing information preparation so advisors can conduct deeper, more personalized conversations.


### [How lakebase architecture delivers 5x faster Postgres writes](https://yomu.fyi/post/how-lakebase-architecture-delivers-5x-faster-postgres-writes.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: David Wein, Vlad Lazar
- Published: May 7, 2026
- Type: Benchmark

Lakebase’s separation of Postgres compute and distributed storage removes the local-disk torn-page risk that necessitates full page writes (FPW) in traditional deployments. The change disables FPW on compute and moves full page image generation into the pageserver, which creates reset points after a configured number of WAL deltas while preserving bounded read reconstruction. In HammerDB TPROC-C tests, 32-vCPU throughput rose from 95,686 to 439,300 new orders per minute, while average WAL per transaction fell from 58KB to under 4KB, a 94% reduction. Production validation reported WAL reductions from 30MB/s to 1MB/s on a 56-vCPU project, p99 read latency improvements of 30% to 50%, and up to threefold regional storage p99 improvement. The rollout reached all Lakebase Serverless and Neon databases without restarts or customer interruption, using the existing XLOG\_FPW\_CHANGE mechanism.


### [Why talent transformation is the missing focus of enterprise AI](https://yomu.fyi/post/why-talent-transformation-is-the-missing-focus-of-enterprise-ai.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Christy Seto, Pratyarth Rao
- Published: May 7, 2026
- Type: Announcement

Enterprise AI adoption is presented as a talent problem as much as a technology or strategy problem: insufficient worker skills are identified as the biggest barrier to integrating AI into work, while hiring alone cannot fill demand. The proposed response is continuous upskilling for both technical practitioners and line-of-business users, replacing one-time enablement with learning tied to current work and changing platform capabilities. Databricks Academy Pro is an annual, per-seat subscription combining self-paced courses, weekly live reviews, unlimited public instructor-led classes, hands-on labs in hosted Databricks environments, and one certification exam voucher per user each year. Starter, Growth, and Enterprise tiers add private online community access, a custom Academy portal, and a designated Talent Transformation program manager as seats increase. The offering is available today, alongside private instructor-led training and standalone certification vouchers.


### [Public health intelligence shouldn't require a data scientist](https://yomu.fyi/post/public-health-intelligence-shouldn-t-require-a-data-scientist.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Kacey Hertan
- Published: May 7, 2026
- Type: Problem & solution

State, tribal, local, and territorial (STLT) health agencies manage data across surveillance, vital records, Medicaid, WIC, and emergency preparedness systems, but those systems are fragmented and difficult to query together. That limits rapid public health intelligence: questions linking emergency-department visits with pharmacy dispensing, school absenteeism, vaccination, demographic, or geographic data can require epidemiologists to assemble manual queries over weeks, even when decisions require answers within hours. The post presents Databricks Genie as a natural-language interface for querying this environment, backed by a Databricks engine that handles petabyte-scale datasets across real-time streams and historical records. It describes cross-program synthesis, Unity Catalog row- and column-level access controls, HIPAA-compliant governance, traceability to the underlying query, and validation controlled by health experts. Examples include county-level influenza-like illness trends overlaid with vaccination coverage and identifying counties with high opioid overdose rates and low treatment utilization; Genie is described as available today.


### [Mean time to detect is a data access problem](https://yomu.fyi/post/mean-time-to-detect-is-a-data-access-problem.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Taylor Kain
- Published: May 7, 2026
- Type: Problem & solution

Security operations centers measure MTTD, MTTR, false-positive rates, and analyst utilization, yet investigations often stall because analysts assemble evidence across fragmented systems. A single alert may require separate queries for logs, identity records, asset information, prior alerts, and cross-source timelines, making the analyst the integration layer and creating an MTTI bottleneck. The post presents Lakewatch with Databricks Genie as an agentic interface powered by Anthropic Claude models: analysts ask natural-language questions while autonomous agents hunt, summarize, correlate, and reconstruct timelines across security, IT, and business data. It argues that this architecture can reduce investigation work from manual, multi-system workflows to answers in seconds, while retaining analyst-level access controls and governed data access as exploit time has shrunk to 1.3 days.


### [Rethinking Distributed Systems for Serverless Performance and Reliability](https://yomu.fyi/post/rethinking-distributed-systems-for-serverless-performance-and-reliabil.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Aaron Davidson, Roland Fäustlin, Zach Williams
- Published: May 6, 2026
- Type: Explainer

Building serverless compute for Apache Spark requires more than warm machine pools or basic autoscaling because traditional deployments couple user applications to infrastructure, allowing contention and inefficiencies to undermine performance and reliability. The proposed architecture separates these concerns through Spark Connect’s client-server model over gRPC, a gateway that routes workloads using query size, cluster utilization, and latency profile, and an adaptive autoscaler that adjusts capacity horizontally and vertically. Spark Connect isolates user applications from drivers, while the gateway continually re-evaluates placement to reduce interference between workloads. The autoscaler offers Standard and Performance-Optimized modes and can respond to out-of-memory errors by restarting tasks on larger VMs without manual intervention. Reported outcomes include a 99.998% upgrade success rate across more than 4.5 billion workloads, 2–5x faster Unilever pipelines, and operational-cost reductions of 25%.


[Newer posts](https://yomu.fyi/company/databricks/page/23.md) · [Older posts](https://yomu.fyi/company/databricks/page/25.md)