---
title: "Databricks"
description: "161 posts about Databricks, summarised, each linking to the original."
---

# Databricks
> 161 posts about Databricks, summarised, each linking to the original.

## Articles

### [Personalizing Genie Code with instructions, skills, memory, and MCP](https://yomu.fyi/post/personalizing-genie-code-with-instructions-skills-memory-and-mcp.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Samantha Banchik, Gal Oshri, Romain Rigaux, Will Tipton, Chloe Chan
- Published: Jun 1, 2026

Genie Code adds personalization through persistent instructions, task-specific Agent Skills, shared repository conventions, and MCP servers. Instructions apply across every agent-mode session, while AGENTS.md and CLAUDE.md files can be discovered automatically for team conventions. Skills are markdown-based packages with names, descriptions, guidance, reusable code, or scripts, scoped to workflows and available personally or through Workspace/.assistant/skills/. MCP servers expose external tools, data, and workflows from managed services, Unity Catalog-connected servers, or Databricks Apps, with workspace admin controls and permission-based user selection. Managed OAuth flows for Google Drive, SharePoint, and GitHub are currently in beta, while the MCP marketplace and Genie Code settings support server discovery and setup.


### [AI Doesn't Scale Until You Stop Calling It Innovation](https://yomu.fyi/post/ai-doesn-t-scale-until-you-stop-calling-it-innovation.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Aly McGue
- Published: May 29, 2026

Enterprises often stall between AI proofs of concept and production, and Philippe Rambach argues the remedy is to manage AI as a product rather than innovation. At Schneider Electric, Rambach built a 400-person AI organization split evenly between customer-facing products and internal operations, with business cases owned by lines of business and cross-functional scrum teams responsible through production and support. It standardizes a single core technology set, with Databricks managing infrastructure, data, and data flows, while gate reviews and quarterly portfolio decisions test technical readiness, commercial viability, and the business plan. Models are combined with context, guardrails, interfaces, forecasting, optimization, and real-time decisions; Microgrid Advisor reports up to a 20 percent reduction in energy costs, while Genie’s internal rollout remains early and accuracy is still being addressed.


### [Scaling for MHHS: 50x cost-efficient margin data engineering at Octopus Energy](https://yomu.fyi/post/scaling-for-mhhs-50x-cost-efficient-margin-data-engineering-at-octopus.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Saad Ali, David Poulet, Daniel Taylor, Ismail Makhlouf
- Published: May 23, 2026

Octopus Energy’s move to Market-wide Half-Hourly Settlement (MHHS) increases settlement data volume 48x for its more than 8 million customers, exposing the cost limits of a monthly, single-grain margin pipeline. The rebuild separates margin processing into half-hourly settlement cost, half-hourly revenue for smart tariffs, and daily revenue for standard tariffs, all backed by a unified multi-grain consumption layer. A “Job of Jobs” Databricks workflow coordinates the streams, while Change Data Feed where available and microbatch dbt elsewhere enable incremental processing; Spark work uses pruning, broadcast joins, partition tuning, and Adaptive Query Execution. Rows processed per run fell from 25 billion to 300 million, and freshness improved from weekly to daily. Projected cost per settlement date fell from $23.63 to $0.48, with approximately $1 million in annualised cost avoidance, excluding upstream savings.


### [Accelerating LLM Inference with Prompt Caching for Open‑Source Models on Databricks](https://yomu.fyi/post/accelerating-llm-inference-with-prompt-caching-for-open-source-models.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Pei-Lun Liao, Asfandyar Qureshi, Roshan Regula, Bruce Fontaine, James Thomas, Chenyang Yu
- Published: May 22, 2026

Repeatedly processing identical system or instruction prompts for LLM requests wastes compute, increases latency and costs, and limits throughput. Databricks extends its built-in prompt-caching capability from proprietary models to open-weights models used through Foundation Model APIs, including batch inference, pay-per-token, provisioned throughput, and higher-level services such as Agent Bricks, Genie, and AI Functions. Supported models include GPT-OSS 20B and 120B, Gemma 3 12B, fine-tuned Llama 3.1 8B via PEFT serving, and Llama 3.1 8B and 3.3 70B. By implicitly reusing KV caches for identical prompts, the system skips prefill on cache hits; caches are isolated, held only in volatile memory, and never persisted. In a production GPT-OSS batch-inference pipeline, per-replica input-token throughput rose 2.5x and P50 latency fell 3x despite a 30% cache hit ratio, with no customer configuration required.


### [Observability for any agent, anywhere: Production-ready tracing with OpenTelemetry & Unity Catalog on Databricks](https://yomu.fyi/post/observability-for-any-agent-anywhere-production-ready-tracing-with-ope.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Firas Farah, Bruno Faria, Anoop Sunke
- Published: May 22, 2026

Databricks supports writing OpenTelemetry (OTel) traces directly to Unity Catalog, where real-time telemetry is stored in Delta tables for governed analytics and retention. AI traces capture prompts, tool calls, responses, latency, and execution paths, enabling debugging, evaluation, and monitoring, while lakehouse storage also allows SQL queries, dashboards, joins with business data, and PII controls. The managed, serverless ingestion layer uses Zerobus Ingest to accept OTLP over gRPC from collectors and REST integrations, streaming spans, logs, and metrics to Unity Catalog without intermediate message buses. A LangGraph support manager assistant demonstrates instrumentation with mlflow.langchain.autolog() and an @MLflow.trace root span, while Genie is invoked through MCP for data-driven questions. The resulting traces can be searched in MLflow, evaluated at scale, and monitored continuously, with FAQ details stating 200 QPS starting throughput, no storage limit, and Unity Catalog governance options for access control, masking, and filtering.


### [How World Bank Group uses Databricks to eradicate poverty through shared knowledge](https://yomu.fyi/post/how-world-bank-group-uses-databricks-to-eradicate-poverty-through-shar.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Kacey Hertan
- Published: May 22, 2026

The World Bank Group is building a unified data and AI platform on Databricks to make tens of millions of repository documents and structured operational data findable for teams pursuing shared prosperity. Legacy databases and disconnected document repositories forced researchers to search manually, slowing decisions and limiting the ability to surface lessons learned across its global portfolio. The implementation migrates operational data, applies Unity Catalog governance, uses Databricks Volumes for unstructured content, Genie for natural-language structured queries, a metrics layer for deterministic reporting, and vector search for retrieval-augmented generation over project documents. Because each Genie serves a specific metrics domain, an agentic layer combines intent and domain classification with query decomposition, routing requests among domain agents, RAG, and visualization components before assembling one response; a multi-region feedback prototype was deployed in approximately two and a half days.


### [Using observability data to prevent incidents](https://yomu.fyi/post/using-observability-data-to-prevent-incidents.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Madelyn Mullen
- Published: May 22, 2026

Engineering teams often remain reactive not because they lack telemetry, but because metrics, logs, traces, error budgets, and SLO burn rates are difficult to query at decision speed. The post frames reliability intelligence as trend analysis that identifies accumulating risk before a user-facing incident, contrasting it with observability focused on current system state. It proposes unified telemetry access and natural-language self-service, using Databricks Genie to examine p99 latency, error-budget burn, deployment frequency, incident rates, dependencies, and capacity across services. The Trade Desk’s Genie Router handles questions across more than 10,000 tables, while the post says ad-hoc analysis runs five times faster with Genie and can support mitigation before a page fires.


### [How Databricks Genie democratizes data access in financial services](https://yomu.fyi/post/how-databricks-genie-democratizes-data-access-in-financial-services.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Kim Hatton
- Published: May 22, 2026

Financial services organizations have built sophisticated lakehouses, streaming pipelines, model-serving infrastructure, and self-service BI, but access remains concentrated among technical teams. Business leaders still often rely on analysts because they may lack SQL skills, BI training, or analyst access, creating the “last mile” of data democratization. Databricks Genie addresses this gap through a conversational AI interface that converts plain-English questions into governed SQL queries executed against the Databricks Lakehouse without an analyst in the loop. It operates within Unity Catalog access policies, restricts users to authorized data, makes queries read-only, and logs interactions for audit purposes, while its semantic layer maps organizational terminology such as NIM, LTV, and NII to the organization’s meanings. The stated outcome is faster, auditable answers for business questions and usage data that can inform data-product priorities.


### [How Databricks Genie improves retail personalization](https://yomu.fyi/post/how-databricks-genie-improves-retail-personalization.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Sarah Duffy
- Published: May 20, 2026

Retail personalization often stalls when CX leaders must wait for analysts to answer questions about segment behavior, loyalty performance, channel preference, or churn risk, narrowing the window for action. Databricks Genie addresses this access gap by letting business users query unified structured and unstructured enterprise data in plain English instead of SQL. Its retail differentiators include identity-resolved queries across channels and devices, lifecycle-stage awareness, campaign response and control-group data integration, and privacy controls enforced through Unity Catalog. The source says Genie reduces routine analyst requests rather than replacing data science teams, while enabling merchandisers, category managers, loyalty marketers, and CX leaders to self-serve operational questions. It cites 7-Eleven’s use of Databricks SQL, Unity Catalog, and AI/BI Genie to launch, refine, and measure personalized offers within a secure, unified platform.


### [Databricks for Good and Virtue Foundation: Partnering to Connect Medical Volunteers to Critical Health Services in 72 Countries](https://yomu.fyi/post/databricks-for-good-and-virtue-foundation-partnering-to-connect-medica.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Priyanka Mehta, Shaunak Sen
- Published: May 20, 2026

Virtue Foundation’s VF Match platform connects medical professionals with volunteer opportunities across 72 low and low-middle income countries, but scaling its initial LLM proof of concept into production required a robust data pipeline. The Databricks-based Foundational Data Refresh combines Overture Maps and Bright Data, processes more than 25 million web pages with targeted GPT extraction steps, and uses Spark, Photon, and Lakeflow Jobs to orchestrate checkpointed, modular workloads. Splink resolves duplicate facilities and nonprofits across messy sources; Photon reduced the slowest entity-resolution partition from 30 minutes to approximately two minutes, a 15x improvement. A prototype VF Agent adds natural-language access through LangGraph, Databricks Model Serving, AI Search, and Genie, while the resulting system provides an up-to-date global view of healthcare infrastructure and volunteer needs.


### [A CFO’s guide to managing value-based care financial performance](https://yomu.fyi/post/a-cfo-s-guide-to-managing-value-based-care-financial-performance.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Adam Crown
- Published: May 19, 2026

Value-based care changes healthcare finance by tying payment to outcomes, cost efficiency, and equity rather than service volume, while shifting greater financial risk to providers under ACO, bundled, and capitated contracts. A CFO must therefore track attributed populations, PMPM cost trends, utilization, clinical outliers, quality thresholds, and contract performance against benchmarks. Databricks Genie is presented as a conversational interface over integrated attribution, claims, clinical, and benchmark data, allowing questions such as PMPM divergence among high utilizers and DRG drivers to be answered in real time. The source also identifies incorrect attribution and risk-adjustment coding errors as major threats, and describes predictive risk stratification and quality-gap forecasting as ways AI can support earlier intervention.


### [How Deutsche Börse built a generative AI tool to tackle the large-scale migration of   Zeppelin notebooks to Databricks](https://yomu.fyi/post/how-deutsche-bo-rse-built-a-generative-ai-tool-to-tackle-the-large-sca.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Evan Pandya, Tobi Wole-Fasanya
- Published: May 19, 2026

Deutsche Börse Group is migrating a high volume of Zeppelin notebooks used by more than 2,000 users from Cloudera to Databricks, after Cloudera’s 2027 Zeppelin decommissioning and the group’s move to cloud analytics. Because the notebooks combine SQL, Python, custom interpreters, data-source references, visualizations, widgets, and scheduling logic, the team rejected fully rule-based rewriting. Its Zeppelin to Databricks Notebook Converter, delivered as a Databricks App, deterministically maps paragraphs to cells, translates interpreter syntax, reformats metadata as .ipynb JSON, and preserves original content. A generated, environment-specific prompt then gives Genie context about interpreters, data sources, and configuration patterns so it can reconstruct logic, ask clarifying questions, and handle the variable work. The hybrid workflow reduced redevelopment to 15–20 minutes per notebook, while keeping SQL, Python, visualizations, widgets, Oracle and HDFS references, scheduling, and custom business logic untouched; large-scale testing is next.


### [Announcing the Databricks analytics engineer learning pathway](https://yomu.fyi/post/announcing-the-databricks-analytics-engineer-learning-pathway.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Maroua Lazzarou, Pratyarth Rao
- Published: May 18, 2026

Databricks is launching the Analytics Engineer Learning Pathway, a curriculum for SQL practitioners that covers data modeling, pipelines, metrics, and Genie spaces on the lakehouse. The pathway teaches learners to transform raw data into governed, AI-ready semantic models and metric views, the foundation for analytics, dashboards, and AI agents. Courses cover Analytics Fundamentals, production SQL ETL with Materialized Views, Streaming Tables, AUTO CDC, and Lakeflow Jobs, plus data modeling with Delta Lake and Unity Catalog. The curriculum also addresses metric views, Genie spaces, governance with Unity Catalog permissions and ABAC policies, and Spark Declarative Pipelines with expectations, event logs, and metrics. Courses are available in self-paced and instructor-led formats through Databricks Academy, with the full pathway included in active learning subscriptions.


### [PipelineIQ: Forward‑Looking Sales Intelligence That Drives Action](https://yomu.fyi/post/pipelineiq-forward-looking-sales-intelligence-that-drives-action.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Sam Le Corre, Dael Williamson, Luis Herrera
- Published: May 15, 2026

PipelineIQ addresses the administrative drain and unreliable revenue predictability caused by incomplete, inconsistent, and backward-looking CRM data in B2B sales. Rather than build another forecasting system, it applies prescriptive analytics to identify forward signals and produce immediate actions for reps and managers. Built on Databricks, it uses Foundation Model APIs, Unity Catalog, Delta Lake, and AI/BI Dashboards; its confidence scorer sends CRM fields to ai\_query() with a Gemma 3 12B model, scores eight MEDDPICC dimensions from 0–10, and limits missing fields to scores of 3 or below. Weighted confidence is refreshed daily, with a fail-safe override to Low when a use case has more than three active blockers. Dashboards and Genie queries connect evidence-based risk explanations, remediation steps, and portfolio views to sales execution.


### [The Rosetta stone of CPS: Claroty’s AI-powered library](https://yomu.fyi/post/the-rosetta-stone-of-cps-claroty-s-ai-powered-library.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ben Hazan, Anton Berlinsky, Ohad Avni, Itay Wagner, Guy Zalcman, Dor Bdolach, Ravid Ariely, Gal Sberro
- Published: May 13, 2026

CPS asset identification is difficult because 88% of CPS assets do not transmit an exact product code and 76% use product codes that differ from the vendor's official records. Claroty's AI-Powered CPS Library addresses this entity-resolution problem with a hybrid architecture that combines classic matching, statistical inference, generative AI, NLP, reasoning agents, and human-in-the-loop review to reconcile protocol data, vendor records, manuals, and vulnerability information. Its Databricks-based lakehouse uses Delta Lake, Unity Catalog, Spark pipelines, Model Serving, MLflow evaluations, Lakebase, and Databricks Apps to maintain governed, auditable mappings across a catalog of more than 17 million assets. The post reports 25% improved vulnerability-attribution accuracy and says early tests gave 56% of analyzed devices new or updated recommendations for outdated firmware, while low-confidence mappings remain subject to expert review.


### [Clinical operations intelligence belongs on the Lakehouse](https://yomu.fyi/post/clinical-operations-intelligence-belongs-on-the-lakehouse.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Nicholas Siebenlist, Surya Sai Turaga
- Published: May 13, 2026

The Site Feasibility Workbench addresses fragmented clinical operations data by placing a site-selection decision-support application, its models, and data on Databricks. Released as a fully open-source Databricks App, it combines a FastAPI backend, React frontend, Unity Catalog-governed data, Lakebase operational state, and AI/BI Genie within the workspace. TA-segmented LightGBM models use the organization’s CTMS, EDC, and IRT history alongside real-world evidence, patient access, qualification, Open Payments, and protocol-execution signals to rank sites. Each prediction stores SHAP attributions in a governed Unity Catalog Delta table, while MLflow tracks training runs and model versions; saved shortlists persist to Lakebase and Genie answers governed-data questions. The six-step workflow supports protocol selection through final shortlisting, including diversity considerations; the release is a decision-support layer rather than a source of record, and deployment reportedly takes about 30 minutes before security review and validation.


### [How CFOs in consulting can recover margin with Databricks](https://yomu.fyi/post/how-cfos-in-consulting-can-recover-margin-with-databricks.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Rajaram Suresh, Kateryna Savchyn, Pavithra Rao, Mark Hahn, Kim Hatton, Sharmey Shah, Lisa Fox
- Published: May 12, 2026

Consulting firms are facing margin pressure despite growing deal pipelines, with 2024 EBITDA margins at 9.8%, utilization at 68.9%, and revenue growth at 4.6%. The post attributes this to fragmented finance workflows spanning Salesforce, SAP, Workday, Concur, and spreadsheets, plus inconsistent metrics and manual reconciliation. It proposes using Databricks to create a governed data layer with Lakeflow, Lakehouse, and Unity Catalog, then adding Genie for natural-language analysis and finance apps that trigger workflows and write approvals back in real time. A global consulting firm reportedly cut cash forecasting cycles by 3–5 days and reduced reporting-cycle FTE hours by 80%, while Databricks’ own close fell from 15 days to 8. The proposed rollout spans 10–12 weeks and begins with one tightly scoped workflow such as revenue close, unbilled WIP, T&E leakage, or practice-margin analysis.


### [Announcing Databricks student fellows](https://yomu.fyi/post/announcing-databricks-student-fellows.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Trang Le, Elise Hollowed, Joe Nash
- Published: May 12, 2026

Databricks is launching the Student Fellows Program, an opportunity for university and college students interested in computer science, AI, and data engineering to become leaders in their campus communities. Fellows receive specialized training from Databricks experts, build hands-on skills with the Databricks platform, and organize campus activities such as hackathons, tech talks, and study groups while connecting peers with resources. The initiative also offers opportunities for free or discounted certification exam vouchers and practical experience intended to support applications for internships with Databricks, its customers, or partners. Student Fellows’ primary mission is to foster a community of learners and serve as a bridge between Databricks and their university communities, with the first cohort currently accepting applications.


### [Faster Queries and New Capabilities with the Open-Source Databricks JDBC Driver](https://yomu.fyi/post/faster-queries-and-new-capabilities-with-the-open-source-databricks-jd.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Toussaint Webb, Gopal Lal, Kaitlin Baumgardner
- Published: May 12, 2026

The open-source Databricks JDBC driver 3.x and later releases address connectivity needs affecting dashboards, spreadsheets, applications, and other data workflows. Compared with the legacy 2.x driver, they deliver up to 30% faster retrieval of large query results while preserving backward compatibility. Architectural changes add full Arrow-based transfer for JDK 16+, a first-class asynchronous execution API, stream-based ingestion into Databricks Volumes, and integration with the Statement Execution API. The driver also supports stored procedures, multi-statement transactions, Unity Catalog metric views, query tags, geospatial and complex data types, plus built-in client telemetry for query latency, metrics, and errors. Its open-source ownership and Databricks maintenance are presented as enabling faster fixes, new features, transparency, contributions, and tighter platform integration.


### [Unlocking the Archives: Turning Unstructured Documents into a Searchable Database for Groundwater Discovery](https://yomu.fyi/post/unlocking-the-archives-turning-unstructured-documents-into-a-searchabl.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Andres David Blandon Restrepo, Mofeed Nagib
- Published: May 11, 2026

MapAid and the Sudan Association for Archiving Knowledge (SUDAAK) needed to make nearly 700 scanned geological reports searchable and recover well data for MapAid’s WellMapr groundwater models. Databricks built a Databricks-only pipeline that renders pages as images, uses multimodal AI Functions for page classification and schema-constrained JSON, and applies targeted sampling to cut classification processing by more than 70%. Water-relevant documents receive page-by-page OCR through the Foundation Model API, with well and borehole identifiers linking information across pages before structured records capture locations, depths, static water levels, and yields. In its first full run, 654 documents and 5,570 pages were classified in under three hours; 95% received excellent or good evaluations, about half contained water data, and 299 well or borehole records were extracted.


[Newer posts](https://yomu.fyi/topic/databricks/page/5.md) · [Older posts](https://yomu.fyi/topic/databricks/page/7.md)