---
title: "Databricks"
description: "Data and AI platform for data engineering, analytics, machine learning, and generative AI."
---

# Databricks
> Data and AI platform for data engineering, analytics, machine learning, and generative AI.

## Articles

### [Stop hand-coding change data capture pipelines](https://yomu.fyi/post/stop-hand-coding-change-data-capture-pipelines.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Matt Jones, Zoé Durand, Phoebe Weiser, Bilal Aslam, Ray Zhu
- Published: Apr 22, 2026
- Type: Problem & solution

Hand-coded change data capture (CDC) and slowly changing dimension (SCD) pipelines become fragile as they handle out-of-order updates, duplicate events, deletes, late-arriving data, retries, backfills, and schema evolution. The post presents AutoCDC in Lakeflow Spark Declarative Pipelines as a declarative alternative: engineers specify keys, sequencing, delete conditions, and SCD Type 1 or Type 2 semantics while the platform manages ordering, state, deduplication, versioning, and incremental processing. Examples show a current-state table that keeps Mercedes’s latest city and removes deleted user 123, alongside a Type 2 table that preserves validity windows through \_\_START\_AT and \_\_END\_AT. The supplied comparison reports roughly 6–10 lines of declarative definition versus 40–200+ lines of custom logic, and says AutoCDC supports both change data feeds and snapshot-based CDC.


### [Personalization Strategies for Media Companies](https://yomu.fyi/post/personalization-strategies-for-media-companies.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 22, 2026
- Type: Explainer

Media companies are expected to tailor content and messaging across channels, with personalization programs typically targeting lower churn, deeper sessions, and higher subscription revenue. The proposed foundation starts with an audit of CRM, email, warehouse, and customer data platform sources, followed by required-field definitions, ownership mapping, consent tagging, and ingestion-quality checks. Teams then map segment behaviors and conversion or retention paths, including negative signals, before selecting a CDP with real-time ingestion, identity resolution, unified profiles, segmentation, activation, and governance. Identity rules should connect anonymous and authenticated activity across web, mobile, and connected TV, while event-level streaming delivers messages through email, push, in-app, and advertising channels with minimal latency. The roadmap recommends an MVP comprising unified ingestion, a homepage recommendation surface, and triggered re-engagement email, with performance, churn, revenue lift, and model drift monitored through iteration.


### [A Modern AI Risk Management Framework](https://yomu.fyi/post/a-modern-ai-risk-management-framework.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 22, 2026
- Type: Explainer

AI systems introduce probabilistic behavior, model drift, adversarial manipulation, bias, and auditability challenges that traditional IT risk practices may not address. The framework described combines NIST AI RMF, EU AI Act, and ISO/IEC 23894:2023, using Govern, Map, Measure, and Manage as its operational structure while recognizing regulatory and societal context. It recommends cross-functional governance, clear risk ownership, an AI-BOM covering systems, data flows, dependencies, and accountability, plus continuous measurement of fairness, explainability, data quality, security vulnerabilities, and harm likelihood and severity. Across the AI lifecycle, the text identifies risks including data poisoning, model drift, malicious library injection, prompt injection, hallucinations, and platform vulnerabilities, and points to the AI RMF Playbook for checklists, timelines, and governance artifacts.


### [Machine Learning Solutions: A Complete Implementation Guide](https://yomu.fyi/post/machine-learning-solutions-a-complete-implementation-guide.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 21, 2026
- Type: Tutorial

The guide presents machine learning delivery as a lifecycle problem, arguing that failures arise more often from planning, scoping, communication, and expectations than from algorithmic limitations. It recommends assessing data readiness before development by inventorying sources, checking quality and coverage, planning labeling, and engineering features. Once a measurable business outcome is defined, teams should match the learning paradigm and model complexity to the problem, establish cross-validation and holdout or temporal validation, and estimate compute and storage early. Baselines, reproducible experiments, deployment monitoring, model versioning, CI/CD, and retraining workflows support the transition from prototype to production. The guide also calls for representative proof of concepts and A/B testing or control groups to attribute business impact before scaling.


### [Data Governance Platforms: Evaluation & Feature Guide](https://yomu.fyi/post/data-governance-platforms-evaluation-feature-guide.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 21, 2026
- Type: Tutorial

This guide presents a framework for evaluating data governance platforms for enterprise deployment, distinguishing governance—the policies, roles, and controls for data use—from data management, the operational execution of those policies. It recommends assessing metadata-centered capabilities including continuous data profiling, catalog search and enrichment, end-to-end lineage, RBAC and ABAC, sensitive-data detection, masking, audit trails, compliance reporting, and data-subject request workflows. Vendor assessment should also cover pre-built connectors, REST APIs and SDKs, near-real-time metadata synchronization, schema-drift handling, cross-cloud federation, usability, support, implementation timelines, and three-to-five-year total cost of ownership. The proposed decision process selects three leading candidates, pilots them with representative structured and unstructured datasets, defines quality, lineage, adoption, and exit metrics, and uses executive review before procurement; governance is framed as an ongoing program that expands with AI workloads and regulatory change.


### [Introducing the Databricks Excel add-in for business users](https://yomu.fyi/post/introducing-the-databricks-excel-add-in-for-business-users.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Toussaint Webb, Kaitlin Baumgardner
- Published: Apr 21, 2026
- Type: Announcement

Databricks has introduced the Excel Add-in, now in public preview, to let business users import and analyze lakehouse data directly in Excel without writing SQL or configuring ODBC drivers. Built on Databricks SQL and Unity Catalog, it provides point-and-click access to tables and metric views, native Excel pivot tables, refreshable results, reusable Workspace queries, and optional saved SQL. Metric views let data teams define business semantics once while Unity Catalog centrally manages permissions and controls, helping avoid duplicated logic and metric drift. The preview supports selecting tables and metric views, parameterized custom functions, manual refreshes, pivot-table filtering, query reuse, and workspace changes; scheduled refresh, AI integrations, and other usability improvements are planned next.


### [Building an Enterprise Data Management Strategy](https://yomu.fyi/post/building-an-enterprise-data-management-strategy.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 21, 2026
- Type: Explainer

Enterprise data management (EDM) is presented as an organizational capability for governing, integrating, securing, and operationalizing data across its lifecycle as volumes and regulatory scrutiny increase. The strategy addresses silos, inconsistent schemas, undocumented transformations, compliance exposure, and the difficulty of supporting analytics and AI across fragmented systems. It combines governance councils and ownership, automated profiling and quality thresholds, classification with RBAC and encryption, reusable ETL/ELT pipelines, lineage tracking, and architecture choices such as lakehouses and medallion patterns. Unity Catalog is described as a unified governance layer for reads and writes across open formats including Delta Lake and Apache Iceberg, while federation can provide access to external catalogs without copying data. Execution begins with cross-functional sponsorship, a high-impact pilot, measurable KPIs, and quarterly reviews; the source frames EDM as continuous rather than one-time work.


### [Real-Time Decisioning for AI Agents: Why you Need a Customer Context Layer First](https://yomu.fyi/post/real-time-decisioning-for-ai-agents-why-you-need-a-customer-context-la.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Alex Dean
- Published: Apr 21, 2026
- Type: Explainer

Real-time decisioning for AI agents depends on more than customer records: it requires a customer context layer that captures what people are doing now and connects that behavior to identity. The post positions the data platform as the center of a composable martech architecture, with agents and applications operating on shared data in open formats such as Apache Iceberg and Delta Lake. Snowplow’s described approach places structured event collection, schema validation, enrichment, and identity resolution before data reaches the platform, while operating in the customer’s cloud environment. It distinguishes historical profiles from granular behavioral event streams and argues that agent interaction outcomes should return to the foundation as first-class events. The resulting four-stage loop—collect, resolve and enrich, serve, and learn—supports simultaneous real-time and historical context, with decision quality dependent on source data coherence.


### [A Practical Guide to LLM Fine Tuning](https://yomu.fyi/post/a-practical-guide-to-llm-fine-tuning.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 21, 2026
- Type: Tutorial

The guide presents LLM fine tuning as a way to adapt a pre-trained model to specific tasks, domains, or applications without full pretraining. It explains when prompting or few-shot learning is sufficient and when fine tuning is justified by quality gaps, domain terminology, latency, cost, or behavioral control. The recommended lifecycle covers scoping, data collection and cleaning, base-model and method selection, training, evaluation, deployment, and monitoring, with production-format consistency and held-out splits treated as important safeguards. It compares supervised and full fine tuning with parameter-efficient fine tuning (PEFT), including LoRA and QLoRA, which update fewer parameters and reduce compute and memory requirements. The main risks are noisy data, overfitting, and catastrophic forgetting; the proposed default is a pilot that compares a PEFT-adapted model with its base model, while combining fine tuning with RAG and prompt engineering when appropriate.


### [AI Data Transformation Guide for Data Engineers and Data Scientists](https://yomu.fyi/post/ai-data-transformation-guide-for-data-engineers-and-data-scientists.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 21, 2026
- Type: Tutorial

AI data transformation converts raw source data into clean, structured formats for analytics, reporting, and AI systems, while this guide describes production workflows for data engineers and data scientists. It covers discovery and profiling, documented cleaning rules, source-to-target mapping, lineage, AI-generated SQL or Python, validation, testing, and governance. Human review remains required before AI-generated code reaches production, and the guide recommends versioning scripts with datasets, monitoring drift, and involving data scientists in field mapping early. It also compares ETL, which transforms before loading, with ELT, which transforms inside the warehouse, and recommends piloting one representative workflow before broader rollout. The stated goals are reducing repetitive work, improving data quality early, and delivering analysis-ready outputs without manual fixes.


### [Concurrency Control in DBMS: How Locking, MVCC and Optimistic Strategies Keep Data Consistent](https://yomu.fyi/post/concurrency-control-in-dbms-how-locking-mvcc-and-optimistic-strategies.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 21, 2026
- Type: Explainer

The guide explains how database management systems coordinate simultaneous transactions to preserve consistency and serializability, preventing dirty reads, lost updates, non-repeatable reads and phantom reads. It compares lock-based control, including shared and exclusive locks and two-phase locking, with MVCC, which gives transactions consistent snapshots and lets readers and writers proceed without blocking. The discussion covers pessimistic and optimistic strategies, isolation levels, deadlocks, retries, lock-granularity trade-offs and MVCC’s storage and garbage-collection costs, including PostgreSQL’s VACUUM. For lakehouse workloads, Delta Lake uses a read-validate-commit process with optimistic concurrency, snapshot isolation and conflict detection for concurrent writes. The source presents Databricks Lakebase as a managed implementation offering optimistic concurrency control, snapshot isolation for reads and write-serializable isolation for writes without lock or deadlock management.


### [LLM apps: Real-world use cases, core building blocks and risks to know](https://yomu.fyi/post/llm-apps-real-world-use-cases-core-building-blocks-and-risks-to-know.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 21, 2026
- Type: Explainer

LLM applications are presented as production systems that extend beyond chatbots by connecting foundation models to external data, tools, APIs, memory and workflows. Modern categories include assistants and copilots, retrieval-augmented generation (RAG), autonomous agents, multi-agent systems and local or on-device inference, each with distinct capabilities and trade-offs. A typical RAG pipeline uses vector search over a curated knowledge base, then supplies retrieved passages to the model; chunking, source freshness and retrieval accuracy affect results. Production systems also require orchestration, evaluation, observability, governance, guardrails, audit trails and human-in-the-loop checkpoints, especially when agents can call tools or execute workflows. The source distinguishes prompt engineering, which guides behavior without changing model weights, from fine-tuning, which adapts models to domain-specific tasks, and describes a unified path combining data, models, retrieval, orchestration, evaluation and governance for production deployment.


### [From Static Policies to Autonomous Insurance: How AI enables Real-Time Coverage](https://yomu.fyi/post/from-static-policies-to-autonomous-insurance-how-ai-enables-real-time.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Amee Vora, Anindita Mahapatra, Marcela Granados
- Published: Apr 20, 2026
- Type: Explainer

Insurance products often rely on annual pricing, periodic risk assessment, and coverage changes tied to major life events, even though driving behavior, digital fraud exposure, and travel risk can change in seconds. The post presents embedded insurance and usage-based insurance as a shift toward contextual, just-in-time coverage delivered through apps, checkout flows, vehicle purchases, and partner platforms. Its automotive example combines telematics, geospatial data, and real-time weather signals to update risk scores and premiums from trip behavior such as braking, following distance, mileage, location, and environmental conditions. A Databricks Lakehouse architecture registers devices, ingests telemetry through Zerobus, curates data in a governed Medallion architecture, and exposes features through Lakebase and Databricks One, with Unity Catalog supporting access controls, lineage, explainability, and audit trails. The proposed trajectory leads from embedded coverage toward autonomous insurance whose policies continuously adapt to risk.


### [Bridging data science and marketing: Databricks unveils Delta Sharing integration for Adobe Experience Platform and agentic marketing workflows](https://yomu.fyi/post/bridging-data-science-and-marketing-databricks-unveils-delta-sharing-i.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Justin Fenton, Dan Zuckerberg, Katy Yuan
- Published: Apr 20, 2026
- Type: Announcement

Databricks and Adobe announce three partnership milestones aimed at connecting governed Lakehouse data with marketing activation in Adobe Experience Platform. The changes include Delta Sharing support for zero-copy access, a Databricks Genie connection to Adobe Marketing Agent through Model Context Protocol (MCP), and an upcoming Adobe Marketing Agent beta for the Databricks Marketplace. Using Unity Catalog as the source, Experience Platform can access shared tables and views as virtual tables, while Adobe Data Distiller queries live Databricks data without copying underlying records. Through MCP, agents in both platforms can use governed datasets, metadata, models, campaign metrics, SQL Warehouses, and Model Serving endpoints with natural-language interactions. The stated outcome is a shared, real-time context for audience creation, campaign analysis, closed-loop attribution, and model improvement, while reducing data movement, duplication, latency, egress, and governance overhead.


### [Take Control: Customer-Managed Keys for Lakebase Postgres](https://yomu.fyi/post/take-control-customer-managed-keys-for-lakebase-postgres.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ben Hagan
- Published: Apr 20, 2026
- Type: Announcement

Lakebase Customer Managed Keys (CMK) gives regulated enterprises control of the root of trust through customer-owned keys from AWS KMS, Azure Key Vault, or Google Cloud KMS. Because Lakebase separates persistent storage from elastic Postgres compute that can scale to zero, CMK protects both layers and their caches rather than storage alone. Its hierarchical envelope-encryption model uses the CMK as the root, transient Key Encryption Keys (KEKs) to wrap Data Encryption Keys (DEKs), and unique DEKs for each data segment, while plaintext CMKs remain in the customer’s KMS. Account Admins bind a configured key to a workspace, after which new projects inherit the CMK; rotation needs no re-encryption or downtime, revocation terminates compute instances, and audit services record cryptographic operations.


### [Get hands-on with agents, vibe coding and more at Data+ AI Summit](https://yomu.fyi/post/get-hands-on-with-agents-vibe-coding-and-more-at-data-ai-summit.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Pratyarth Rao
- Published: Apr 20, 2026
- Type: Announcement

Data + AI Summit returns to San Francisco from June 14 through June 18, 2026, with Databricks Training and Certification offering more than 20 hands-on courses across AI agents, vibe coding, data engineering, and Lakebase. Training-exclusive days on Sunday and Monday are designed to provide dedicated learning time, while sessions continue during the conference week. New courses cover agentic development with Genie Code, Claude Code, Cursor, MCP servers, and structured prompting; production agent deployment with MLflow’s ResponsesAgent framework and tracing; Lakeflow Spark Declarative Pipelines; and AI/BI dashboards and Genie Spaces. Onsite certification exams cost $100 plus taxes, a 50% discount, and seven listed exams are available. The Learning Hub adds labs, skills assessments, customized learning plans, and community access.


### [Mercedes-Benz builds a cross-cloud data mesh with Delta Sharing and intelligent replication, cutting costs by 66%](https://yomu.fyi/post/mercedes-benz-builds-a-cross-cloud-data-mesh-with-delta-sharing-and-in.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Alexander Summa, Aleksandar Dragojevic
- Published: Apr 20, 2026
- Type: Problem & solution

Mercedes-Benz built a cross-cloud data mesh to share after-sales data between AWS and Azure while supporting its data-defined vehicle strategy. Existing direct queries incurred egress costs, and weekly full loads delayed updates for datasets that included vehicle over-the-air events and workshop visits; source tables were in Iceberg while many consumers expected Delta. Using Unity Catalog and Delta Sharing, the company federated AWS Glue tables, established cross-metastore shares, and used periodic serverless Sync Jobs with Delta Deep Clone to incrementally replicate selected tables into Azure object storage for local querying. This hybrid model preserves direct sharing for fresher workloads while reducing transfer for less time-sensitive ones. For the initial ten data products, egress costs fell 66%, update frequency improved from weekly to every second day, and the source reports an approximately 93% annual reduction in its 50-use-case comparison.


### [What Is a Transactional Database?](https://yomu.fyi/post/what-is-a-transactional-database.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Databricks Staff
- Published: Apr 20, 2026
- Type: Explainer

Transactional databases handle large volumes of short, real-time read and write operations for systems such as orders, payments, account balances, and customer records. They support online transaction processing (OLTP) by treating related steps as one logical unit: successful work commits, while failures roll back the full transaction. ACID—atomicity, consistency, isolation, and durability—protects integrity under concurrency and failures, with row-oriented storage optimized for individual-record access, locking and isolation levels coordinating concurrent work, and write-ahead logging (WAL) supporting crash recovery. The source positions these databases as foundations for banking, e-commerce, healthcare, reservations, and telecommunications, while noting that row-oriented execution is less suitable for large analytical workloads and that horizontal scaling is more complex than in distributed NoSQL systems.


### [Introducing Genie Agent Mode](https://yomu.fyi/post/introducing-genie-agent-mode.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Hanlin Sun
- Published: Apr 17, 2026
- Type: Announcement

Databricks introduces Agent mode in Genie spaces, an agentic process that plans, explores, and reasons over data to answer business questions. For each question, Genie can confirm an issue, test potential contributors through multiple queries, reflect on results, and decide what to investigate next, using Unity Catalog metadata and author-defined semantics from the Genie space. It produces a findings report with quantified results, identified contributors, visualizations, and references to the underlying SQL, and may include actionable recommendations. The agent scales its reasoning to task complexity, using validation for straightforward questions and more planning for multi-step investigations; in AI/BI Dashboards, Genie uses Agent mode by default. Workspace admins enable the feature in Workspace Previews, after which users turn on the Agent toggle in Genie spaces; API support and unstructured document analysis are planned additions.


### [Governing coding agent sprawl with Unity AI Gateway](https://yomu.fyi/post/governing-coding-agent-sprawl-with-unity-ai-gateway.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Aarushi Shah, Ankit Mathur, Bilal, Kevin Stumpf, Rohit Agrawal, Harish Gaur, Ana Nieto
- Published: Apr 17, 2026
- Type: Announcement

Databricks introduces coding agent support in Unity AI Gateway to address security, cost, and visibility challenges created by organizations using multiple coding tools. The gateway provides a unified governance hub for Codex, Cursor, and Gemini CLI, combining access controls, usage statistics, operational observability, cost management, guardrails, and inference capacity. Agent access can be governed centrally, with audit logs in Unity Catalog, MCP servers managed in Databricks, MLflow tracing, shared cost limits, and coding metrics and traces ingested into Unity Catalog-managed Delta tables through OpenTelemetry. The announcement says developers can authenticate with Databricks credentials across connected services, use centralized budgets and model capacity, and let organizations analyze adoption, developer velocity, and rate-limit pressure in the lakehouse; support for Cursor, Gemini CLI, and Codex CLI is available to all Databricks customers.


[Newer posts](https://yomu.fyi/company/databricks/page/27.md) · [Older posts](https://yomu.fyi/company/databricks/page/29.md)