Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
432 postsWebsite
Latest articles
Data + AI Summit 2026: Insider’s Guide for Financial Services Leaders
Data + AI Summit 2026 features a dedicated financial services program for leaders evaluating AI transformation across banking, payments, insurance, and professional services. It lists sessions on proprietary data for underwriting, responsible AI in banking and payments, and AI delivery in professional services featuring First American, American Modern Insurance Group, Vantage Bank Texas, Santander, FIS, Acxiom, EXL, Bain, and EY. The Financial Services Forum includes executive firesides with leaders from Morgan Stanley, JPMorganChase, Mastercard, and RBC Capital Markets. A financial services lounge at the Moscone Expo offers demos, Databricks experts, and Agentic Banker and Virtual CFO use cases. Training courses on AI Agents, Lakebase, and apps plus hands-on labs and certification sessions are presented as a route from strategy to execution with executives and technical teams splitting focus.
Kim HattonThe AI and Data Transformation Playbook for Enterprise Teams
The playbook presents AI and data transformation as an enterprise capability built on trustworthy, well-governed data, rather than a model-only project. It explains how governance, scalable pipelines, and continuous quality monitoring address incompatible formats, missing values, duplicate records, and schema inconsistencies from systems such as CRM platforms, operational databases, IoT sensors, and cloud applications. Operational guidance covers KPI design, named pipeline ownership, review checkpoints, data mapping and lineage, automated de-duplication, deterministic enrichment, regulatory classification, and role-based access. It states that deterministic tests and human review remain necessary for AI-assisted code, while telemetry and quarterly audits help detect reliability, governance, and data quality degradation. The concluding position is that operational discipline—not model sophistication—supports reliable machine learning, predictive analytics, and generative AI outcomes.
Databricks StaffWhat is AI Search?
AI search interprets natural-language questions using semantic understanding and large language models, then returns synthesized answers grounded in cited sources. Unlike traditional search, which matches query words to indexed pages, it tracks context and intent for conversational follow-ups. Its pipeline includes query understanding, embeddings, vector search, retrieval, LLM synthesis, and citations; retrieval-augmented generation (RAG) connects generated responses to source material. The article distinguishes consumer tools that search the open web from enterprise systems that retrieve private, governed company data while respecting access permissions. It concludes that AI search can improve direct information access but should not be treated as definitive, and that sensitive business use requires purpose-built platforms with governance, security, and retrieval quality.
Databricks StaffYour guide to the Telecommunications Industry Experience at Data and AI Summit 2026
Data + AI Summit 2026 presents a Telecommunications Industry Experience for operators responding to surging network traffic, regulatory pressure, cybersecurity threats, competition, and customer churn. The event, scheduled for June 15–18 in San Francisco, positions unified data and AI, governed workflows, and production use cases as the basis for operationalized, AI-native telecom models. Its June 17 Telecommunications Industry Forum features keynotes, presentations, and executive panels on customer experience, autonomous network operations, fraud prevention, secure agent deployment, and the return from modernizing legacy data warehouses. Breakout sessions cover automated metadata generation for Genie, conversational AI/BI, Lakeflow pipelines with Agent Bricks, and data exfiltration protection with egress monitoring. The industry lounge will demonstrate Model as a Service and agentic real-time decisioning, while the agenda emphasizes peer examples and architectural blueprints for scaling AI under telecom governance and compliance.
Elena Tesser, Nevash Pillay3x Faster Search: Parallel Test-Time Scaling with Instructed-Retriever-1
Instructed-Retriever-1 is introduced as the retrieval-specialized model behind an update to Agent Bricks Knowledge Assistant, targeting search quality and latency. It parallelizes query and filter generation for broader candidate recall, then applies multi-pivot groupwise reranking to improve precision, using one model for both retrieval stages. The model was trained on synthetic enterprise-style environments and evaluated on realistic Knowledge Assistant workloads, where it scored 81.0 nDCG@10 for reranking and matched Claude Sonnet 4.5 retrieval quality on KARLBench. Serving uses a Mixture-of-Experts architecture, FP8 quantization, speculative decoding, and infrastructure tuning; the update reports more than 3x faster search, 2x faster answer generation, and TTFT around two seconds. End-to-end latency was consistently below 10 seconds in the offline evaluation setup.
The Databricks AI Research TeamApache Spark Real-Time Mode for Gaming: A Better Way to Do Real-Time Sessionization
Gaming platforms need sub-second session data for personalization, recommendations, content scheduling, and operational controls across millions of devices. The described pipeline uses Apache Spark Real-Time Mode with transformWithState to process Kafka events, maintain per-device session state, and emit scheduled heartbeats and timeout events without a separate streaming engine. Events are grouped by deviceId; a StatefulProcessor uses MapState keyed by session ID, while handleInputRows() handles starts and ends and handleExpiredTimer() registers and fires 30-second timers. The implementation was tested at about 500K input events per minute, 4M active sessions, and 8M heartbeat records, producing 16x output amplification. End-to-end latency reached 432 ms at p99, reported as 20x faster than micro-batch mode, and the source states that Real-Time Mode is generally available.
Neha Prabhu, Murali TalluriBring Databricks into Kiro IDE with the AI Dev Kit Power
AI-assisted development can produce unreliable SQL and models when an assistant guesses schema details or exceeds the user's data access. Kiro can connect to Databricks through Model Context Protocol (MCP) in two ways: four Databricks-managed remote servers for Genie, SQL, Unity Catalog Functions, and Vector Search, or the Databricks AI Dev Kit Power, which installs a local Python MCP server and broader skills. The AI Dev Kit now supports Kiro through its unified installer, while the Power provides one-click onboarding with authentication detection and skill loading. Both paths use Unity Catalog permissions, including row-, column-, and tag-based grants, so the assistant sees the user's effective access; Path A uses token-based configuration, while Path B supports OAuth U2M, OAuth M2M, profiles, or PATs. Examples show schema-grounded SQL, dbt joins using real columns, query comparisons, lineage checks, and generation of Databricks jobs or Asset Bundles.
Antony Prasad Thevaraj, Venkatavaradhan ViswanathanScaling Enterprise Conversational Intelligence: Cross-industry Technology and Functional Solutions Powered by Databricks Genie
Databricks Genie is presented as a cross-industry technology layer for enterprise challenges including financial planning, legal compliance, and IT operations. As a “Research Agent,” it can generate multi-step research plans to explain business anomalies and support answers with verifiable proof from the lakehouse. The post showcases partner solutions across technology, sales, marketing, HR, finance and procurement, supply chain, customer service, and IT operations, with examples spanning governed analytics, multi-agent orchestration, data observability, causal analysis, and incident management. These implementations aim to replace fragmented or static workflows with real-time, contextualized intelligence and production-grade agentic workflows, supporting anomaly investigation, root-cause analysis, ticket classification, and conversational troubleshooting. The stated goal is faster, more confident decision making across departments through governed self-service access to insights.
Amit SinghBeyond parsing X12: Closing the gap for revenue cycle workflows in healthcare
Healthcare X12 pipelines can normalize remittance and enrollment data without giving medical billers a place to act on denials, leaving short-pays in SQL queries and spreadsheets while filing deadlines approach. Genpact and Databricks address that workflow gap with a two-layer operational workbench inside Databricks: production extensions to the open-source x12-edi-parser and a secure web application over Databricks SQL. The engine adds contextual 834 enrollment tracking and decoded 835 CAS adjustments, while the interface provides claim and service-line drill-downs, denial queues, dashboards, quality gates, and appeal or correction drafting. Live Unity Catalog gold-view queries avoid ETL shadow copies and synchronized caches; PHI remains inside the secure perimeter, with row-level security and audit logging for reveals and workflow changes. The proposed next step is Claude-based appeal drafting through Databricks Foundation Model APIs, and a two-week configuration-based trial uses an organization's schema and twenty existing denials.
Aaron Zavora, Neel ShapurAgentic BI: A Practical Guide for BI Teams and Business Users
Agentic BI uses autonomous AI agents to automate work between raw business data and actionable insight, including data preparation, query execution, chart and narrative generation, and report distribution. Traditional BI depends on analysts to gather data, write queries, maintain dashboards, and assemble reports, while agentic systems let business users ask natural-language questions and receive governed answers. The guide identifies a governed semantic layer as foundational, because shared metric definitions and deterministic execution help keep outputs consistent, auditable, and trustworthy, with human approval checkpoints for higher-risk handoffs. It recommends inventorying data structure, schema drift risk, and integration costs, then piloting a narrowly defined workflow and measuring time to insight, analyst hours reclaimed, satisfaction, and accuracy before expansion.
Databricks StaffData Science vs Data Analytics: Compare Careers, Skills, and Degrees
Data analytics and data science are compared as data-focused career paths, with analytics interpreting existing, usually structured data for trends and business decisions, while science builds models and automated systems to predict future outcomes. The guide contrasts typical questions, outputs, tools, education paths, and roles: analysts use SQL, Excel, Tableau, or Power BI for dashboards and reports, whereas data scientists use Python, R, Apache Spark, and MLflow for predictive models and algorithms. It describes analytics types from descriptive through prescriptive, and a data science workflow spanning collection, feature engineering, training, validation, and deployment, including unstructured data such as text, images, and sensor streams. It also explains collaboration, including analysts defining problems and baselines before scientists build models, and offers portfolio suggestions and questions for choosing a path.
Databricks StaffPractical Data Lakehouse Examples and Use Cases
The article presents practical data lakehouse patterns for streaming analytics, IoT pipelines, machine learning workflows, and enterprise reporting, addressing the gap between theoretical definitions and deployable examples. It explains how a lakehouse combines low-cost, schema-flexible object storage with schema enforcement, ACID transactions, data versioning, lineage tracking, and query performance, allowing SQL and ML workloads to use shared open-format tables. Examples include second-level fraud detection, medallion-based historical analytics, predictive maintenance from sensor data, and governed customer 360 profiles. The implementation guidance covers raw storage and partitioning, centralized catalogs, decoupled compute, role-based access, time travel, migration coexistence, SLAs, observability, and lifecycle policies. Together, these patterns are presented as a unified architecture that reduces duplication and data movement while supporting governed analytics at scale.
Databricks StaffData Governance Architecture: A Complete Blueprint for Modern Organizations
Data governance architecture is presented as a blueprint for aligning policies, roles, processes, and technologies with business outcomes. It defines objectives including consistent data definitions, data integrity, layered access controls, and secure self-service analytics, while assigning responsibilities across executives, architects, engineers, analysts, managers, and compliance teams. Core principles are accountability, transparency, consistency, and stewardship, supported by federated ownership through councils, data owners, and embedded stewards. The discussion compares DAMA-DMBOK, TOGAF, and Zachman according to organizational scale, regulatory context, and architecture maturity, and describes modern patterns including lakehouse, data mesh, and data fabric. It concludes that effective programs require executive sponsorship, documented roles, measurable quality controls, iterative implementation, and sustained change management.
Databricks StaffQuery Tags: The Context Your Warehouse Queries Have Been Missing
Databricks SQL Query Tags address a visibility gap in warehouse workloads: built-in logs identify users, warehouses, and tools, but not dashboards, projects, teams, or cost centers. Query Tags, now in Public Preview, attach multiple custom key-value pairs to each SQL execution, carry them into the Query History System Table, and expose them in the Query Profile UI. Partner integrations can automatically tag dbt models, while Power BI and Tableau support connection-level tags, and APIs and connectors support connection- or statement-level metadata. Users can also issue SET QUERY_TAGS in SQL Editor, notebooks, dashboards, or alerts to label subsequent session statements. These tags enable SQL or Genie queries for cost allocation, regression analysis, workload filtering, and environment comparisons; future plans include broader connector and workload support plus Query History search.
JooHo Yeo, Jiabin HuPractical Data Warehouse Design and Architecture Guide
The guide presents data warehouse design as a business-aligned process for building, modernizing, or scaling analytics infrastructure, rather than merely storing data. It recommends defining analytics use cases and stakeholder needs first, then organizing a three-tier architecture of source, storage, and semantic output layers; cloud designs can decouple compute and storage and use open formats. A Bronze–Silver–Gold medallion flow preserves raw lineage, applies cleansing and deduplication, and produces consumption-ready dimensional models, while retention and archival policies control sprawl. For modeling, it favors star schemas for user-facing BI, uses snowflake normalization when redundancy is material, and stresses explicit fact-table granularity, domain-owned data marts, and workload-specific refresh cadences. Governance and operations include Unity Catalog, access controls, masking, lineage, multi-region deployment, disaster recovery, and CI/CD, followed by phased rollout through high-value domains.
Databricks StaffAI Governance Maturity Model: Matrix, Assessment, and Roadmap
The AI governance maturity model assesses how deeply governance practices are embedded across an organization’s data, process, and people dimensions, using five stages from Ad Hoc to Optimized. It frames the model as a diagnostic and roadmap for boards and executive sponsors, while a five-dimension matrix separately scores strategy and leadership, policy and ethics, risk management, data governance, and monitoring and observability. The progression moves from discovery and basic ownership through standardized controls, quantified risk, real-time indicators, lineage tracking, and automated, context-aware enforcement. The recommended roadmap starts with a baseline within 30 days, targets Level 3 across the five dimensions within 12 months, runs a 90-day pilot on two or three high-priority systems, then scales effective controls through CI/CD integration and monitoring, with quarterly reviews and annual reassessment.
Databricks StaffRAG vs Fine Tuning: Enterprise Decisions for AI Models and AI Systems
The guide frames enterprise choice between retrieval-augmented generation (RAG) and fine tuning as a trade-off between injecting knowledge at inference time and changing model weights before deployment. RAG connects an LLM to external stores by embedding a query, searching vector databases for similar chunks, and adding retrieved context to the prompt, making current information and citations possible. Fine tuning uses curated input-output examples and supervised training to update parameters; full-model training is expensive, while LoRA trains a smaller set of added weights and can support a single-GPU pilot. The decision depends on whether the priority is stable behavior and output format or fresh, traceable knowledge, alongside data quality, latency, compute, and maintenance requirements. For many mature deployments, the guide recommends combining both: fine tuning for domain understanding and consistency, and RAG for real-time data.
Databricks StaffIntroducing Cross-Engine ABAC
Cross-engine ABAC is announced in Beta, extending Unity Catalog's fine-grained governance to external engines through Iceberg REST Catalog APIs. It supports tag-based row filters and column masks, including conditional logic and SQL UDFs, while allowing policies to be defined once and enforced across engines. For an external query, the engine sends a scan request, Unity Catalog evaluates entitlements and applicable policies, and returns a filtered scan plan before the engine processes authorized files. Enforcement remains at the catalog layer, so engines need not implement governance logic and can use the open scan APIs. Apache Spark is supported today through Iceberg-Spark and Delta-Spark connectors, with Starburst and DuckDB integrations coming soon; the Beta also points toward Apache Iceberg label exchange for future governance metadata sharing.
Alex Jiang, Alex Reid, Michelle LeonPersonalizing Genie Code with instructions, skills, memory, and MCP
Genie Code adds personalization through persistent instructions, task-specific Agent Skills, shared repository conventions, and MCP servers. Instructions apply across every agent-mode session, while AGENTS.md and CLAUDE.md files can be discovered automatically for team conventions. Skills are markdown-based packages with names, descriptions, guidance, reusable code, or scripts, scoped to workflows and available personally or through Workspace/.assistant/skills/. MCP servers expose external tools, data, and workflows from managed services, Unity Catalog-connected servers, or Databricks Apps, with workspace admin controls and permission-based user selection. Managed OAuth flows for Google Drive, SharePoint, and GitHub are currently in beta, while the MCP marketplace and Genie Code settings support server discovery and setup.
Samantha Banchik, Gal Oshri, Romain Rigaux, Will Tipton, Chloe ChanDebunking 8 data layout myths: why Liquid Clustering outperforms partitioning
Partitioning has long been the standard physical layout for Hadoop- and Hive-era processing, but the post argues that changing Lakehouse workloads make fixed choices brittle. Liquid Clustering treats keys as guidance for file organization, allowing layouts to evolve, supporting low- and high-cardinality filters, maintaining file sizes, and enabling row-level concurrency. Using transaction-log metadata and per-column statistics, the post says modern Delta and Iceberg engines prune files rather than directories, and reports 35% lower clustering time, 22% faster queries, roughly 90% faster metadata-only deletes, and 23-minute planning for a 10-PB table. Conversion examples include Bolt’s 138% higher write throughput and up to 63% faster reads, while a 1.1-PB workload saw 5.9x faster queries, 86% fewer bytes read, and a 27% smaller table after clustering by date, hour, source, and id.
Jeffrey Gong, Yu Xu, Rahul Mahadev