Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
- 541 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
Modern BSA/AML compliance on Databricks
AML operations are strained by fragmented systems, high false-positive volumes, manual case documentation, and opaque vendor scoring, leaving analysts focused on backlog rather than financial-crime intelligence. The proposed Databricks Data + AI Platform unifies transaction monitoring, KYC, sanctions, case history, and policy data under Unity Catalog, using Lakeflow Connect and a Bronze–Silver–Gold Delta architecture with masking, row-level security, and lineage. MLflow, Model Serving, Lakehouse Monitoring, and inference tables support institution-specific detection models, while Agent Bricks coordinates agents for evidence gathering, recommendations, and SAR drafting with analysts retaining final decisions. The architecture also uses Lakebase for governed operational state and Databricks Apps for analyst and executive experiences. Reported outcomes include a 75% reduction in false positives reaching the analyst queue and compressing three-to-six-hour investigations to minutes.
Claude Fable 5 is now available on Databricks, fully governed through Unity AI Gateway
Claude Fable 5 is now generally available on Databricks, with rollout across AWS, Azure, and Google Cloud through Unity AI Gateway. The Mythos-class model targets long-running, complex, and ambiguous work, including autonomous enterprise workflows, document question answering, code investigation, and multimodal tasks. In Databricks' OfficeQA Pro benchmark, Fable 5 achieved 57.9% correctness, setting a state of the art; compared with Claude Opus 4.8, it was 20% more accurate and used 12% fewer tool calls, but ran approximately 30% slower and generated 2.5x more output tokens. Unity AI Gateway provides unified API access, fine-grained permissions, Unity Catalog logging, request and tool-call guardrails, and spend controls. Agent Bricks supports domain-specific agents, while Anthropic's policy includes 30-day retention for trust and safety purposes only.
Announcing the winners of the 2026 Databricks Customer Awards
The 2026 Databricks Customer Awards recognize organizations and leaders using the Databricks Data + AI Platform across eight categories and four regions. The announcement names winners including Applied Materials, Virgin Atlantic, Fonterra Co-operative Group, Telefónica | Vivo, Virtue Foundation, Octopus Energy, Axpo, Atlassian, Wassym Bensaid at Rivian and Volkswagen Group Technologies, and Kenan Colson at Lippert. Examples include Applied Materials’ move from a Hadoop-based data lake to a governed lakehouse, with 1,500-plus analysts, more than 100 production machine-learning models and faster pipeline development, while Fonterra centralized data to improve supply-chain and compliance work. Rivian unified vehicle, factory and enterprise data, consolidated legacy systems into Delta and Unity Catalog, and designed for 500–600 petabytes; Lippert deployed AI tools across customer care, finance and HR. The announcement presents data and AI adoption as a cross-functional operating model.
Announcing the 2026 Databricks Customer Awards Industry winners
Databricks announced its 2026 Customer Awards Industry winners, recognizing 10 organizations across financial services, communications, health and life sciences, manufacturing, retail and CPG, energy and utilities, enterprise technology, public sector, digital-native businesses and cybersecurity. The cited work uses data and AI to address industry-specific needs, including SMBC Group’s governed lakehouse for risk and finance, Hospital for Special Surgery’s full-system ingestion strategy, Lumen’s conversational service-operations workflows and Superhuman’s high-volume model serving. Reported results include HSS ingesting more than 40 source systems and creating over 14,500 production tables, Lumen recording 3 million-plus AI-powered diagnostics and 35% ticket deflection, and Superhuman handling peaks above 200,000 queries per second. Adobe is also recognized for applying software engineering practices to cybersecurity detection workflows, reducing false positives and improving development speed.
Transforming solar and wind maintenance reports with Genie and AI agents
Plenitude and Databricks built an agent-based system that turns solar and wind plant maintenance PDFs into structured data for cross-plant analysis. Event-driven ingestion uses Databricks Jobs and the ai_parse_document AI Function to extract text, tables, figures, and metadata, then stores page- and object-level JSON records in Delta Lake with coordinates, version history, and links to source reports. A Genie space uses Unity Catalog metadata, knowledge-store instructions, and SQL generation to answer natural-language questions, produce visualizations, and export results, while Agent Bricks can orchestrate multi-step workflows and downstream actions. The design also applies automatic liquid clustering to dynamic queries and row-level security to restrict results by country. The resulting data layer supports historical trends, plant comparisons, recurring-fault analysis, and a foundation for predictive maintenance, although the source frames predictive use as a future improvement.
Enterprise Data Strategy Roadmap for Business Outcomes
An enterprise data strategy connects organizational data assets to measurable business outcomes, while fragmented architectures can leave data investments uncoordinated and limit real-time analysis and action. The roadmap starts with purpose, scope, executive sponsorship, measurable objectives, KPI mapping, and use-case prioritization based on business impact, feasibility, time to value, and organizational readiness. It then organizes governance, lifecycle management, data quality, target-state architecture, integration, analytics, team structure, compliance, and measurement as interdependent capabilities, emphasizing owners, stewards, decision rights, executable quality rules, and automated cleansing. Implementation proceeds through a time-boxed cross-functional pilot, documented learnings, and incremental scaling, with steering-committee oversight and governance that evolves through feedback. The text gives indicative timelines of 60 to 90 days for a focused pilot, 12 to 18 months for a foundational platform across multiple business units, and multiple years for a mature data-driven culture.
What is Human-in-the-Loop (HITL)?
Human-in-the-loop (HITL) is an AI and machine learning approach that places people in training, supervision, or decision-making to improve accuracy, safety, and ethical alignment. Its feedback loop can include data labeling, output review, escalation, approval, override, and continuous feedback, with confidence thresholds and risk scoring routing only selected decisions to people. The explainer distinguishes HITL, where review occurs before flagged actions, from human-on-the-loop monitoring and human-over-the-loop governance, and separates HITL from RLHF, a training-specific technique. It describes uses in medical imaging, moderation, autonomous vehicles, financial services, and AI agents handling consequential actions. Databricks Agent Bricks is presented as supporting governed traces and Agent Learning from Human Feedback, including a case where 32 feedback items improved instruction-following from roughly 12% to 80%.
What is Explainable AI (XAI)?
Explainable AI (XAI) comprises techniques that help people understand how AI systems produce specific outputs, particularly when machine-learning and deep-learning models operate as black boxes. It distinguishes intrinsically interpretable models, such as decision trees and linear or logistic regression, from post-hoc methods including SHAP, LIME, counterfactuals, saliency maps and Grad-CAM. A typical workflow selects a model and prediction, applies a method suited to the model and audience, reviews outputs such as feature scores or heatmaps, and uses them to assess accuracy, fairness, reliability and compliance. The article stresses that post-hoc explanations are approximations rather than definitive proof, so teams should validate them with domain expertise and, where appropriate, combine methods; MLflow and Unity Catalog can preserve explanation artifacts, lineage and auditability.
Enabling Evolutionary Database Development: database branching with Lakebase, continued
This installment revisits Evolutionary Database Design and argues that Databricks Lakebase removes the infrastructure constraints that kept several of its database-change practices aspirational. Lakebase is a managed Postgres database with compute separated from shared durable storage, while copy-on-write branches create a new pointer and divergence marker in roughly one second without copying parent data. That enables per-developer, per-PR, and per-experiment databases, real Postgres test branches instead of mocks or in-memory substitutes, and CI isolation at pull-request granularity. The updated playbook adds idempotent migrations, destructive testing, and database-level A/B prototyping, with Unity Catalog governance inherited by branches and agents receiving branches rather than production access. Jen’s example corrupts production-shaped data to test an inventory-code migration and compares column and lookup-table designs, rejecting the latter because its common read path requires a join.
Data + AI Summit 2026: Insider’s Guide for Financial Services Leaders
Data + AI Summit 2026 features a dedicated financial services program for leaders evaluating AI transformation across banking, payments, insurance, and professional services. It lists sessions on proprietary data for underwriting, responsible AI in banking and payments, and AI delivery in professional services featuring First American, American Modern Insurance Group, Vantage Bank Texas, Santander, FIS, Acxiom, EXL, Bain, and EY. The Financial Services Forum includes executive firesides with leaders from Morgan Stanley, JPMorganChase, Mastercard, and RBC Capital Markets. A financial services lounge at the Moscone Expo offers demos, Databricks experts, and Agentic Banker and Virtual CFO use cases. Training courses on AI Agents, Lakebase, and apps plus hands-on labs and certification sessions are presented as a route from strategy to execution with executives and technical teams splitting focus.
The AI and Data Transformation Playbook for Enterprise Teams
The playbook presents AI and data transformation as an enterprise capability built on trustworthy, well-governed data, rather than a model-only project. It explains how governance, scalable pipelines, and continuous quality monitoring address incompatible formats, missing values, duplicate records, and schema inconsistencies from systems such as CRM platforms, operational databases, IoT sensors, and cloud applications. Operational guidance covers KPI design, named pipeline ownership, review checkpoints, data mapping and lineage, automated de-duplication, deterministic enrichment, regulatory classification, and role-based access. It states that deterministic tests and human review remain necessary for AI-assisted code, while telemetry and quarterly audits help detect reliability, governance, and data quality degradation. The concluding position is that operational discipline—not model sophistication—supports reliable machine learning, predictive analytics, and generative AI outcomes.
What is AI Search?
AI search interprets natural-language questions using semantic understanding and large language models, then returns synthesized answers grounded in cited sources. Unlike traditional search, which matches query words to indexed pages, it tracks context and intent for conversational follow-ups. Its pipeline includes query understanding, embeddings, vector search, retrieval, LLM synthesis, and citations; retrieval-augmented generation (RAG) connects generated responses to source material. The article distinguishes consumer tools that search the open web from enterprise systems that retrieve private, governed company data while respecting access permissions. It concludes that AI search can improve direct information access but should not be treated as definitive, and that sensitive business use requires purpose-built platforms with governance, security, and retrieval quality.
Your guide to the Telecommunications Industry Experience at Data and AI Summit 2026
Data + AI Summit 2026 presents a Telecommunications Industry Experience for operators responding to surging network traffic, regulatory pressure, cybersecurity threats, competition, and customer churn. The event, scheduled for June 15–18 in San Francisco, positions unified data and AI, governed workflows, and production use cases as the basis for operationalized, AI-native telecom models. Its June 17 Telecommunications Industry Forum features keynotes, presentations, and executive panels on customer experience, autonomous network operations, fraud prevention, secure agent deployment, and the return from modernizing legacy data warehouses. Breakout sessions cover automated metadata generation for Genie, conversational AI/BI, Lakeflow pipelines with Agent Bricks, and data exfiltration protection with egress monitoring. The industry lounge will demonstrate Model as a Service and agentic real-time decisioning, while the agenda emphasizes peer examples and architectural blueprints for scaling AI under telecom governance and compliance.
3x Faster Search: Parallel Test-Time Scaling with Instructed-Retriever-1
Instructed-Retriever-1 is introduced as the retrieval-specialized model behind an update to Agent Bricks Knowledge Assistant, targeting search quality and latency. It parallelizes query and filter generation for broader candidate recall, then applies multi-pivot groupwise reranking to improve precision, using one model for both retrieval stages. The model was trained on synthetic enterprise-style environments and evaluated on realistic Knowledge Assistant workloads, where it scored 81.0 nDCG@10 for reranking and matched Claude Sonnet 4.5 retrieval quality on KARLBench. Serving uses a Mixture-of-Experts architecture, FP8 quantization, speculative decoding, and infrastructure tuning; the update reports more than 3x faster search, 2x faster answer generation, and TTFT around two seconds. End-to-end latency was consistently below 10 seconds in the offline evaluation setup.
Apache Spark Real-Time Mode for Gaming: A Better Way to Do Real-Time Sessionization
Gaming platforms need sub-second session data for personalization, recommendations, content scheduling, and operational controls across millions of devices. The described pipeline uses Apache Spark Real-Time Mode with transformWithState to process Kafka events, maintain per-device session state, and emit scheduled heartbeats and timeout events without a separate streaming engine. Events are grouped by deviceId; a StatefulProcessor uses MapState keyed by session ID, while handleInputRows() handles starts and ends and handleExpiredTimer() registers and fires 30-second timers. The implementation was tested at about 500K input events per minute, 4M active sessions, and 8M heartbeat records, producing 16x output amplification. End-to-end latency reached 432 ms at p99, reported as 20x faster than micro-batch mode, and the source states that Real-Time Mode is generally available.
Bring Databricks into Kiro IDE with the AI Dev Kit Power
AI-assisted development can produce unreliable SQL and models when an assistant guesses schema details or exceeds the user's data access. Kiro can connect to Databricks through Model Context Protocol (MCP) in two ways: four Databricks-managed remote servers for Genie, SQL, Unity Catalog Functions, and Vector Search, or the Databricks AI Dev Kit Power, which installs a local Python MCP server and broader skills. The AI Dev Kit now supports Kiro through its unified installer, while the Power provides one-click onboarding with authentication detection and skill loading. Both paths use Unity Catalog permissions, including row-, column-, and tag-based grants, so the assistant sees the user's effective access; Path A uses token-based configuration, while Path B supports OAuth U2M, OAuth M2M, profiles, or PATs. Examples show schema-grounded SQL, dbt joins using real columns, query comparisons, lineage checks, and generation of Databricks jobs or Asset Bundles.
Scaling Enterprise Conversational Intelligence: Cross-industry Technology and Functional Solutions Powered by Databricks Genie
Databricks Genie is presented as a cross-industry technology layer for enterprise challenges including financial planning, legal compliance, and IT operations. As a “Research Agent,” it can generate multi-step research plans to explain business anomalies and support answers with verifiable proof from the lakehouse. The post showcases partner solutions across technology, sales, marketing, HR, finance and procurement, supply chain, customer service, and IT operations, with examples spanning governed analytics, multi-agent orchestration, data observability, causal analysis, and incident management. These implementations aim to replace fragmented or static workflows with real-time, contextualized intelligence and production-grade agentic workflows, supporting anomaly investigation, root-cause analysis, ticket classification, and conversational troubleshooting. The stated goal is faster, more confident decision making across departments through governed self-service access to insights.
Beyond parsing X12: Closing the gap for revenue cycle workflows in healthcare
Healthcare X12 pipelines can normalize remittance and enrollment data without giving medical billers a place to act on denials, leaving short-pays in SQL queries and spreadsheets while filing deadlines approach. Genpact and Databricks address that workflow gap with a two-layer operational workbench inside Databricks: production extensions to the open-source x12-edi-parser and a secure web application over Databricks SQL. The engine adds contextual 834 enrollment tracking and decoded 835 CAS adjustments, while the interface provides claim and service-line drill-downs, denial queues, dashboards, quality gates, and appeal or correction drafting. Live Unity Catalog gold-view queries avoid ETL shadow copies and synchronized caches; PHI remains inside the secure perimeter, with row-level security and audit logging for reveals and workflow changes. The proposed next step is Claude-based appeal drafting through Databricks Foundation Model APIs, and a two-week configuration-based trial uses an organization's schema and twenty existing denials.
Agentic BI: A Practical Guide for BI Teams and Business Users
Agentic BI uses autonomous AI agents to automate work between raw business data and actionable insight, including data preparation, query execution, chart and narrative generation, and report distribution. Traditional BI depends on analysts to gather data, write queries, maintain dashboards, and assemble reports, while agentic systems let business users ask natural-language questions and receive governed answers. The guide identifies a governed semantic layer as foundational, because shared metric definitions and deterministic execution help keep outputs consistent, auditable, and trustworthy, with human approval checkpoints for higher-risk handoffs. It recommends inventorying data structure, schema drift risk, and integration costs, then piloting a narrowly defined workflow and measuring time to insight, analyst hours reclaimed, satisfaction, and accuracy before expansion.
Data Science vs Data Analytics: Compare Careers, Skills, and Degrees
Data analytics and data science are compared as data-focused career paths, with analytics interpreting existing, usually structured data for trends and business decisions, while science builds models and automated systems to predict future outcomes. The guide contrasts typical questions, outputs, tools, education paths, and roles: analysts use SQL, Excel, Tableau, or Power BI for dashboards and reports, whereas data scientists use Python, R, Apache Spark, and MLflow for predictive models and algorithms. It describes analytics types from descriptive through prescriptive, and a data science workflow spanning collection, feature engineering, training, validation, and deployment, including unstructured data such as text, images, and sensor streams. It also explains collaboration, including analysts defining problems and baselines before scientists build models, and offers portfolio suggestions and questions for choosing a path.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.