Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
432 postsWebsite
Latest articles
Inside one of the first production deployments of Lakebase: LangGuard's agentic workflow governance engine
LangGuard is presented as a runtime enforcement layer for enterprise agentic workflows, monitoring actions, decisions, tools, credentials, and intent across connected systems. Its GRAIL data fabric records multidimensional trace data, builds a live knowledge graph, and evaluates allow/deny/modify decisions against policy before tools, datasets, or models are accessed. The deployment uses Databricks Lakebase as the operational system of record, relying on PostgreSQL, serverless autoscaling, scale-to-zero, compute-storage disaggregation, hot-data caching, and copy-on-write branching for policy testing. LangGuard chose this architecture to handle bursty trace writes and low-latency enforcement reads without provisioning for peak demand, while keeping operational data available to Databricks analytics and AI capabilities without additional ETL. The stated next step is predictive governance: training behavioral models on historical traces to flag anomalous agent behavior before a policy violation.
Venkat Raghavan, Jason Keirstead, Ravi Srinivasan, Nina Williams, Amelia WestbergThe next generation of Databricks Genie
The next generation of Databricks Genie expands the assistant beyond individual Genie Spaces, combining structured and unstructured enterprise data in a unified chat experience. It reuses logic from certified Genie Spaces, governed dashboards, and Databricks Apps, with metadata-based routing that prioritizes higher-trust sources, while new reasoning models and agent architecture handle questions spanning multiple domains. Built-in connectors for Google Drive and SharePoint, plus MCP support, let Genie access knowledge stores and take actions; Unity Catalog AI Gateway manages these connections. The experience replaces Databricks One as an account-level global home, adding domains, custom URLs, unified login, automated identity management, and governance through Unity Catalog. Native iOS and Android apps extend access to dashboards, apps, and chat beyond desktop environments.
Ken Wong, Dillon Morrison, Richard TomlinsonModel risk management in 2026: A banker's guide to the revised interagency guidance
The post presents the April 17, 2026 revision to model risk management guidance as a shift toward risk-based, principles-driven oversight, replacing several earlier issuances from the Federal Reserve, FDIC, and OCC. It identifies five operational changes: materiality tiering, end-to-end lifecycle governance, reproducible effective challenge, continuous drift and performance monitoring, and application of MRM principles to GenAI and agentic systems. Its proposed response is a Databricks reference architecture built on Unity Catalog, Delta Lake, Lakeflow Declarative Pipelines, Feature Store, MLflow, Model Registry, Model Serving, and assurance tools. The design makes tiering metadata-driven and turns lineage, validation, approvals, monitoring, documentation, and retirement records into evidence generated during normal model work. The stated conclusion is that a unified substrate can reduce integration and manual evidence work, allowing regulatory changes to become configuration exercises rather than multi-quarter programs.
Pavithra Rao, Jennifer Miller, Chaitanya Varanasi, Kim HattonOpenAI GPT-5.5 + Codex, now available and fully governed on Databricks
Databricks announces native support for OpenAI’s GPT-5.5, making it available for coding workflows with Codex, enterprise agents, document pipelines, and data-driven employee workflows. Access is governed through Unity AI Gateway, which provides permissions and rate limits, configurable guardrails for PII, prompt injection, and content safety, MCP tool-call auditing, failover, and request-level observability for model and Codex interactions. The announcement describes GPT-5.5 use with Genie for natural-language analytics, Agent Bricks Custom Agents for multi-step workflows, and Lakeflow Spark Declarative Pipelines for document-focused GenAI ETL involving summarization, extraction, and classification. GPT-5.5 is available on Databricks across AWS, Azure, and GCP, with entry points including AI Playground, governed Codex endpoints, Agent Bricks, and the Foundation Model API.
Patrick Wendell, Hanlin Tang, Ahmed Bilal, Arnav Singhvi, Ivan Zhou, Harish GaurOperational databases: How they work and when to use them
Operational databases, or online transaction processing (OLTP) databases, support day-to-day operations by processing real-time CRUD transactions with low latency, high concurrency, and ACID guarantees. The guide contrasts them with data warehouses and OLAP systems, which favor historical analysis, complex aggregations, and read-heavy workloads, while describing ETL, CDC, and operational data stores in the OLTP-to-OLAP pipeline. It argues that traditional OLTP systems can struggle with real-time analytics and AI because of siloed data, structured-data constraints, rigid schemas, scaling limits, and missing governance features. Databricks Lakebase is presented as a hybrid solution using separate storage and compute, serverless Postgres, elastic scaling, branching, shared lakehouse storage, and open formats. The proposed workflow connects existing OLTP systems through CDC or streaming into Delta Lake for fresh SQL, BI, ML, and AI access.
Databricks StaffDatabricks partners with OpenAI on GPT-5.5
Databricks announces a partnership with OpenAI around GPT-5.5, described as OpenAI’s strongest frontier model for enterprise agentic work, complex document reasoning, and long-horizon coding agents. The model powers Codex and is presented as able to research, analyze data, create documents and spreadsheets, operate software, use tools, check outputs, recover from ambiguity, and continue through multi-part tasks. Databricks evaluated it on OfficeQA, a benchmark built from 89,000 pages of U.S. Treasury Bulletins that tests document retrieval, table interpretation, and precise calculation. With retrieval handled, GPT-5.5 scored 64.66% versus GPT-5.4’s 57.14%; in the full-agent OfficeQA Pro Agent Harness, it scored 52.63% versus 36.10%, representing reported improvements of roughly 13% and a 46% reduction in errors.
Hanlin Tang, Ahmed Bilal, Arnav Singhvi, Ivan Zhou, Harish GaurGuide to BI Reporting and Maximizing Intelligence Effectiveness
Business intelligence (BI) reporting turns data from operational systems and analytical repositories into structured reports, dashboards, and visualizations for organizational decision-making. The guide explains managed and ad hoc reporting, then traces a typical workflow through data sources, ETL, deduplication, normalization, validation, data modeling, visualization, and automated distribution. It compares established platforms such as Microsoft Power BI, Tableau, SAP BusinessObjects, Qlik, and Looker with newer AI-native tools, while noting evaluation criteria including connectivity, scale, governance, usability, and cost. It also addresses dashboard overload, stale semantic models, and specialist bottlenecks, and cites reports of 10x faster query creation from AI-assisted natural-language querying alongside recommendations to start with one decision, one primary source, and three to five KPIs.
Databricks StaffAnnouncing the Public Preview of Lakeflow Designer
Databricks announces the Public Preview of Lakeflow Designer, a visual, no-code, AI-native experience for data preparation and analytics built directly into the Databricks platform. It targets analysts, domain experts, and other less technical users with drag-and-drop workflows and natural-language authoring, while representing each transformation as an operator with previews. Genie Code can use Unity Catalog metadata, descriptions, lineage, popularity, and example queries to generate or modify workflows, inspect transformation results, and iterate when needed. Every visual transformation produces production-ready Python code that can be reviewed, versioned in Git, and scheduled through Lakeflow Jobs, while users pay for compute rather than per-user licenses. The preview is available in all workspaces, although an administrator may need to enable Visual data prep in the preview portal.
Jason Messer, Emanuel Zgraggen, V Maharajh, Matt Jones, Tracy YangAre LLM agents good at join order optimization?
Join ordering is difficult because relational optimizers choose among exponentially many plans and depend on potentially inaccurate cardinality estimates. Databricks and UPenn researchers built an LLM agent that uses structured outputs to generate valid join reorderings and executes candidates through a tool returning runtime and intermediate subplan sizes. It can test up to 50 iterations, while the evaluation tested 15 rollouts per query across 113 Join Order Benchmark queries after duplicating each dataset row ten times. With a frontier model, the best tested orders improved geometric-mean latency by 1.288 times and reduced P90 latency by 41% versus the standard Databricks optimizer. A five-way example links its gain to LIKE predicates that are exceptionally difficult for cardinality estimators.
Eric Liang, Ryan Marcus, Sid Taneja, Yuhao ZhangHow conversational analytics removes the BI bottleneck
The interview presents conversational analytics as a way to move beyond fixed dashboards and turn governed data into decisions and workflows. Databricks’ Genie lets nontechnical users ask open-ended questions in plain language across Lakebase, Lakehouse, and federated sources, while Unity Catalog applies shared definitions and access controls. Lakebase provides transactional storage, instant provisioning, environment forking without copying underlying data, and scale described as billions or trillions of records; examples include real-time matchmaking, routing, and purchase decisions. The discussion stresses that human oversight, refreshed data, business context, and controls against hallucination remain necessary, citing Fox Sports’ chatbot deployment as an example. It concludes that organizations combining trusted semantics with operational data systems can make faster decisions and automate work, while the competitive gap may compound for slower adopters.
Catherine BrownHow to transform document activation workflows with Genie and Agent Bricks
Enterprise document workflows often require people to read PDFs, copy fields into spreadsheets, and re-enter data into business systems, creating delays, error risk, and weak auditability. The proposed alternative combines AI/BI Genie for natural-language queries over Delta tables, Agent Bricks for production agents, and Unity Catalog for permissions, lineage, and traceability. A five-phase workflow extracts fields into Lakeflow Bronze, Silver, and Gold layers, then supports structured queries, clause-level RAG answers, multi-agent routing, and actions through MCP servers. Examples include syncing validated rights data to ERP or CRM systems, updating billing entitlements, and triggering workflows from detected obligations. The approach targets media, advertising, ad tech, and telecommunications, where customers see improvements including faster month-end close, recovered revenue, reduced leakage, and lower operational risk.
Elena TesserBeyond the spreadsheet: How Databricks is delivering the modern CFO in financial services
Financial-services CFOs are being asked to act as Strategists and Catalysts, but fragmented legacy systems, T+1 batch processing, opaque reporting logic, and mismatched business semantics keep them focused on stewardship and retrospective operations. The proposed answer is Databricks as a unified, governed platform combining real-time streaming, centralized lineage, data and AI, with Unity Catalog, Lakeflow, Genie, and Agent Bricks addressing trust, latency, access, and model reproducibility. Unity Catalog can connect semantic definitions and end-to-end lineage from transactions through regulatory reports and models, while Lakeflow supports continuous ledger and liquidity processing. The post cites a global bank reducing liquidity-reporting processing from 10 hours to 8 minutes and Nationwide Insurance reporting a 5-point combined-ratio improvement and 3-point expense-ratio improvement. It presents a Lakehouse-based CFO stack as a shift from reporting historical results toward real-time capital management and previews AI-driven deposit and PPNR modeling.
Jennifer Miller, Marcela Granados, Andrea DeSosa, Alex Oberlander, Kim Hatton, Pavithra Rao, Naeem Rehman, Pravin Varma, Olga Deriy, Prasanna SelvarajAI App Development: Guide To Building AI-Powered Apps
Production-grade AI app development combines model orchestration, transactional state, governance, security, and live-data integration rather than only interface construction. The guide recommends defining users, outcomes, success metrics, timelines, and AI-relevant journeys, then shipping must-have features before enhancements. It evaluates builders by end-to-end support, technical audience, platform integration, code export, CI/CD, data connections, pricing, and deployment limits. For Databricks teams, it describes Databricks Apps, Declarative Automation Bundles, and Lakebase: serverless app compute, version-controlled deployment, and managed PostgreSQL with synced Unity Catalog Delta tables. It also recommends monitoring outputs, collecting correction feedback, refining prompts with automated evaluations, and governing models through MLflow, concluding that integrated infrastructure can leave teams focused on production AI workflows.
Databricks StaffIoT in Manufacturing: Strategy, Components, Use Cases, and Challenges
IoT in manufacturing is presented as an operational architecture for collecting machine data, improving production and supply-chain visibility, and preventing equipment failures under strict latency, reliability, and safety requirements. The guide covers device selection, sensor placement, tiered sampling, edge-to-cloud pipelines, platform criteria, security, compliance, workforce training, and a phased deployment roadmap. Its technical split sends high-frequency vibration and acoustic signals to edge gateways for local processing while cloud systems support stateful analytics, predictive maintenance scoring, and model training; protocols such as MQTT, OPC-UA, AMQP, and Modbus are evaluation criteria. Reported examples include multi-factory OEE monitoring across more than 200 production lines, reducing reporting lag from 24 hours to under five minutes, digital twins for maintenance simulation, and IoT logistics routing. The action plan recommends starting with predictive maintenance and OEE monitoring on one line before scaling.
Databricks StaffMultimodal Data Integration: Production Architectures for Healthcare AI
Production healthcare AI often depends on combining genomics, imaging, clinical notes, and wearables, but separate modality stacks can leave initiatives stalled before deployment. A lakehouse pattern addresses this by landing each modality in governed Delta tables under Unity Catalog controls while supporting cross-modal features, lineage, time travel, and reproducible training sets. Genomics processing can use Glow on Spark, imaging embeddings can feed governed tables and vector search, notes can become temporally aware entities, and wearables can flow through Lakeflow Spark Declarative Pipelines for streaming feature windows. The post compares early, intermediate, late, and attention-based fusion, matching them to dimensionality, temporal, and availability constraints, with late fusion presented as a practical baseline when modalities are missing. It argues that modality masking, sparse attention, and transfer learning are recommended responses because incomplete data is the default and connects the architecture to governed tumor board review, faster cohort assembly, and more clinical workflows.
Maks KhomutskyiStop hand-coding change data capture pipelines
Hand-coded change data capture (CDC) and slowly changing dimension (SCD) pipelines become fragile as they handle out-of-order updates, duplicate events, deletes, late-arriving data, retries, backfills, and schema evolution. The post presents AutoCDC in Lakeflow Spark Declarative Pipelines as a declarative alternative: engineers specify keys, sequencing, delete conditions, and SCD Type 1 or Type 2 semantics while the platform manages ordering, state, deduplication, versioning, and incremental processing. Examples show a current-state table that keeps Mercedes’s latest city and removes deleted user 123, alongside a Type 2 table that preserves validity windows through __START_AT and __END_AT. The supplied comparison reports roughly 6–10 lines of declarative definition versus 40–200+ lines of custom logic, and says AutoCDC supports both change data feeds and snapshot-based CDC.
Matt Jones, Zoé Durand, Phoebe Weiser, Bilal Aslam, Ray ZhuPersonalization Strategies for Media Companies
Media companies are expected to tailor content and messaging across channels, with personalization programs typically targeting lower churn, deeper sessions, and higher subscription revenue. The proposed foundation starts with an audit of CRM, email, warehouse, and customer data platform sources, followed by required-field definitions, ownership mapping, consent tagging, and ingestion-quality checks. Teams then map segment behaviors and conversion or retention paths, including negative signals, before selecting a CDP with real-time ingestion, identity resolution, unified profiles, segmentation, activation, and governance. Identity rules should connect anonymous and authenticated activity across web, mobile, and connected TV, while event-level streaming delivers messages through email, push, in-app, and advertising channels with minimal latency. The roadmap recommends an MVP comprising unified ingestion, a homepage recommendation surface, and triggered re-engagement email, with performance, churn, revenue lift, and model drift monitored through iteration.
Databricks StaffA Modern AI Risk Management Framework
AI systems introduce probabilistic behavior, model drift, adversarial manipulation, bias, and auditability challenges that traditional IT risk practices may not address. The framework described combines NIST AI RMF, EU AI Act, and ISO/IEC 23894:2023, using Govern, Map, Measure, and Manage as its operational structure while recognizing regulatory and societal context. It recommends cross-functional governance, clear risk ownership, an AI-BOM covering systems, data flows, dependencies, and accountability, plus continuous measurement of fairness, explainability, data quality, security vulnerabilities, and harm likelihood and severity. Across the AI lifecycle, the text identifies risks including data poisoning, model drift, malicious library injection, prompt injection, hallucinations, and platform vulnerabilities, and points to the AI RMF Playbook for checklists, timelines, and governance artifacts.
Databricks StaffMachine Learning Solutions: A Complete Implementation Guide
The guide presents machine learning delivery as a lifecycle problem, arguing that failures arise more often from planning, scoping, communication, and expectations than from algorithmic limitations. It recommends assessing data readiness before development by inventorying sources, checking quality and coverage, planning labeling, and engineering features. Once a measurable business outcome is defined, teams should match the learning paradigm and model complexity to the problem, establish cross-validation and holdout or temporal validation, and estimate compute and storage early. Baselines, reproducible experiments, deployment monitoring, model versioning, CI/CD, and retraining workflows support the transition from prototype to production. The guide also calls for representative proof of concepts and A/B testing or control groups to attribute business impact before scaling.
Databricks StaffData Governance Platforms: Evaluation & Feature Guide
This guide presents a framework for evaluating data governance platforms for enterprise deployment, distinguishing governance—the policies, roles, and controls for data use—from data management, the operational execution of those policies. It recommends assessing metadata-centered capabilities including continuous data profiling, catalog search and enrichment, end-to-end lineage, RBAC and ABAC, sensitive-data detection, masking, audit trails, compliance reporting, and data-subject request workflows. Vendor assessment should also cover pre-built connectors, REST APIs and SDKs, near-real-time metadata synchronization, schema-drift handling, cross-cloud federation, usability, support, implementation timelines, and three-to-five-year total cost of ownership. The proposed decision process selects three leading candidates, pilots them with representative structured and unstructured datasets, defines quality, lineage, adoption, and exit metrics, and uses executive review before procurement; governance is framed as an ongoing program that expands with AI workloads and regulatory change.
Databricks Staff