Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
- 542 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
Data Warehouse Types: A Complete Guide to Architectures and Use Cases
A data warehouse is a centralized repository for structured data, supporting complex queries, reporting, and business intelligence rather than transaction processing. The guide compares architectures by scale, latency, cost, scope, ownership, and governance. Enterprise data warehouses integrate organization-wide sources through ETL, apply cleansing and validation, and provide a governed source of truth, while data marts focus on departmental analysis and may be dependent or independent. Operational Data Stores replicate current or recent operational data for reporting refreshed from minutes to hours, whereas virtual, cloud, hybrid, and lakehouse designs trade physical consolidation, scalability, flexibility, and governance differently. The comparison also frames lakehouses as combining open-format data lake flexibility with warehouse-style governance and transactional reliability.
What is Row-Level Security?
Row-level security (RLS) is a database access control that filters which table rows a user may read or change according to identity, role, or session context. It supports multi-tenant SaaS, regional segregation, departmental access, compliance, healthcare, and partner use cases, while typically layering table-level grants with column-level security or data masking. The database applies a policy or predicate automatically at query time across BI dashboards, notebooks, ad-hoc SQL, APIs, and third-party tools, with separate USING and WITH CHECK rules for reads and writes. On Databricks, Unity Catalog row filters use SQL user-defined functions and apply across Databricks SQL, notebooks, jobs, and connected BI tools. The source recommends simple predicates, indexed keying columns, identity-based testing, and explicit enforcement for both reads and writes.
Payment Fraud Detection: How Banks and Businesses Stop Fraudulent Transactions
Payment fraud detection is presented as a real-time practice for identifying and blocking unauthorized transactions amid evolving card-not-present, account takeover, authorised push payment, friendly, and card-testing attacks. Modern systems combine predefined rules, machine learning, behavioral analytics, device fingerprints, geolocation, transaction history, and velocity signals to calculate risk scores within milliseconds, declining or routing high-risk payments to review. Rules are fast and interpretable but require continual tuning, while machine-learning and behavioral models can identify complex or user-specific patterns that static thresholds miss. The source also emphasizes prevention controls such as MFA, tokenization, 3D Secure, and device binding, alongside unified real-time data infrastructure that supports cross-channel signals, continuous retraining, and deployment into payment authorization flows.
What is an AI Agent Harness?
An AI agent harness is the software infrastructure around a large language model (LLM) that lets it act on tasks rather than only answer prompts. The model reasons and chooses actions, while the harness connects tools, execution environments, memory, external systems and safety controls. Many agents follow a reason–act–observe loop: the model proposes an action, the harness executes it, captures the result and returns that result as context for the next decision. Production harnesses typically include system prompts, tool execution, sandboxes, filesystems and durable storage, context management, feedback and self-verification, guardrails, human-in-the-loop approvals, and observability; their design increasingly shapes reliability as model capabilities converge.
Databricks and NVIDIA: Building for the Agentic Era
Databricks and NVIDIA describe a full-stack partnership for enterprise AI, spanning governed model training, inference, agent infrastructure, and industry-specific AI. Databricks AI Runtime brings NVIDIA Hopper GPUs and Quantum InfiniBand to governed data for distributed training, with Blackwell preparation, Free Edition GPU support, and planned NGC container support. Model Serving uses NVIDIA hardware and Triton Inference Server for low-latency, high-throughput production inference, while NVIDIA Vera targets agent harnesses, tool calls, analytics, and multi-step orchestration with up to 3x faster SQL queries and 80% faster agentic performance. The collaboration also places NVIDIA Agent Toolkit on Databricks Apps, integrates GPU debugging through Genie Code, and brings frameworks such as BioNeMo, Parabricks, cuOpt, and Isaac Sim to Databricks workflows. The stated vision is an end-to-end stack in which GPUs handle models and purpose-built CPUs handle surrounding agentic workloads, with governance supplied by the Databricks platform.
Data Warehouse Modernization: Roadmap, Architecture, and Services
Data warehouse modernization addresses legacy systems that cannot scale efficiently with growing data volumes, real-time analytics, machine learning, and self-service access. The proposed roadmap spans two to four years for large estates, moving from assessment and architecture design through high-impact workload migration, governance embedding, and optimization rather than relying on a risky big-bang cutover. Its target architecture favors a lakehouse or enhanced cloud data warehouse, with open formats such as Apache Iceberg or Delta Lake, separate compute and storage, and Bronze, Silver, and Gold layers supporting incremental ELT and lineage. The source says modernization can reduce infrastructure maintenance costs by 30–50%, compress query latency from hours to seconds, and halve redundant ETL pipelines, while also improving governance for sensitive data and enabling BI, machine learning, and generative AI workloads on a shared foundation.
The Partner Well-Architected Framework: What's New and What's Next
The Partner Well-Architected Framework (PWAF) has moved partner guidance from static PDFs to AI-ready guidance spanning Built-On, Connected, and Data Collaboration architectures. It combines an architecture center with technical standards, patterns, and instrumentation guidance intended to align integrations with partner-engineering validation while making adoption and DBU impact measurable. Since February, Databricks has added a Dev Kit containing 15+ tested AI-developed skills, new or refreshed guidance for Clean Rooms, software-defined storage, Marketplace apps, Genie, Lakebase, and MCP server onboarding, and an open-source Firefly Analytics reference implementation. The framework is positioned as an evolving, agent-compatible way to shift routine integration work to AI tools, freeing partner and Databricks engineers to focus on complex architecture; more patterns, skills, reference implementations, and demos are planned.
What’s coming next to Free Edition
Databricks is expanding Free Edition, launched twelve months ago to help people learn modern data and AI, after providing more than $10M in credits to over 500,000 learners. The five additions are Genie Code, serverless GPUs subject to availability, Lakebase, Agent Bricks, and Lakeflow Designer. Genie Code can generate, run, interpret, and refine code autonomously for tasks such as dataset analysis, pipeline cleaning, and visualization, while GPUs support deep-learning workloads including training, fine-tuning, inference, and large-dataset processing. Lakebase adds a fully managed Postgres-compatible database for data apps and AI agents; Agent Bricks supplies composable tools, memory, orchestration, and evaluation; and Lakeflow Designer enables visual pipeline construction. Together, these additions are presented as a complete free toolkit covering data engineering, analytics, machine learning, app development, and AI agents, allowing users to build end-to-end data and AI projects in Free Edition.
Becoming the most comprehensive data & AI ecosystem on earth
Databricks describes a year of expanding its partner ecosystem for independent software vendors and data providers, responding to requests for stronger go-to-market support, technical guidance, and Marketplace transactions. It says it introduced partner tiering, the Partner Well-Architected Framework, a Marketplace Commit Drawdown pilot, and the ability to list Databricks Apps and Genie Agents. Under the pilot, customers with a Universal Commit can submit invoices for eligible solutions that run on or share data to Databricks; validated purchases reduce their commitment obligation, while Databricks compensates its sales team. OpenSharing is presented as an open-source, vendor-agnostic protocol for cross-cloud, cross-platform sharing of structured and unstructured data, volumes, agent skills, AI models, and semantics. The post says Apps adoption has grown fivefold since last year's summit, with more than 5,000 accounts running Apps in production weekly, and that more than 20,000 customers rely on Databricks.
Design Beautiful Dashboards in AI/BI
This guide presents dashboard design as a way to apply brand identity consistently while helping viewers interpret business data and trust the result. It recommends choosing a 3-, 4-, or 6-column subdivision of AI/BI’s 12-column grid according to audience needs, then using F- or Z-pattern hierarchy and unequal widget sizes to prioritize information. The visual system combines readable sans-serif or local fonts, high-contrast text, neutral backgrounds, a 60-30-10 color allocation, and distinct interactive and visualization palettes. For the Wanderbricks example, the final five-color palette is #15AFDD, #2375A8, #52A870, #C85070, and #C89930, tested in light and dark modes and with color-blindness simulations; workspace themes extend the resulting styling across dashboards.
What’s new in Genie Code at Data + AI Summit 2026
At Data + AI Summit 2026, Databricks announced expansions to Genie Code for complex, agentic data and ML work. The changes include a full-page command center for managing concurrent threads and assets, upgrades across production ML engineering, and scheduled tasks that run prompts while users are away. For ML workflows, Genie Code uses Databricks production expertise and Genie Ontology, integrates with MLflow and Model Serving, and can move GPU jobs to AI Runtime while using workspace environment features. It can write features, coordinate edits, run and debug code, compare candidates, inspect endpoint health, and diagnose issues, with users deciding what to keep. Scheduled tasks are described as coming soon, creating reviewable threads from prompts and optional Databricks assets.
What’s new in Databricks Data + AI Platform security and compliance at Data + AI Summit 2026
At Data + AI Summit 2026, Databricks announced security and compliance capabilities for scaling Genie, Lakebase, serverless workloads, and AI-powered applications without relying solely on manual provisioning, static network controls, or siloed compliance programs. Automatic Identity Management (AIM) for Microsoft Entra ID is generally available on AWS and Google Cloud, AIM for Okta is in Public Preview, and Context-Based Ingress is in Public Preview across all three clouds for policies based on network source, identity, and access scope. Private Network Gateway, in Private Preview on Azure Databricks, provides one secure connection from serverless workloads to private networks, while expanded Private Link support extends to Lakebase and other services. Compliance additions include Azure Serverless coverage, HITRUST across AWS, Azure, and Google Cloud, ISMAP on Azure and AWS, expanded AWS GovCloud availability, and planned FedRAMP High support on Azure Commercial.
Building an open ecosystem for AI governance with Unity AI Gateway
Databricks announced the Unity AI Gateway partner ecosystem, extending enterprise AI governance beyond models to runtime interactions among models, agents, MCP servers, skills, and AI tools. Built on Unity Catalog, the gateway lets organizations apply policies, monitor activity, manage spend, and govern AI across providers and frameworks, while integrating security, identity, and governance products they already use. The announcement groups the integrations into runtime AI security, observability and guardrails; agent identity and access governance; and AI observability and risk monitoring. Named integrations include Alice, CrowdStrike Falcon AI Detection and Response, Cyera, HiddenLayer, Netskope, Noma Security, Obsidian Security, Openlayer, Okta, Ping Identity, SailPoint, and Saviynt, with described capabilities including prompt-injection detection, data-loss prevention, agent discovery, authorization, and lifecycle governance.
What’s New in the AI Platform: Agents for ML Engineering, Our Deep Learning Platform, and New Capabilities for Real-Time ML
The announcement presents three additions to the Databricks AI Platform: Genie Code support for ML engineering, AI Runtime’s serverless GPU environment, and expanded real-time ML capabilities. Genie Code integrates with Unity Catalog, Feature Store, training, serving, monitoring, and MLflow, assisting with feature engineering, model training, deployment, evaluation, and production operations. AI Runtime provides on-demand serverless NVIDIA A10 and H100 GPUs, supports high-performance multinode training with RDMA and high-performance data loading, and adds Lakeflow Jobs, DABs, MLflow, and Unity Catalog integration. For real-time ML, the platform adds declarative feature engineering, streaming features, online feature serving on Lakebase, and enhanced Model Serving targeting 300K+ QPS with under 10ms p99 latency overhead. Reported customer examples include faster workflows, lower infrastructure costs, and production scaling beyond 100K QPS.
Introducing the Agentic CDP: A New Species of CDP for a New Era of Agents
Traditional customer data platforms (CDPs) were built for human-managed, batch-based campaigns, but the post argues that agentic buying requires millisecond speed, hyper-personalization, and richer context. It contrasts the familiar Golden Record with Golden Context, which combines customer data with current business goals and the history and outcomes of prior decisions. The proposed Agentic CDP uses “Infinity Campaigns,” always-on engagement loops that use LLMs and agents to adapt messaging, timing, and channels for individuals. It is also embedded in the data foundation, bringing customer, business, and decision context together under existing governance, and is designed for agents and humans from the outset. Databricks presents CustomerLake as an implementation of these principles for its platform.
What is data pipeline architecture?
Data pipeline architecture is the end-to-end blueprint for collecting, processing, storing and delivering data from source systems to people, applications and models. It distinguishes logical design, which defines stages and responsibilities, from physical design, which assigns tools and infrastructure to those stages. The common four layers are ingestion, processing and transformation, storage, and serving and consumption, with orchestration and observability spanning the pipeline. The brief compares batch, streaming, Lambda, Kappa and medallion patterns, explaining trade-offs involving freshness, cost, complexity and operational burden. It also contrasts ETL with ELT and presents governance, monitoring and right-sized processing as reliability practices, concluding that architecture should match the use case and balance freshness, cost and reliability.
Enabling Governed Vibe Coding for Enterprise Apps on Databricks
Databricks introduces three capabilities intended to bring vibe coding to enterprise applications, where speed alone does not provide business-data context, deployment safety, or cost control. App Spaces lets admins define resource and data access, on-behalf-of-user API scopes, and security policies for groups of apps, with each app inheriting those settings. Genie App Builder turns plain-language descriptions into working internal apps through generated plans, live previews, AppKit, and awareness of workspace data assets and Unity Catalog semantics. Serverless micro apps run in isolated lightweight virtual machines, start quickly when needed, scale to zero while idle, and use usage-based rather than reserved-capacity infrastructure. Together, the capabilities are presented as a way for business-proximate users to build on enterprise data while organizations apply consistent governance and support broader app portfolios; all three are coming to Databricks Apps, with private previews coming soon.
Introducing OpenSharing: the Next Evolution of Delta Sharing for the Agentic Era
OpenSharing is presented as the next evolution of Delta Sharing, extending an open zero-copy data-sharing protocol from tables and files to models, agents, semantic context, unstructured data, and reusable AI logic. The protocol is now an independent open-source project hosted by the Linux Foundation, while Databricks OpenSharing adds Unity Catalog governance and audit logging, Marketplace discoverability, and enterprise features. Genie Agent Sharing supports governed AI experiences across organizational boundaries, with controls for proprietary instructions, data access, daily prompt quotas, and row exports. SecureConnect removes per-recipient firewall changes through a Databricks-managed proxy, while Global Distribution uses local replicas to reduce egress fees and latency. The launch also supports Apache Iceberg REST Catalog API, external catalogs, and on-premises storage partners; providers define shares in Unity Catalog, recipients query live data through existing tools, and governance enforces access controls.
Announcing Apps on Databricks Marketplace
Databricks announces the Public Preview of Apps on Databricks Marketplace, allowing customers to discover, install, and run third-party data and AI applications inside secure Databricks workspaces. The offering addresses procurement challenges involving data movement, lengthy security reviews, custom integrations, and fragmented identity management by bringing applications to the customer’s data. Installed apps run in isolated sandboxes within the consumer’s Databricks account, inherit Unity Catalog governance, and use dedicated serverless compute with consumer-controlled external access through Serverless Egress Gateway policies. Providers can publish closed-source containerized apps once for no-egress distribution without maintaining per-customer infrastructure, while applications connect natively to services including SQL Warehouse, Lakebase, Model Serving, and Foundation Model APIs. The Public Preview launches with 20 partners, and planned additions include bundled assets, provider analytics, and commercial monetization.
Introducing OpenSharing SecureConnect
OpenSharing SecureConnect addresses the networking burden of sharing live data from provider storage behind private networks, where providers and recipients otherwise exchange firewall and egress details manually. It is a Databricks-managed proxy that routes recipient storage access through Databricks endpoints after a one-time provider setup, while the data remains in the provider’s bucket. Providers allowlist Databricks Serverless Data Plane endpoints and enable SecureConnect for a metastore; serverless recipients require no configuration, while classic and open recipients allowlist stable inbound IPs. Optional NCC provides private link connectivity, and mutual TLS is available for recipients. SecureConnect is in Public Preview and supports cross-region, cross-cloud sharing plus customer-managed and Databricks Default Storage.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.