Loading…
Databricks
161 posts about Databricks. Every summary links to the original.
What Is Serverless PostgreSQL?
Serverless PostgreSQL is presented as a fully managed cloud database model that decouples compute and storage, allowing each to scale independently with demand. Traditional deployments require teams to size infrastructure, manually manage scaling, and absorb costs from idle capacity. In serverless systems, the provider provisions compute on demand, can suspend it when idle, and bills according to active usage; scale-to-zero may introduce cold-start latency. The architecture can also support database branching through copy-on-write, creating isolated environments without duplicating data. The article distinguishes this model from lakebase architecture, which combines transactional and analytical workloads on a shared foundation using decoupled compute, durable object storage, log-based storage systems, and orchestration.
Databricks StaffHow Daikin Applied Americas builds consistent data pipelines at scale with Genie Code
Daikin Applied Americas needed to scale reliable data pipelines across growing analytics and AI use cases involving operational, manufacturing, and service data while coordinating development across teams. It adopted Databricks Genie Code within a structured operating model, using Unity Catalog context, reusable MECE skills, and explicit checkpoints across Bronze, Silver, and Gold layers to guide planning and execution. The framework defines competencies such as source grain, transformation patterns, canonical alignment, governance, and business-entity modeling, moving standards out of long prompts and into the development environment. The team reports that pipelines that once took days to prototype could be generated in minutes, with faster iteration, more consistent outputs, less structural correction, reduced architectural drift, and greater trust in AI-assisted results.
Trent Lezer, James VanGordonWhat if the answer was already in your data?
Kythera Labs is building an AI-native healthcare strategy platform on Databricks to give health systems access to strategic intelligence that historically required specialized analysts or consulting firms. Its foundation converts 339 billion medical and prescription claims covering more than 300 million patients into governed, event-based data, resolving providers, harmonizing codes across 130 vocabularies, and reconstructing patient journeys. Healthcare Strategy Agent, built with Agent Bricks, lets executives ask questions such as where oncology referrals are going and receive analyses of leakage, competing providers, physicians, and reimbursement opportunity in minutes. A Louisiana health system went live within ten days and reported 150% greater visibility into encounters, 12% more keepage, 22% less leakage, and $3.8 million in estimated annualized retained-encounter value. Unity Catalog, Lakebase, Delta Lake, Delta Sharing, and serverless infrastructure provide shared governance, lineage, access controls, and operational integration.
Bryan Smith, Morgan Wilkie, Kaitlin RyanDatabricks positioned highest in execution and furthest in vision for the second consecutive year in Gartner Magic Quadrant
Databricks says Gartner positioned it highest for execution and furthest for vision in the Magic Quadrant for the second consecutive year. The post connects this recognition to a category reclassified from “Data Science and Machine Learning” to “AI Platforms for Data Science and Machine Learning,” and argues that agentic applications require enterprise data, governance, observability, and business context. Databricks presents a unified approach combining the lakehouse, Lakebase, Agent Bricks, Unity Catalog, and Unity AI Gateway to build, monitor, and govern agents, models, data, apps, and tools. Reported examples include YipitData’s 20x increase in company coverage with 92–95% tagging accuracy, Block’s unified AI and data estate, and Novo Nordisk’s attribution of more than $157 million in net new value to governed clinical-trial optimization.
Craig Wiley, Kasey Uhlenhuth, Kayli Berlin, Cynthya PeranandamGenesis Workbench: A blueprint for industry AI in life sciences, powered by Databricks and NVIDIA
Genesis Workbench is an open blueprint for a Databricks-native life-sciences application that brings computational drug-discovery stages into one governed workbench. It combines Unity Catalog governance, MLflow tracking, Model Serving, serverless GPU compute, Databricks AI Search, and NVIDIA technologies including CUDA-X libraries, Parabricks, BioNeMo tools, GenMol, and Proteina-Complexa. Independent modules cover genomics, single-cell analysis, large- and small-molecule workflows, and model fine-tuning, with handoffs spanning gene-to-sequence resolution, structure prediction, docking, ADMET, and candidate ranking. A point-and-click React interface supports bench scientists, while declarative workflow generation and MCP exposure let pipelines and external clients use the workbench; inference runs on GPU endpoints inside the governed workspace without runtime external API dependencies. The stated aim is to let teams move from disease hypotheses to ranked therapeutic candidates on their own data, with a roadmap for automated workflow generation, BioNeMo Skills integration, and additional MCP services.
Mark Lee, Srijit NairData scientists: Powering the future of AI and analytics
Data scientists connect statistics, programming, and domain knowledge to turn raw data into models, recommendations, and decisions with business consequences. The role now extends beyond classical modeling to large language models, generative AI applications, agentic systems, and production workflows involving deployment, monitoring, and retraining. Modern practice combines Python, SQL, and libraries such as scikit-learn, PyTorch, Spark, and MLflow with data engineering basics, communication, and domain expertise. Data scientists contribute across problem framing, data access, preparation, feature engineering, experimentation, deployment, and lifecycle monitoring, with governed access and lineage helping align training data with production data. The article concludes that AI assistants and agents can automate routine work, but human judgment remains necessary for trustworthy evaluation, business framing, and actionable recommendations.
Databricks StaffHow Stagwell built privacy-safe ID matching on Databricks
Stagwell built a Databricks Marketplace App to let brands match first-party records against its Identity Spine without sending raw customer data outside their own workspaces. It combines Databricks Clean Rooms, Unity Catalog, Jobs and Notebooks, Delta Sharing, and a React/Express interface. Workspace OAuth and On-Behalf-Of authorization enforce the user's Unity Catalog permissions, while separate service principals and a scoped personal access token support app and clean-room operations. Within the clean room, a packaged notebook joins brand data with the Identity Spine, resolves multiple identifiers, and computes match rates, coverage metrics, and household and consumer IDs; results include aggregated audience insights for activation, while raw customer data and proprietary matching code remain separated.
Sridhar Sundaresan, Suvan KaulWhat is Row-Level Security?
Row-level security (RLS) is a database access control that filters which table rows a user may read or change according to identity, role, or session context. It supports multi-tenant SaaS, regional segregation, departmental access, compliance, healthcare, and partner use cases, while typically layering table-level grants with column-level security or data masking. The database applies a policy or predicate automatically at query time across BI dashboards, notebooks, ad-hoc SQL, APIs, and third-party tools, with separate USING and WITH CHECK rules for reads and writes. On Databricks, Unity Catalog row filters use SQL user-defined functions and apply across Databricks SQL, notebooks, jobs, and connected BI tools. The source recommends simple predicates, indexed keying columns, identity-based testing, and explicit enforcement for both reads and writes.
Databricks StaffDesign Beautiful Dashboards in AI/BI
This guide presents dashboard design as a way to apply brand identity consistently while helping viewers interpret business data and trust the result. It recommends choosing a 3-, 4-, or 6-column subdivision of AI/BI’s 12-column grid according to audience needs, then using F- or Z-pattern hierarchy and unequal widget sizes to prioritize information. The visual system combines readable sans-serif or local fonts, high-contrast text, neutral backgrounds, a 60-30-10 color allocation, and distinct interactive and visualization palettes. For the Wanderbricks example, the final five-color palette is #15AFDD, #2375A8, #52A870, #C85070, and #C89930, tested in light and dark modes and with color-blindness simulations; workspace themes extend the resulting styling across dashboards.
Maggie Li, Mohit HingoraniWhat’s new in Databricks Data + AI Platform security and compliance at Data + AI Summit 2026
At Data + AI Summit 2026, Databricks announced security and compliance capabilities for scaling Genie, Lakebase, serverless workloads, and AI-powered applications without relying solely on manual provisioning, static network controls, or siloed compliance programs. Automatic Identity Management (AIM) for Microsoft Entra ID is generally available on AWS and Google Cloud, AIM for Okta is in Public Preview, and Context-Based Ingress is in Public Preview across all three clouds for policies based on network source, identity, and access scope. Private Network Gateway, in Private Preview on Azure Databricks, provides one secure connection from serverless workloads to private networks, while expanded Private Link support extends to Lakebase and other services. Compliance additions include Azure Serverless coverage, HITRUST across AWS, Azure, and Google Cloud, ISMAP on Azure and AWS, expanded AWS GovCloud availability, and planned FedRAMP High support on Azure Commercial.
Jason Wu, Samrat Ray, Filippo Seracini, Alex Esibov, Vijay Raja, Kelly Albano, Robert Zhang, Mia Penfold LopezEnabling Governed Vibe Coding for Enterprise Apps on Databricks
Databricks introduces three capabilities intended to bring vibe coding to enterprise applications, where speed alone does not provide business-data context, deployment safety, or cost control. App Spaces lets admins define resource and data access, on-behalf-of-user API scopes, and security policies for groups of apps, with each app inheriting those settings. Genie App Builder turns plain-language descriptions into working internal apps through generated plans, live previews, AppKit, and awareness of workspace data assets and Unity Catalog semantics. Serverless micro apps run in isolated lightweight virtual machines, start quickly when needed, scale to zero while idle, and use usage-based rather than reserved-capacity infrastructure. Together, the capabilities are presented as a way for business-proximate users to build on enterprise data while organizations apply consistent governance and support broader app portfolios; all three are coming to Databricks Apps, with private previews coming soon.
Evan Pandya, Justin DeBrabant, Cong XuAnnouncing Apps on Databricks Marketplace
Databricks announces the Public Preview of Apps on Databricks Marketplace, allowing customers to discover, install, and run third-party data and AI applications inside secure Databricks workspaces. The offering addresses procurement challenges involving data movement, lengthy security reviews, custom integrations, and fragmented identity management by bringing applications to the customer’s data. Installed apps run in isolated sandboxes within the consumer’s Databricks account, inherit Unity Catalog governance, and use dedicated serverless compute with consumer-controlled external access through Serverless Egress Gateway policies. Providers can publish closed-source containerized apps once for no-egress distribution without maintaining per-customer infrastructure, while applications connect natively to services including SQL Warehouse, Lakebase, Model Serving, and Foundation Model APIs. The Public Preview launches with 20 partners, and planned additions include bundled assets, provider analytics, and commercial monetization.
Tia Chang, Akram Chetibi, Harish Gaur, Stephen Orban, Mengxi ChenIntroducing OpenSharing SecureConnect
OpenSharing SecureConnect addresses the networking burden of sharing live data from provider storage behind private networks, where providers and recipients otherwise exchange firewall and egress details manually. It is a Databricks-managed proxy that routes recipient storage access through Databricks endpoints after a one-time provider setup, while the data remains in the provider’s bucket. Providers allowlist Databricks Serverless Data Plane endpoints and enable SecureConnect for a metastore; serverless recipients require no configuration, while classic and open recipients allowlist stable inbound IPs. Optional NCC provides private link connectivity, and mutual TLS is available for recipients. SecureConnect is in Public Preview and supports cross-region, cross-cloud sharing plus customer-managed and Databricks Default Storage.
Huey Han, William Chau, Harish GaurSciene AI Companion: building an autonomous Customer Success platform on Databricks
Sciene built AI Companion for Quartile’s Customer Success organization, where CSMs support more than 1,000 brands and previously spent substantial time preparing decks, reconstructing context, and investigating account changes. The platform addresses personalization at scale, high-volume content generation, and root-cause diagnosis by combining account data, CSM communication styles, company principles, and cross-domain business data. Its Email Hub cuts reply time from 15–30 minutes to about three minutes, Meeting Hub reduces preparation for 80+ slide decks from over two hours to around 10 minutes, and Account Flagging reduces diagnosis of flagged accounts from 30+ minutes to about five. Databricks provides the shared governed foundation: Delta Sharing supplies data without copies, Lakebase stores operational state, and SQL Warehouses serve analytical, AI, and operational workloads from the same tables. The design keeps CSMs responsible for judgment while giving them current context for customer interactions.
Renata Fencz, Solano Campos, Rodrigo Mohr, Ricardo MorandiniIntroducing Genie ZeroOps: Put your data and AI operations on autopilot
Genie ZeroOps is an autonomous background agent for monitoring and operating data and AI assets, including jobs, pipelines, tables, and ML models. It continuously detects visible and silent failures, uses Unity Catalog lineage and platform observability to assess root causes, generates remediation through development workflows, and verifies fixes in isolated sandboxes. These environments use shallow, zero-copy table clones, scoped permissions, and network isolation, so proposed changes run against real data without touching production or applying anything before approval. For ML workloads, the agent can diagnose degraded predictions, train a candidate on corrected features, evaluate it against the production model’s existing eval suite and criteria, and support live-traffic ramping when it is measurably better. Genie ZeroOps is entering private preview in the coming weeks, initially supporting jobs, pipelines, tables, and ML workloads; Apps and Lakebase databases are on the roadmap.
Bilal Aslam, Lennart Kats, Ray Zhu, Mike Del Balso, Ori ZoharAnnouncing the new Databricks Startup Program
The Databricks Startup Program has been updated for venture-backed, early-stage startups building with data and AI. Qualifying companies can receive up to $200,000 in credits across Databricks and Neon, along with hands-on technical guidance, partner and community access, and connections to go-to-market teams and founder communities. The program is aimed especially at startups that recently raised institutional funding from pre-seed through Series A, and it is intended to provide an app backend, data, and AI stack from idea through product-market fit. Databricks and Neon are presented as providing a database and access to foundation models on day one, while Databricks supplies analytics, data warehousing, AI systems, and enterprise-grade governance as companies launch. Applications are available through the Databricks Startup Program.
Brad Van Vugt, Arjun RajeswaranUnifying Data and Governance in the Agentic Era: What’s New with Azure Databricks
At Data + AI Summit 2026, Azure Databricks announced capabilities aimed at moving enterprises from experimental AI pilots to production-grade automated workflows by unifying data, productivity tools, marketing, and governance on Azure. Its Agentic Data foundation introduces LTAP, combining analytical data, streaming pipelines, and live application transactions in one lakehouse storage copy; Lakebase adds a managed serverless Postgres engine with copy-on-write branching, while Lakehouse//RT targets millisecond responses for high-concurrency workloads. Genie integrations for Microsoft Teams, M365 Copilot, Excel, and SharePoint bring governed lakehouse intelligence and ingestion into daily work, alongside tools for agents, applications, pipelines, and autonomous operations. CustomerLake adds Profile Agents and Campaign Agents for customer profiles and personalization, while Genie Ontology and Unity AI Gateway provide semantic context, rate limits, content filtering, and spend controls.
Isaac Gritz, Toussaint Webb, Ben Tripp, Kiriana StukasAccelerate search queries with full-text search indexes on Databricks
Databricks introduces full-text search indexes in Beta on Databricks Runtime 18.2 to accelerate substring and keyword queries on large open-format tables without changing their layouts. The indexes tokenize text columns into a compact lookup structure mapping tokens to matching rows; at query time, the engine uses it to identify candidate files and skip most of the table. They are maintained asynchronously, require no query hints, preserve complete results when stale by scanning indexed and non-indexed data as needed, and support Unity Catalog managed Delta and Iceberg tables on serverless and classic compute. A Trust and Safety team reported a substring search running more than 100x faster on a petabyte-scale table, while Liquid clustering remains complementary because it optimizes column-value filters rather than text within fields.
Yu Xu, Yingyi Bu, Ivan VezilićIntroducing CustomerLake: The Agentic CDP embedded in Databricks
Databricks announces CustomerLake, an Agentic Customer Data Platform embedded natively in its lakehouse, bringing Customer 360, identity resolution, audience building, campaign automation, activation, and personalization alongside governed data and AI models. The announcement addresses fragmented identities, stale audiences, manual campaign workflows, and the duplication and governance burden created by separate martech systems. CustomerLake uses Unity Catalog and Lakehouse Federation to access customer data across Databricks, Snowflake, Google BigQuery, cloud object storage, operational databases, and other enterprise systems, while Profile Agents create business-ready profiles and Campaign Agents build audiences, recommend actions, activate channels, and optimize engagement. Its operating model is described as embedded, democratized, and autonomous, with Agentic Identity Resolution combining deterministic, probabilistic, and agentic workflows. CustomerLake is now available in Private Preview and launches with an open partner ecosystem.
Tasso Argyros, Justin DeBrabant, Michael Trapani, Dan Morris, Katy YuanWhat is customer segmentation?
Customer segmentation divides an existing customer base into smaller groups based on shared demographic, geographic, psychographic, behavioral, firmographic or value-based characteristics. Unlike market segmentation, it uses first-party data about existing customers, and a customer can belong to multiple segments at once. The guide distinguishes rule-based, survey-based, RFM, k-means, decision-tree and AI/ML-driven methods, with choices depending on data maturity and business goals. Effective implementation defines an objective, audits and unifies sources, resolves duplicate identities, selects a method, validates segments and measures outcomes. It also describes Databricks' CustomerLake capabilities, including governed Customer 360, agentic identity resolution, natural-language segmentation through Genie and bidirectional activation connectors; stated benefits include improved retention, conversion, customer lifetime value and marketing efficiency.
Databricks Staff
