Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
432 postsWebsite
Latest articles
How Deutsche Börse built a generative AI tool to tackle the large-scale migration of Zeppelin notebooks to Databricks
Deutsche Börse Group is migrating a high volume of Zeppelin notebooks used by more than 2,000 users from Cloudera to Databricks, after Cloudera’s 2027 Zeppelin decommissioning and the group’s move to cloud analytics. Because the notebooks combine SQL, Python, custom interpreters, data-source references, visualizations, widgets, and scheduling logic, the team rejected fully rule-based rewriting. Its Zeppelin to Databricks Notebook Converter, delivered as a Databricks App, deterministically maps paragraphs to cells, translates interpreter syntax, reformats metadata as .ipynb JSON, and preserves original content. A generated, environment-specific prompt then gives Genie context about interpreters, data sources, and configuration patterns so it can reconstruct logic, ask clarifying questions, and handle the variable work. The hybrid workflow reduced redevelopment to 15–20 minutes per notebook, while keeping SQL, Python, visualizations, widgets, Oracle and HDFS references, scheduling, and custom business logic untouched; large-scale testing is next.
Evan Pandya, Tobi Wole-FasanyaAnnouncing the Databricks analytics engineer learning pathway
Databricks is launching the Analytics Engineer Learning Pathway, a curriculum for SQL practitioners that covers data modeling, pipelines, metrics, and Genie spaces on the lakehouse. The pathway teaches learners to transform raw data into governed, AI-ready semantic models and metric views, the foundation for analytics, dashboards, and AI agents. Courses cover Analytics Fundamentals, production SQL ETL with Materialized Views, Streaming Tables, AUTO CDC, and Lakeflow Jobs, plus data modeling with Delta Lake and Unity Catalog. The curriculum also addresses metric views, Genie spaces, governance with Unity Catalog permissions and ABAC policies, and Spark Declarative Pipelines with expectations, event logs, and metrics. Courses are available in self-paced and instructor-led formats through Databricks Academy, with the full pathway included in active learning subscriptions.
Maroua Lazzarou, Pratyarth RaoThe question your commercial data should already be able to answer
In ATTR-CM, a progressive and often fatal form of heart failure, 70 to 80 percent of patients do not know they have the disease, creating a need for better commercial prioritization than static call lists provide. Databricks and Veeva position embedded Genie agents and AI/BI dashboards in Veeva Vault CRM as a way to connect commercial data bidirectionally and answer role-specific questions inside existing workflows. Sales reps can see geographic HCP views with suspected patient counts, formulary scores, office availability, NRx-weighted priorities, and talking points, then rebuild a day after a cancellation. MSLs can generate cited pre-call briefs from approved sources such as Veeva Link, PubMed, ClinicalTrials.gov, and ASNC guidelines. Territory managers receive personalized views of call patterns, unworked signals, and dormant HCPs, while Unity Catalog provides shared access, lineage, and compliance governance.
Christina Busmalis, Tony FariasPipelineIQ: Forward‑Looking Sales Intelligence That Drives Action
PipelineIQ addresses the administrative drain and unreliable revenue predictability caused by incomplete, inconsistent, and backward-looking CRM data in B2B sales. Rather than build another forecasting system, it applies prescriptive analytics to identify forward signals and produce immediate actions for reps and managers. Built on Databricks, it uses Foundation Model APIs, Unity Catalog, Delta Lake, and AI/BI Dashboards; its confidence scorer sends CRM fields to ai_query() with a Gemma 3 12B model, scores eight MEDDPICC dimensions from 0–10, and limits missing fields to scores of 3 or below. Weighted confidence is refreshed daily, with a fail-safe override to Low when a use case has more than three active blockers. Dashboards and Genie queries connect evidence-based risk explanations, remediation steps, and portfolio views to sales execution.
Sam Le Corre, Dael Williamson, Luis HerreraBackstage with Lakebase, part 2
Part 2 shows how moving Backstage’s operational Postgres database to Databricks Lakebase brings security, auditing, masking, and cost controls under Unity Catalog. The proof of concept used Lakehouse Federation to expose the catalog as the lakebase_bs foreign catalog, replacing cross-service RDS investigations with Unity Catalog grants and system.access.audit records. Audit events identify OAuth users and source IPs, while billing attributed 31.6130 DBU to production and 0.0107 DBU to a dropped test branch. Branch-level attribute masking propagates automatically to feature, CI, and QA copies, while the post cites a Perforce report stating that 60% of organizations experienced breaches or theft in non-production environments. It also introduces LakebaseOps, with three agents and seven scheduled jobs, and Lakebase MCP, exposing 46 governed tools across four profiles, so DBAs can design policies and workflows instead of handling repetitive provisioning.
Cameron Casher, Kevin Hartman, Surya Sai TuragaExpanded interoperability with Unity Catalog Open APIs
Unity Catalog’s expanded Open APIs address data silos by letting organizations use multiple compute engines while retaining centralized governance and a single copy of data. In beta, Apache Spark, Apache Flink, and DuckDB can create, read, write, and stream to or from UC managed Delta tables, with catalog commits providing serialized commits, transactional safety, and auditability. Delta Kernel, an open source Java and Rust library, abstracts low-level protocol details, helping connectors integrate external writes with catalog-managed commits while Predictive Optimization continues to run on accessed tables. Credential vending, now GA for tables, issues short-lived, scoped cloud credentials and supports M2M OAuth plus automatic refresh; volume credential vending is in Public Preview for unstructured data. The roadmap includes functionality for fine-grained row- and column-level ABAC on external reads, while external managed-table access remains in beta.
Alex Jiang, Tathagata DasFrom manual to autonomous: how AI agents are transforming electric grid operations
Electric utilities are facing rising demand, retiring generation, extreme weather, aging infrastructure, and fragmented operational data that manual processes cannot manage at scale. AI agents are presented as a human-centered alternative that synthesizes heterogeneous data, learns from outcomes, and progresses from human-approved recommendations to exception-based control and eventually autonomous operations within defined parameters. Hawaiian Electric used a Retrieval Augmented Generation proof-of-concept with Databricks AI Search, Unity Catalog, and Lakeflow Declarative Pipelines to query regulatory documents and provide page-specific citations. The system reduced response times from five minutes to five seconds and was implemented in two weeks, while the article describes broader potential for predictive maintenance, outage response, load forecasting, and customer service.
Julien Debard, Edward TavaresData quality is the AI strategy
NYU Langone Health’s AI strategy starts with data quality, arguing that healthcare AI cannot be reliable when source data is fragmented or inconsistent. The institution standardized on common transactional platforms, including one electronic health record and one ERP system, established authoritative data sources, and fixes data at the source rather than mapping it in the warehouse layer. Its Databricks-based unified data and AI platform, with Unity Catalog, supports clinicians, analysts, scientists, and corporate users across care, operations, and research, while real-time feeds power emergency-room decision-support models. Mherabi also describes a three-layer analytics model: structured visualizations, conversational tools such as Genie, and answers delivered in formats suited to the user. The stated conclusion is that upstream data discipline, governance, literacy, and adaptable platforms provide the foundation for trustworthy AI and timely clinical insight.
Aly McGueThe Rosetta stone of CPS: Claroty’s AI-powered library
CPS asset identification is difficult because 88% of CPS assets do not transmit an exact product code and 76% use product codes that differ from the vendor's official records. Claroty's AI-Powered CPS Library addresses this entity-resolution problem with a hybrid architecture that combines classic matching, statistical inference, generative AI, NLP, reasoning agents, and human-in-the-loop review to reconcile protocol data, vendor records, manuals, and vulnerability information. Its Databricks-based lakehouse uses Delta Lake, Unity Catalog, Spark pipelines, Model Serving, MLflow evaluations, Lakebase, and Databricks Apps to maintain governed, auditable mappings across a catalog of more than 17 million assets. The post reports 25% improved vulnerability-attribution accuracy and says early tests gave 56% of analyzed devices new or updated recommendations for outdated firmware, while low-confidence mappings remain subject to expert review.
Ben Hazan, Anton Berlinsky, Ohad Avni, Itay Wagner, Guy Zalcman, Dor Bdolach, Ravid Ariely, Gal SberroClinical operations intelligence belongs on the Lakehouse
The Site Feasibility Workbench addresses fragmented clinical operations data by placing a site-selection decision-support application, its models, and data on Databricks. Released as a fully open-source Databricks App, it combines a FastAPI backend, React frontend, Unity Catalog-governed data, Lakebase operational state, and AI/BI Genie within the workspace. TA-segmented LightGBM models use the organization’s CTMS, EDC, and IRT history alongside real-world evidence, patient access, qualification, Open Payments, and protocol-execution signals to rank sites. Each prediction stores SHAP attributions in a governed Unity Catalog Delta table, while MLflow tracks training runs and model versions; saved shortlists persist to Lakebase and Genie answers governed-data questions. The six-step workflow supports protocol selection through final shortlisting, including diversity considerations; the release is a decision-support layer rather than a source of record, and deployment reportedly takes about 30 minutes before security review and validation.
Nicholas Siebenlist, Surya Sai TuragaABAC row filtering and column masking policies, governed tags, and data classification are now generally available in Unity Catalog
Unity Catalog now generally offers three complementary data-governance capabilities: Attribute-Based Access Control (ABAC) policies for row filtering and column masking, Governed Tags, and agentic Data Classification. They address per-object access rules, coordination gaps, and manual detection by letting governance teams define tag-based policies once, automatically classify sensitive data, and protect matching objects across catalogs and schemas. Governed tags provide an account-level vocabulary inherited across catalogs, schemas, tables, and columns, while ABAC applies row filters and column masks using tag-based conditions. Classification uses built-in compliance classifiers, custom classifiers, metadata, pattern recognition, and large language models, with human-in-the-loop validation and false-positive exclusions. General availability adds 10x larger policy limits, support for 10,000+ policies per metastore, lifecycle management through SQL, APIs, UI, and Terraform, expanded compliance coverage, and custom classifiers in beta.
Adriana Ispas, Kristen Wilder, Jacqueline Li, Corey Sunwold, Menglei Sun, Viswesh PeriyasamyHow CFOs in consulting can recover margin with Databricks
Consulting firms are facing margin pressure despite growing deal pipelines, with 2024 EBITDA margins at 9.8%, utilization at 68.9%, and revenue growth at 4.6%. The post attributes this to fragmented finance workflows spanning Salesforce, SAP, Workday, Concur, and spreadsheets, plus inconsistent metrics and manual reconciliation. It proposes using Databricks to create a governed data layer with Lakeflow, Lakehouse, and Unity Catalog, then adding Genie for natural-language analysis and finance apps that trigger workflows and write approvals back in real time. A global consulting firm reportedly cut cash forecasting cycles by 3–5 days and reduced reporting-cycle FTE hours by 80%, while Databricks’ own close fell from 15 days to 8. The proposed rollout spans 10–12 weeks and begins with one tightly scoped workflow such as revenue close, unbilled WIP, T&E leakage, or practice-margin analysis.
Rajaram Suresh, Kateryna Savchyn, Pavithra Rao, Mark Hahn, Kim Hatton, Sharmey Shah, Lisa FoxThe Convergence of Open Table Formats and Open Catalogs: Catalog Commits is Generally Available
Catalog Commits is generally available for Unity Catalog managed Delta tables, aligning Delta with Iceberg’s catalog-oriented model and making the catalog responsible for table discovery, access, and latest table state. The change addresses three coordination problems: metadata “split brain” when engines write directly to storage, fragmented multi-engine governance, and the historical inability to coordinate atomic writes across multiple tables. With Catalog Commits enabled, Unity Catalog brokers table access through standardized APIs, keeping catalog and table state synchronized and enabling consistent authorization, holistic auditability, automated optimizations, and multi-statement, multi-table ACID transactions on Databricks. The release supports Databricks products and engines including Delta Spark, Delta Flink, Starburst Trino, DuckDB, and StreamNative, while Delta Kernel provides a shared path for connector support.
Benjamin Mathew, Michelle Leon, Lukas Rupprecht, Ryan JohnsonAnnouncing Databricks student fellows
Databricks is launching the Student Fellows Program, an opportunity for university and college students interested in computer science, AI, and data engineering to become leaders in their campus communities. Fellows receive specialized training from Databricks experts, build hands-on skills with the Databricks platform, and organize campus activities such as hackathons, tech talks, and study groups while connecting peers with resources. The initiative also offers opportunities for free or discounted certification exam vouchers and practical experience intended to support applications for internships with Databricks, its customers, or partners. Student Fellows’ primary mission is to foster a community of learners and serve as a bridge between Databricks and their university communities, with the first cohort currently accepting applications.
Trang Le, Elise Hollowed, Joe NashFaster Queries and New Capabilities with the Open-Source Databricks JDBC Driver
The open-source Databricks JDBC driver 3.x and later releases address connectivity needs affecting dashboards, spreadsheets, applications, and other data workflows. Compared with the legacy 2.x driver, they deliver up to 30% faster retrieval of large query results while preserving backward compatibility. Architectural changes add full Arrow-based transfer for JDK 16+, a first-class asynchronous execution API, stream-based ingestion into Databricks Volumes, and integration with the Statement Execution API. The driver also supports stored procedures, multi-statement transactions, Unity Catalog metric views, query tags, geospatial and complex data types, plus built-in client telemetry for query latency, metrics, and errors. Its open-source ownership and Databricks maintenance are presented as enabling faster fixes, new features, transparency, contributions, and tighter platform integration.
Toussaint Webb, Gopal Lal, Kaitlin BaumgardnerUnlocking the Archives: Turning Unstructured Documents into a Searchable Database for Groundwater Discovery
MapAid and the Sudan Association for Archiving Knowledge (SUDAAK) needed to make nearly 700 scanned geological reports searchable and recover well data for MapAid’s WellMapr groundwater models. Databricks built a Databricks-only pipeline that renders pages as images, uses multimodal AI Functions for page classification and schema-constrained JSON, and applies targeted sampling to cut classification processing by more than 70%. Water-relevant documents receive page-by-page OCR through the Foundation Model API, with well and borehole identifiers linking information across pages before structured records capture locations, depths, static water levels, and yields. In its first full run, 654 documents and 5,570 pages were classified in under three hours; 95% received excellent or good evaluations, about half contained water data, and 299 well or borehole records were extracted.
Andres David Blandon Restrepo, Mofeed NagibPredictive quality starts where defect detection stops
Manufacturing quality teams often receive defect-rate reports after the conditions behind them have changed, because inspection, supplier, and environmental data are disconnected. Predictive quality combines production, inspection, and supplier data with machine learning to forecast defects before final inspection, shifting quality management from reactive documentation to proactive intervention. Databricks Genie is presented as a natural-language interface for querying those sources together, including questions about first-pass yield, supplier lots, root-cause contributors, and process conditions; its answers include citations and can surface unusual patterns. The described capabilities include contextual understanding of terms such as NCR, CAPA, and CPK threshold, multi-source reasoning, and traceable outputs tied to records. The intended outcome is faster analysis and earlier action to reduce scrap before its cost is incurred.
Caitlin GordonRetail markdown optimization: from reactive markdowns to proactive
Retail markdown optimization addresses the gap between changing demand signals and delayed merchandising action. The post defines it as strategically reducing prices on slow-moving or end-of-life inventory, using demand forecasts, sell-through rates, weeks of supply, and price-elasticity models to choose which SKUs to mark down, when to start, how deep to discount, and where to act. It presents Databricks Genie as a natural-language interface across e-commerce, store, and wholesale data, with supplier, margin, and historical-pattern context; one example is Coop’s Microsoft Teams assistant, AskCap, which reported a 30% retention rate among internal users. Earlier detection of sell-through deceleration can give merchants more time to protect margin, adjust open-to-buy, and reallocate capital, while Genie supports decisions rather than making them.
Sarah DuffyUsing MemAlign to Improve Evaluation of Traditional Machine Learning in Genie Code
Genie Code’s traditional machine learning workflows are difficult to evaluate because notebook quality depends on code, best practices, and adaptation to data and task. The team built an evaluation pipeline with nine LLM judges, each scoring applicable dimensions—including data exploration, imputation, training, MLflow logging, and cell organization—from 1 to 3, using test cases across classification, regression, and forecasting. On 50 human-graded notebooks, initial judge alignment was weak in several areas, including model training (MAE 0.680) and model use (0.562), with positivity bias contributing to missed technical flaws. Applying MLflow’s MemAlign, which combines generalized semantic guidelines with retrieved episodic examples, reduced judge error by 74–89% on the most important dimensions; removing episodic memory weakened most gains, while data exploration showed a nonsignificant regression.
Stepan Nosov, Pavle Martinović, Tejas Sundaresan, Alkis Polyzotis, Nemanja PetrovicHow Superhuman and Databricks built a 200K QPS inference platform together
Superhuman replaced a DIY vLLM-based serving stack for its grammar-correction model with Databricks model serving, targeting peak traffic above 200,000 QPS, sub-second P99 latency, four-nines reliability, and no quality regression. The platform combines an Endpoint Discovery Service with power-of-two-choices load balancing, concurrency-based autoscaling, and asymmetric scale policies to reduce hotspots and handle rapid traffic ramps. Lazy-loading container images changed startup from several minutes to a few seconds by fetching metadata first and retrieving 4MB blocks on demand. Runtime work on H100 GPUs raised per-pod throughput from 750 to 1,200 QPS: FP8 weight quantization contributed up to 30%, while a multiprocessing server added 20%, alongside C++ and asynchronous scheduling optimizations. The teams validated changes through shadow testing and evaluation harnesses, and Superhuman is migrating additional models and workflows to the platform.
Myke Troianovskyi, Christoph Stüber, Wai Wu, Arjun DCunha, Amine El Helou, Tian Ouyang, Jarek Odzga, Alex Coleman