Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
- 542 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
ABAC row filtering and column masking policies, governed tags, and data classification are now generally available in Unity Catalog
Unity Catalog now generally offers three complementary data-governance capabilities: Attribute-Based Access Control (ABAC) policies for row filtering and column masking, Governed Tags, and agentic Data Classification. They address per-object access rules, coordination gaps, and manual detection by letting governance teams define tag-based policies once, automatically classify sensitive data, and protect matching objects across catalogs and schemas. Governed tags provide an account-level vocabulary inherited across catalogs, schemas, tables, and columns, while ABAC applies row filters and column masks using tag-based conditions. Classification uses built-in compliance classifiers, custom classifiers, metadata, pattern recognition, and large language models, with human-in-the-loop validation and false-positive exclusions. General availability adds 10x larger policy limits, support for 10,000+ policies per metastore, lifecycle management through SQL, APIs, UI, and Terraform, expanded compliance coverage, and custom classifiers in beta.
How CFOs in consulting can recover margin with Databricks
Consulting firms are facing margin pressure despite growing deal pipelines, with 2024 EBITDA margins at 9.8%, utilization at 68.9%, and revenue growth at 4.6%. The post attributes this to fragmented finance workflows spanning Salesforce, SAP, Workday, Concur, and spreadsheets, plus inconsistent metrics and manual reconciliation. It proposes using Databricks to create a governed data layer with Lakeflow, Lakehouse, and Unity Catalog, then adding Genie for natural-language analysis and finance apps that trigger workflows and write approvals back in real time. A global consulting firm reportedly cut cash forecasting cycles by 3–5 days and reduced reporting-cycle FTE hours by 80%, while Databricks’ own close fell from 15 days to 8. The proposed rollout spans 10–12 weeks and begins with one tightly scoped workflow such as revenue close, unbilled WIP, T&E leakage, or practice-margin analysis.
The Convergence of Open Table Formats and Open Catalogs: Catalog Commits is Generally Available
Catalog Commits is generally available for Unity Catalog managed Delta tables, aligning Delta with Iceberg’s catalog-oriented model and making the catalog responsible for table discovery, access, and latest table state. The change addresses three coordination problems: metadata “split brain” when engines write directly to storage, fragmented multi-engine governance, and the historical inability to coordinate atomic writes across multiple tables. With Catalog Commits enabled, Unity Catalog brokers table access through standardized APIs, keeping catalog and table state synchronized and enabling consistent authorization, holistic auditability, automated optimizations, and multi-statement, multi-table ACID transactions on Databricks. The release supports Databricks products and engines including Delta Spark, Delta Flink, Starburst Trino, DuckDB, and StreamNative, while Delta Kernel provides a shared path for connector support.
Announcing Databricks student fellows
Databricks is launching the Student Fellows Program, an opportunity for university and college students interested in computer science, AI, and data engineering to become leaders in their campus communities. Fellows receive specialized training from Databricks experts, build hands-on skills with the Databricks platform, and organize campus activities such as hackathons, tech talks, and study groups while connecting peers with resources. The initiative also offers opportunities for free or discounted certification exam vouchers and practical experience intended to support applications for internships with Databricks, its customers, or partners. Student Fellows’ primary mission is to foster a community of learners and serve as a bridge between Databricks and their university communities, with the first cohort currently accepting applications.
Faster Queries and New Capabilities with the Open-Source Databricks JDBC Driver
The open-source Databricks JDBC driver 3.x and later releases address connectivity needs affecting dashboards, spreadsheets, applications, and other data workflows. Compared with the legacy 2.x driver, they deliver up to 30% faster retrieval of large query results while preserving backward compatibility. Architectural changes add full Arrow-based transfer for JDK 16+, a first-class asynchronous execution API, stream-based ingestion into Databricks Volumes, and integration with the Statement Execution API. The driver also supports stored procedures, multi-statement transactions, Unity Catalog metric views, query tags, geospatial and complex data types, plus built-in client telemetry for query latency, metrics, and errors. Its open-source ownership and Databricks maintenance are presented as enabling faster fixes, new features, transparency, contributions, and tighter platform integration.
Unlocking the Archives: Turning Unstructured Documents into a Searchable Database for Groundwater Discovery
MapAid and the Sudan Association for Archiving Knowledge (SUDAAK) needed to make nearly 700 scanned geological reports searchable and recover well data for MapAid’s WellMapr groundwater models. Databricks built a Databricks-only pipeline that renders pages as images, uses multimodal AI Functions for page classification and schema-constrained JSON, and applies targeted sampling to cut classification processing by more than 70%. Water-relevant documents receive page-by-page OCR through the Foundation Model API, with well and borehole identifiers linking information across pages before structured records capture locations, depths, static water levels, and yields. In its first full run, 654 documents and 5,570 pages were classified in under three hours; 95% received excellent or good evaluations, about half contained water data, and 299 well or borehole records were extracted.
Predictive quality starts where defect detection stops
Manufacturing quality teams often receive defect-rate reports after the conditions behind them have changed, because inspection, supplier, and environmental data are disconnected. Predictive quality combines production, inspection, and supplier data with machine learning to forecast defects before final inspection, shifting quality management from reactive documentation to proactive intervention. Databricks Genie is presented as a natural-language interface for querying those sources together, including questions about first-pass yield, supplier lots, root-cause contributors, and process conditions; its answers include citations and can surface unusual patterns. The described capabilities include contextual understanding of terms such as NCR, CAPA, and CPK threshold, multi-source reasoning, and traceable outputs tied to records. The intended outcome is faster analysis and earlier action to reduce scrap before its cost is incurred.
Retail markdown optimization: from reactive markdowns to proactive
Retail markdown optimization addresses the gap between changing demand signals and delayed merchandising action. The post defines it as strategically reducing prices on slow-moving or end-of-life inventory, using demand forecasts, sell-through rates, weeks of supply, and price-elasticity models to choose which SKUs to mark down, when to start, how deep to discount, and where to act. It presents Databricks Genie as a natural-language interface across e-commerce, store, and wholesale data, with supplier, margin, and historical-pattern context; one example is Coop’s Microsoft Teams assistant, AskCap, which reported a 30% retention rate among internal users. Earlier detection of sell-through deceleration can give merchants more time to protect margin, adjust open-to-buy, and reallocate capital, while Genie supports decisions rather than making them.
Using MemAlign to Improve Evaluation of Traditional Machine Learning in Genie Code
Genie Code’s traditional machine learning workflows are difficult to evaluate because notebook quality depends on code, best practices, and adaptation to data and task. The team built an evaluation pipeline with nine LLM judges, each scoring applicable dimensions—including data exploration, imputation, training, MLflow logging, and cell organization—from 1 to 3, using test cases across classification, regression, and forecasting. On 50 human-graded notebooks, initial judge alignment was weak in several areas, including model training (MAE 0.680) and model use (0.562), with positivity bias contributing to missed technical flaws. Applying MLflow’s MemAlign, which combines generalized semantic guidelines with retrieved episodic examples, reduced judge error by 74–89% on the most important dimensions; removing episodic memory weakened most gains, while data exploration showed a nonsignificant regression.
How Superhuman and Databricks built a 200K QPS inference platform together
Superhuman replaced a DIY vLLM-based serving stack for its grammar-correction model with Databricks model serving, targeting peak traffic above 200,000 QPS, sub-second P99 latency, four-nines reliability, and no quality regression. The platform combines an Endpoint Discovery Service with power-of-two-choices load balancing, concurrency-based autoscaling, and asymmetric scale policies to reduce hotspots and handle rapid traffic ramps. Lazy-loading container images changed startup from several minutes to a few seconds by fetching metadata first and retrieving 4MB blocks on demand. Runtime work on H100 GPUs raised per-pod throughput from 750 to 1,200 QPS: FP8 weight quantization contributed up to 30%, while a multiprocessing server added 20%, alongside C++ and asynchronous scheduling optimizations. The teams validated changes through shadow testing and evaluation harnesses, and Superhuman is migrating additional models and workflows to the platform.
Addressing HR's widening capacity gap with AI
HR teams face a widening capacity gap as strategic expectations, complex employee issues, workforce volatility, skills shortages, and demands for personalized support collide with largely unchanged headcount and tools. The article presents AI transformation as an incremental journey: first establish a secure Employee 360 from structured and unstructured enterprise data, then build reusable workforce insights, augment workflows with human oversight, and progress toward broader transformation. It emphasizes data governance, including access controls, auditing, quality, standardization, and reliable interpretations, while noting that trust has limited AI’s business impact so far. MathCo and Databricks support this roadmap through NucliOS, whose Data Studio, AI Studio, and Decision Studio environments connect governed data, explainable models, feedback loops, and decision applications; Databricks supplies the lakehouse foundation, lineage, quality checks, and privacy-compliant access.
MCP Marketplace brings real-time intelligence to agentic applications
The MCP Marketplace connects agentic applications with real-time external intelligence alongside enterprise data. The problem appears in use cases such as loan approval, where historical internal records omit market conditions, updated credit signals, property changes, and competitor activity, making manual research a bottleneck. Databricks Marketplace provides governed access to MCP servers from You.com, Moody’s, and Cotality, while Unity Catalog authenticates connections and tracks access and lineage; Lakebase stores state, decisions, and audit trails across multi-step workflows. Examples show agents combining internal data with web research, credit ratings and sector outlooks, or property-resolution and mortgage signals before surfacing decisions for human review, including a commercial-loan flow with recorded sources, timestamps, and approver.
Pushing the Frontier for Data Agents with Genie
Genie is Databricks’ data agent for complex questions across structured enterprise assets—tables, dashboards, and notebooks—and unstructured sources including workspace files, Google Drive, and Sharepoint. Unlike coding agents operating in static environments, it must discover relevant assets at enterprise scale, determine authoritative knowledge from potentially contradictory sources, and handle questions without verifiable tests or guaranteed answers. It combines specialized knowledge search using semantic context and metadata, parallel thinking across sampled trajectories, and Multi-LLM orchestration with optimized prompts for distinct sub-agents. On an internal benchmark of real-world data-analysis tasks, these techniques raised accuracy from 32% to over 90% against a leading coding agent while reducing cost and latency; specialized search alone improved table-discovery performance by up to 40%, while parallel thinking added latency and token costs before further optimization.
Energy trading analytics in a real-time market
Energy trading markets settle every 15 minutes, while many analytics environments still rely on nightly batch pipelines, creating latency between changing conditions and trading decisions. The post presents Databricks Genie as a conversational interface that lets traders and portfolio managers query actual market data, weather integrations, historical position data, and current books. Its example combines a node’s 90-day natural-gas basis distribution with a Northeast temperature anomaly, while the described system can query real-time and historical data together. Genie also includes settlements, curtailments, and ancillary-service data, with governed access so users see only authorized information. The conclusion is that faster data access can remove information latency and compound margin without replacing trading expertise; the source states that Genie is available today.
First-party audience data is the ad sales relationship now
The post argues that media companies face a stronger need to turn first-party audience data into an advertising sales advantage as third-party cookies decline and buyer attribution expectations rise. Advertisers are buying audiences rather than inventory, so sales teams need granular audience insights, validated campaign performance, clear attribution, and revenue context to compete in RFPs. Databricks Genie is presented as a natural-language interface to governed first-party data, allowing leaders to query defined audience segments such as users who completed at least 75% of automotive content in 90 days, segmented by income quartile and household ownership. The same analytical environment connects demographic composition, content and behavioral signals, post-campaign measurement, attribution, and revenue pacing against budget and prior year.
Operating room utilization is hiding in your scheduling data
Operating room utilization measures in-room surgical minutes against allocated block-time minutes, yet most US health systems reportedly run at 65–75% versus an 80% industry target. Daily performance reports arrive the next morning, after schedules are set and opportunities to release unused blocks, redeploy staff, or backfill add-on cases may have passed. The post presents Databricks Genie as a natural-language interface for querying scheduling, utilization, and outcomes data without a data analyst request. Its proposed analytical environment combines scheduling data, actual case logs, block-release records, contribution margin, and staffing costs, with breakdowns by surgeon, service line, facility, and day of week. Genie surfaces specific intervention targets, although the post says it provides data access rather than automating OR management.
Why telecom churn prediction misses the intervention window
Telecom churn programs often intervene after customers have already shifted behavior, contacted support, or decided to leave, even though earlier signals exist in operational data. The post frames the gap as organizational: propensity models may be sophisticated, but retention leaders need timely, specific answers about high-value customers, likely triggers, and historically effective interventions. Databricks Genie is presented as a natural-language interface over customer behavioral and commercial data, able to surface targets such as premium postpaid customers with usage declines above 20%, recent support contacts, and contracts ending within 90 days. Its described capabilities combine usage, support, billing, network experience, competitive tenure, and intervention history while supporting segment and individual analysis. The proposed operating model prioritizes interventions by customer lifetime value and aims to act early enough for retention efforts to change outcomes.
Growth analytics is what comes after growth hacking
Growth analytics is presented as the next stage after growth hacking, arguing that competitive acquisition now depends on precise unit economics, cohort quality, and rapid conversion optimization rather than isolated tactics. It distinguishes product analytics, which explains feature use and user flows, from growth analytics, which connects acquisition sources and costs with activation, revenue, retention, and cohort behavior. The stated bottleneck is fragmented tooling: answering questions such as 90-day LTV by channel correlated with seven-day activation can require manually joining multiple systems and delay weekly budget decisions. Databricks Genie is described as a governed natural-language interface over unified acquisition, behavioral, and billing data, supporting attribution-to-LTV analysis, payback modeling, and paid-versus-organic comparisons. The text reports a 50% relative acquisition-rate lift, from 8% to 12%, for customers using Genie, and shorter onboarding insight cycles from months to weeks.
Real-world evidence for medical affairs: who can actually use it?
Real-World Evidence (RWE) is increasingly sought by payers, providers, and regulators, but the post distinguishes it from abundant Real-World Data (RWD), which must undergo rigorous study design, analysis, and interpretation to become credible evidence. It identifies four Medical Affairs use cases: regulatory submissions and commitments, payer formulary discussions, rapid HCP scientific exchange, and internal pipeline or portfolio decisions. The stated operational problem is that teams with claims, EHR, registry, and other assets often lack the fluency or capacity to answer complex questions within competitive and regulatory timelines. Databricks Genie is presented as a natural-language interface that can query unified RWE assets, including a treatment-initiation and 12-month-persistence example that reportedly surfaces in seconds rather than requiring several days of data-science work. The post also describes governance, logged and attributable requests, treatment-pathway awareness, and access controls for MSL support.
Wealth advisor productivity starts with the client conversation
Wealth management client reviews are intended to turn portfolio information into advisory conversations, but substantial meeting time is spent confirming allocations, benchmark performance, and major positions. Preparing for these meetings requires advisors to combine portfolio, tax, estate, and goal-related information from multiple systems for each client, often under time pressure. Databricks Genie is presented as a conversational way to query client portfolio data in real time, including a tax-loss-harvesting question that combines unrealized losses with an estimated current-year tax rate. Its described capabilities include unified holdings, transaction, tax-lot, and profile data; household-level analysis; authorization controls; and external market context, with the stated aim of reducing information preparation so advisors can conduct deeper, more personalized conversations.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.