Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
- 542 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
Manufacturing runs on capital. Finance protects the margin.
Manufacturing finance must protect margin by keeping capital moving through inventory, receivables, and plant equipment. The post argues that volatile supply chains, shifting demand, rising costs, and increasingly influential agents make that work faster and more complex, while an estimated $1.7T remains trapped in excess working capital across large US companies. It presents ontology as a way to preserve the meaning and context behind figures, including plants, SKUs, customer terms, and changing business conditions. Databricks Genie is described as a data-smart AI coworker whose ontology learns from business systems and questions, produces sourced and governed answers, and helps finance identify trapped inventory cash, aging receivables, and underperforming assets. It readies actions such as releasing inventory, accelerating collections, or redeploying capital, while a person remains responsible for the decision.
Energy runs on volatile markets. Finance protects the margin.
Energy finance must protect margin as hourly power and fuel prices, shifting forward curves, contract settlements, hedges, and major grid and generation investments make business conditions more complex. The post argues that accurate figures are not necessarily correct unless they retain context about the underlying asset, market, and contract, motivating a live ontology that keeps business meaning current. It presents Databricks Genie as a data-smart AI coworker that learns from enterprise systems and questions, returns sourced answers, preserves permissions, governs AI cost, and shows its work. The proposed uses cover live margin, revenue risk across PPAs and hedges, and funding AI-driven buildout, with a person making trading, dispatch, hedging, or capital decisions. Genie is described as helping finance identify risks early and prepare actions while keeping human owners accountable.
Bringing real-time fraud prevention to government benefits
Federal benefits programs face an estimated $233 billion to $521 billion in annual fraud losses and roughly $186 billion in improper payments in fiscal 2025, while pre-payment vetting can delay legitimate aid. The post presents real-time detection as an alternative to “pay and chase,” combining rules engines, machine learning, adaptive and generative AI, entity resolution, behavioral analysis, and graph analytics to score claims before disbursement. It says Databricks can unify these layers and enable cross-agency sharing through OpenSharing, data masking, and Clean Rooms while retaining governance and data ownership. When fraud is detected, the platform can hold payments, add confirmed fraudsters to Do Not Pay, and assemble statute-linked evidence packets with digitally signed audit trails, while designated personnel authorize referrals. The source concludes that this model is ready for implementation at national scale, citing Databricks’ stated processing of 160 billion card payments annually in milliseconds.
Agents for production lines: Trusted decisions in real time
ProdLine CoPilot addresses in-shift production-line decisions when equipment faults threaten output and data is split across PLCs, SCADA, MES, ERP, and LIMS. The proposed system streams operational technology data into Databricks, joins it with slower business and quality records, and routes questions to domain specialists that calculate recovery, depletion, quality-risk, and maintenance signals. Its orchestrator reads current Unity Catalog state, uses SQL, Genie, AI Search, Model Serving, and Python-based tools, while MILP, stochastic scheduling, Monte Carlo, Bayesian risk, and Pareto methods provide analytical outputs. Recommendations are tested against 1,000 scheduling scenarios and returned with trade-offs in cost, overtime, throughput, quality, and service, but execution remains with line managers, quality leads, and maintenance. The demo drafts work orders, quality deviations, and schedule updates for approval, with traceability storing inputs, assumptions, constraints, approvers, and outcomes.
How agentic AI can help telecom finance teams protect the margin when every moment matters
Telecom finance teams face revenue leakage when services go unbilled or uncollected, fraud and partner-settlement errors drain charges, and customer churn removes future spending. The source argues that fragmented ordering, billing, ERP, and spreadsheet data makes month-end reconciliation too slow, turning discrepancies found weeks later into write-offs rather than recoverable revenue. It presents Databricks Genie as a data-smart AI coworker whose ontology captures business meaning across systems, stays current, and grounds sourced answers in governed data. Genie can answer billing, invoice, spend, and variance questions, surface anomalies, and support action while a person remains responsible for decisions; Lumen Technologies is cited as using it across the Office of the CFO. The text says Genie does not make billing, fraud, or pricing decisions.
How NorthStar Anesthesia built a scheduling app for a workforce of 3,000 clinicians in weeks
NorthStar Anesthesia needed a better way for roughly 3,000 physicians and Certified Registered Nurse Anesthetists (CRNAs) to coordinate shifts, because its commercial scheduling platform hid colleagues’ time-off data and its dashboard pilot was not mobile-friendly. Synaptiq built a React TypeScript scheduling app with one software engineer and deployed it through Databricks Apps in a few weeks, using the existing Databricks data foundation, governance, and Microsoft Entra ID single sign-on. The app provides color-coded views by clinician type, facility selection, day/week/month calendars, shift notes, search and filtering, and time-off visibility, with data refreshing every 30 minutes. Daily unique users rose from 75–80 at launch to more than 110 as the first clinician group migrated, while users reported less stress; planned additions include natural-language queries, push notifications, and automated staffing reports.
The audience is the asset. Media finance teams need to understand them to protect the margin.
Media finance teams are tasked with understanding how audience value flows across subscriptions, advertising, and content investment so the business can protect margin. Streaming has turned one wholesale audience into multiple monetization strategies, while ad-supported tiers now account for 59% of new streaming sign-ups, making timely measurement and pricing analysis more important. Ontology preserves the meaning and context of figures as audiences, titles, channels, and business conditions change, distinguishing a correct answer from one that is merely accurate. Databricks Genie is described as a governed, data-smart AI coworker that answers sourced natural-language questions, learns from interactions, and shows its work; Genie-powered apps let hundreds of DIRECTV analysts and leaders query more than 1,200 customer-level attributes, while people retain decision authority.
Chief data officer: role, responsibilities, and career guide
The chief data officer (CDO) is a senior executive who treats enterprise data as a strategic asset, overseeing data strategy, management, quality, governance, and its use in analytics and AI. The role sits between business strategy, information technology, and data science, while distinguishing the CDO’s focus on data from the CIO’s responsibility for technology infrastructure. Responsibilities span the data lifecycle, governance frameworks, access controls, quality metrics, business intelligence, analytics prioritization, and organizational data capability. The role has shifted from compliance-focused stewardship toward enterprise AI strategy, with enablement-oriented governance making reliable, AI-ready data accessible while protecting its integrity. The guide also presents executive authority, data literacy, and leadership judgment as important conditions for delivering measurable business value.
From prototype to production: High QPS for Databricks AI Search
Databricks AI Search now offers generally available high-QPS scaling for Standard endpoints, addressing production workloads such as search bars, recommendations, and real-time entity resolution. Operators set a human-readable target_qps value when creating or updating an endpoint through the Python SDK, REST API, or UI, while Databricks provisions the required compute without manual replica counts, node sizing, or load balancers. Existing Unity Catalog governance and Delta Sync remain in place, and scaling progress is exposed through scaling_info as it moves from SCALING_CHANGE_IN_PROGRESS to SCALING_CHANGE_APPLIED; endpoint observability shows requests per second, latency, and health. Service principal authentication is recommended for high-QPS traffic, while personal access tokens are capped at a few tens of QPS, and new capacity applies when an index is created or synced.
How Databricks manages its own coding agent spend with Unity AI Gateway Budgets
Databricks describes how it manages coding-agent costs as thousands of engineers use Claude Code, Codex, Cursor, and other tools, creating exposure to runaway automation and growing R&D spend. The company routes all agent traffic through Unity AI Gateway Budgets and separates short-term runaway-spend protection from long-term monthly spend governance. A daily budget triggers self-service acknowledgement through Slack, an internal portal, or the CLI, while a high monthly limit uses manager-approved, project-scoped tiers that expire. Both budgets apply simultaneously, so effective usage is capped by the lower of the month-to-date total plus one runaway increment and the monthly maximum. Centralizing metering also gives managers and finance shared usage data, and the company reports that approval queues disappeared, monthly requests became rare, and engineers stopped rationing usage.
Get Started with Genie One: Top AI Cowork Use Cases for Business Users
Genie One is presented as a data-smart, agentic coworker for business users who need work completed across calendars, CRMs, data warehouses, ticketing systems, and shared documents rather than through a standalone chatbot. The guide explains four workflows: recurring business reviews, meeting preparation and follow-up, knowledge-work document automation, and operational monitoring with alerts. Users connect relevant sources, provide templates or natural-language rules, create reusable skills, test outputs, schedule routines, and refine instructions over time; the monitoring workflow can compare metrics with defined thresholds and email alerts. The post recommends starting with one narrow recurring task, connecting its required sources securely with data or IT partners, and iterating toward a library of saved skills that can be used across the organization.
The EU Digital Product Passport: a traceability deadline
The EU’s Ecodesign for Sustainable Products Regulation (ESPR) will require in-scope products sold in the single market to carry a machine-readable Digital Product Passport (DPP), with batteries first from February 2027. The passport links a unique identifier and data carrier to lifecycle information that must remain accurate, while operators retain responsibility for data stored on their own backend, tiered access, supplier inputs, and any Registry registration. Implementation is primarily a data-integration and governance problem, spanning tier-N suppliers, per-unit operational records, lineage, analytics, AI, and controlled sharing rather than QR-code generation. Databricks components including Lakebase, Unity Catalog, Lakeflow Spark Declarative Pipelines, Databricks Apps, and OpenSharing are presented with a battery-focused Solution Accelerator as an operator-side reference architecture. The stated benefits extend beyond market access to faster recalls, sourcing-risk visibility, and lineage-based sustainability reporting, but technology alone does not establish compliance.
AI customer service: strategy, agents, and solutions guide
AI customer service combines natural language processing, machine learning, predictive analytics, generative AI, and automation to route requests, generate responses, and support resolution across channels. The guide distinguishes AI agents from scripted chatbots: agents reason over context, call external tools, and complete multi-step tasks such as checking shipping data and issuing a partial refund. It presents AI as a layer that absorbs routine volume while human agents retain judgment over complex or sensitive cases, citing potential gains including more than 70% query automation, up to 30% lower operating costs, and a 15% improvement in customer satisfaction. It recommends evaluating integration, decision transparency, production support, security, and compliance, then starting with one measurable, high-volume use case and expanding only after consistent resolution quality.
Building AI Literacy: Frameworks, Tools, and Practices
AI literacy is presented as the ability to understand how AI works, use its tools effectively, evaluate outputs critically, and recognize ethical implications. The guide organizes this fluency into functional, critical, and ethical domains, with progression from defining terms and crafting prompts to comparing tools, auditing outputs for bias, and designing human oversight. It explains that large language models generate probable continuations from statistical patterns in training data, so hallucinations and bias require verification against trusted sources rather than assuming factual correctness. For education and organizations, it recommends distributed curriculum or role-specific modules, hands-on assignments, measurable assessment checkpoints, governance policies, internal champions, iterative pilots, and longer-term impact metrics.
AI in supply chain: from demand forecasting to AI agents
AI in supply chain management applies machine learning, generative AI and AI agents to demand forecasting, inventory optimization, supplier-risk assessment and logistics orchestration. It connects ERP records, point-of-sale feeds, supplier communications and external signals to support continuous decision-making rather than periodic planning. The guide maps use cases to required data foundations, covering demand-sensing pilots, warehouse task prioritization, agent guardrails, cross-system APIs and generative-AI grounding. It recommends starting with a product category or region, comparing model output with a baseline, and expanding only when accuracy and bias improve consistently. Reported figures include up to 85% forecast-accuracy improvement, up to 15% lower inventory carrying costs, and 23% lower fulfillment costs among organizations that deployed AI at scale.
AI in healthcare: applications and best practices
Healthcare AI applies artificial intelligence, machine learning, deep learning, and generative AI to clinical, administrative, and research workflows, using patient data, EHRs, medical imaging, and clinical documentation to support decisions and operational efficiency. It distinguishes AI models from algorithms and generative models, and describes supervised classifiers, time-series, natural-language-processing, and computer-vision systems used across healthcare. The data section emphasizes standardized models, consistent coding, interoperability through APIs and FHIR, and unified data lakehouse architecture, noting reported EHR prediction accuracy of 70-72% that varies by condition, data quality, and population. For generative documentation, audio transcription, large language models, templates, and Retrieval-Augmented Generation can draft notes, but clinician review remains required because fluent output may contain factual errors. It also covers imaging AI, privacy practices, FDA and European AI Act obligations, and the expectation that clinical AI will augment rather than replace professional judgment.
AI Applications in Finance: A Practical Use Case Guide
AI in finance spans machine learning, natural language processing, and generative AI for credit scoring, fraud detection, algorithmic trading, finance automation, and decision support across banking, capital markets, and insurance. Finance teams rank use cases by revenue impact, risk reduction, and implementation effort, while data scientists clean and validate the underlying data. Credit scoring can combine traditional and alternative data, with confidence thresholds routing uncertain cases to human underwriters; trading strategies use historical backtests, monitoring, and versioned audit trails. Fraud systems monitor transactions in real time and prioritize alerts, while finance automation uses machine learning, rules-based logic, exception queues, and ERP integration. The guide recommends 90-to-120-day pilots with predefined metrics and ROI measurement before scaling, alongside explainable AI, model governance, and logged decisions for responsible deployment.
How the FDA Built an AI Platform That 85% of Its Staff Now Use Daily
The FDA built ELSA, a generative AI platform for its 16,000 staff, on Halo, a governed Databricks data foundation created to address fragmented systems across eight centers. Those centers had separate chatbots and data stores; consolidating 50 to 60 sources enabled faster sharing, real-time streaming, and centralized access controls through Unity Catalog. Within roughly two months, ELSA adoption rose from less than 1% to 85%, while staff began building hundreds of agents weekly from standard operating procedures, regulatory guidance, and center-specific documents. MCP servers layered over Unity Catalog make governed data and tooling accessible beyond data scientists, and Databricks ML and NLP capabilities through MLflow extracted starting materials and product-supplier-manufacturer relationships from millions of submission pages. A reviewer can now request grounded starting-material information for a drug application in about three minutes instead of days, while the FDA adapts center-specific MCP tools and extends the model across its organization.
Permission isn't purpose: Intent-based authorization in Omnigent
Omnigent’s intent-based authorization addresses a gap in identity-based access control: an agent with valid credentials may follow indirect prompt injections embedded in data and perform authorized actions unrelated to the user’s task. It binds each session to a human-declared purpose and evaluates that intent before every tool call, producing permitted, consent-required, or denied verdicts. In a data-quality example, reading a customers table is allowed, dashboard publication requires approval, and an injected request to grant external access is denied despite the agent’s identity permitting that tool. The intent is fixed at design time for autonomous agents or approved by a human at session start for interactive agents, and cannot be broadened by the agent. Omnigent combines this policy with session-risk scoring and other contextual policies in a single engine where any denial wins.
Provisioning for the Agentic Era: How Databricks Built a Self-Serve Infrastructure Vending Machine
Databricks built the Field Engineering Vending Machine (FEVM) to replace shared environments with isolated, centrally governed infrastructure for its growing field organization. Users or agents describe a use case and configuration details, receiving an environment assembled through a React frontend, Python backend, Databricks Apps, Terraform, Git Runner, and Lakebase across AWS, Azure, and GCP. FEVM supports templates and add-ons such as Lakebase, notebooks, and pre-packaged assets from UC Volumes, applies configurable time-to-live policies, and sends Slack notifications for provisioning, expiration, and deletion. It also manages shared resources independently, including catalogs that persist across workspace lifecycles, and provides administrative controls for limits and audits. At the time described, it managed more than 2,600 active deployments across three clouds and over 5,000 active users, after handling nearly 1,200 requests during one internal event.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.