Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
432 postsWebsite
Latest articles
How nOps rebuilt their cloud optimization platform on Databricks Lakebase, and why other ISVs should too
nOps migrated its production cloud optimization platform to Databricks Lakebase, a fully managed PostgreSQL database integrated with the Databricks Lakehouse. Its previous setup separated customer-facing operational data from analytics, requiring scheduled jobs, cron-based change-detection logic, and synchronization work that introduced lag and operational overhead, especially as workloads expanded to GCP and Azure. In the new architecture, Lakebase is the central Postgres database and single source of truth for the front end and AI infrastructure, while the Lakehouse continuously consumes its data for analytics and metric computation. Databricks Metric Views are automatically surfaced in the application, and data flows one way without direct write-back. The reported results are faster data pipelines, lower operational overhead, and a better customer experience, with auto-scaling, auto-stop, restore, OAuth, and compatibility features supporting adoption.
Bryan SmithPeril predicts: Precision payouts for a volatile world
Parametric insurance pays automatically when objective thresholds—such as wind speed, rainfall, or earthquake magnitude—are met, replacing lengthy loss assessments with event-based payouts. Modern catastrophe modeling combines geospatial data, weather observations, engineering insights, and historical loss records to estimate extreme-event probability and impact and define reliable triggers. Operationalizing these programs requires near-real-time processing of satellite imagery, weather feeds, exposure datasets, and model outputs. Databricks’ Geospatial Lakehouse unifies those sources on Delta Lake while Spark runs spatial joins and catastrophe modeling pipelines. When thresholds are crossed, the system identifies eligible policies, calculates tiered payouts, and surfaces results through dashboards, Lakehouse Apps, and Genie; aerial imagery and multimodal AI can support damage validation and fraud detection, while Unity Catalog governs access and Delta Sharing supports controlled data exchange.
Anindita Mahapatra, Timo Roest, Justin MonaldoThe foundation of AI scalability: One team, one platform, one operating model
Albertsons Companies describes a centralized AI strategy for scaling decisions across merchandising, labor, supply chain, and customer experience across approximately 2,300 stores. The model combines one central AI core, the Databricks Data + AI Platform, and a shared operating model spanning data engineering, ML, governance, and analytics. Reusable ingestion pipelines, templates, feature-store patterns, model monitoring, performance observability, and governance wrappers support local execution, while a company-wide governance committee sets shared standards. Albertsons reports accepting 1.38 million lines of AI-generated code in nine months, with more than 90% of engineers using AI tools, and it provides low-code dashboards, prompt libraries, and conversational agent generation for nontechnical teams. Success is measured through reuse rates, time to deployment, responsible AI compliance, and business outcomes linked to AI uplift, with initiatives required to demonstrate impact before scaling.
Aly McGueThe federal data paradox: Rich in data, poor in access
Federal agencies have invested substantially in data infrastructure, but program directors, policy analysts, oversight officials, and budget examiners often still rely on technical intermediaries to answer operational questions. The post frames this as the unresolved “last mile” of federal data modernization: data lakes, APIs, dashboards, evidence-based policymaking mandates, and agency CDO functions have advanced infrastructure without making it usable by most decision-makers. It presents Databricks Genie as a natural-language interface that lets staff query agency data in plain language, including questions requiring joins across disbursement, eligibility, and geographic data, while retaining existing access controls and policies. Genie runs on Unity Catalog with role-based access controls, audit logging, and data lineage; it also supports federated cross-agency queries and records queries, answers, and sources, which the post associates with oversight, FOIA readiness, and accountability.
Kacey HertanDriving Budapest forward: How BKK uses Databricks to transform city mobility
BKK, Budapest’s unified transport authority, manages public transit, shared mobility, infrastructure, and traffic systems for a city of 1.7 million people. Its legacy on-premises warehouse and fragmented Excel- and PowerBI-based reporting struggled with growing, varied data, limiting efficient access and analysis. BKK phased a migration to Azure Databricks, starting with mobility datasets and carefully modeling and validating data from vehicle GPS, passenger sensors, boarding schedules, and other systems. The Databricks Lakehouse lets analysts process large geospatial datasets in shared SQL, Python, or R notebooks, supporting minute-level shared-mobility tracking, route-performance analysis, airport-bus forecasting, and dynamic scheduling. These capabilities provide faster insights and more responsive decisions, while governed access and cost visibility support broader collaboration and BKK’s vision for a digital twin of Budapest’s mobility system.
Max von Münster, Estilla Híves, Istvan Viz, Engin ErzenginLLM Vs AI: A Practical Guide to Differences, Use Cases, and Tools
This guide distinguishes artificial intelligence, a broad field covering systems that perform tasks associated with human intelligence, from large language models (LLMs), specialized generative AI models for language and code. It places LLMs within generative AI and deep learning, and explains how transformer self-attention processes tokens while training data, parameter scale, and context windows shape capability and limitations. Examples include content drafting, code generation, translation, customer-service chatbots, sentiment analysis, extraction, summarization, and tool-connected agentic workflows, while traditional machine learning remains suited to structured labels and numeric predictions. It recommends retrieval-augmented generation, human review, bias testing, privacy controls, and evidence-based pilots with defined workflows, metrics, budgets, realistic data, and logged outputs.
Databricks StaffModel risk governance is not the same as risk intelligence
Financial institutions have invested heavily in model governance frameworks, stress testing infrastructure, limit monitoring, data feeds, and dashboards, but risk leaders may still lack fast access to what those models are telling them. When a CRO must assess credit concentration, scenario sensitivity, or relationships between market positions and credit exposures, answering can require navigation across model outputs, analyst interpretation, and disconnected data systems. The proposed approach uses Databricks AI/BI Genie to let leaders query risk data in natural language, while retaining access controls, audit logging, Unity Catalog lineage, cross-risk data, and stress-test outputs in one environment. The stated distinction is that governance establishes necessary controls, whereas conversational risk intelligence supports questions that fixed dashboards did not anticipate, including comparisons with internal limits.
Kim HattonGenerative AI for Business: A Complete Strategy and Implementation Guide
Generative AI is presented as an enterprise technology shift with potential value across customer operations, marketing and sales, software engineering, and research and development, which together are described as accounting for approximately 75% of use-case value across industries. The guide distinguishes generative systems, which create text, images, code, audio, or structured data from predictive models, and explains how foundation models and LLMs support open-ended business applications. Its implementation strategy prioritizes reliable data infrastructure, high-impact low-complexity pilots, and governance covering sensitive data, model limitations, monitoring, human review, and compliance. Recommended execution includes a cross-functional squad, predefined KPIs, user training, baseline measurements, and a 90-day executive review with scale, iterate, or discontinue decisions. Retrieval-augmented generation is identified as the most widely adopted way to reduce hallucinations by grounding responses in verified proprietary data, while ROI is expected within six to twelve months for a well-structured pilot.
Databricks StaffData Science vs Data Engineering: Choosing Analysis or Infrastructure
This guide distinguishes data engineering from data science by their roles in the data value chain. Data engineers build and maintain ingestion pipelines, warehouses, lakes, access controls, and production systems, using tools such as SQL, Python, Scala, Java, Apache Spark, Kafka, orchestration platforms, and cloud infrastructure. Data scientists work with clean, accessible data to perform exploratory analysis, engineer features, train and validate statistical or machine-learning models, and communicate findings through visualization and storytelling. The roles collaborate through feedback on data quality, shared documentation, model deployment infrastructure, and versioned schemas, feature pipelines, and experiments. For choosing a path, the guide recommends trying a two-week pipeline project and a two-week ML project; it also outlines portfolio deliverables for each and says both roles require strong analytical and computer-science fundamentals.
Databricks StaffAI Applications: Tools, Use Cases, and Platforms
The guide maps AI applications for data scientists, machine learning engineers, and technical leaders, covering predictive AI, generative AI, conversational AI, and autonomous agents across consumer, developer, and enterprise settings. It distinguishes consumer-facing tools from developer platforms and describes production concerns including model lifecycle management, vector search, data lineage, deployment, monitoring, governance, and evaluation. Generative systems create text, images, code, audio, and video from prompts, while large language models and mixture-of-experts architectures are presented as important foundations for enterprise applications; open models offer control over weights, governance, and deployment. The guide recommends defining use cases, assessing data readiness, and building privacy, bias-auditing, and monitoring controls before production, while noting that agents coordinate multi-step workflows across tools, APIs, and databases.
Databricks StaffDatabase modeling: a practical guide to techniques and best practices
Database modeling defines the structure, relationships and constraints that keep systems coherent across transactional, analytical and flexible workloads. The guide presents a three-phase design process: conceptual modeling captures entities and relationships, logical modeling adds attributes, keys, cardinality and normalization, and physical design translates that model into tables, indexes, constraints, partitioning and other platform-specific choices. It compares relational, document/NoSQL, dimensional, hierarchical and network models, linking each to data shape, workload and consistency requirements. It recommends explicit keys, consistent naming, appropriate normalization, query validation and version-controlled DDL, while warning against premature physical design, missing foreign keys, unclear NULL handling and poorly balanced schemas. It concludes that Databricks Lakebase combines a transactional database engine with the Databricks Lakehouse Architecture so one model can support operational and analytical workloads.
Databricks StaffMLOps vs DevOps: A Practical Guide for Data Scientists and IT Teams
The guide compares DevOps and MLOps, framing both as practices for reliable production systems but distinguishing traditional application code from machine-learning systems involving data and models. DevOps uses source control, automated testing, staging, and CI/CD, while MLOps governs the “holy trinity” of code, data, and models through separate versioning, quality checks, lineage, and access controls. Its central operational difference is Continuous Training: MLOps pipelines can validate model quality, monitor drift, and trigger retraining when data distributions shift, alongside testing model-serving infrastructure and promoting models through a registry. The guide also assigns responsibilities across data scientists, ML engineers, data engineers, and IT operations, and recommends dedicated MLOps for business-critical, frequently retrained models, DevOps for applications without ML, and split pipelines for hybrid products.
Databricks StaffTop Data Warehouse Tools For Modern Data Analytics
Data warehouse tools are presented as a strategic choice for analytics and ML teams facing fragmented estates of warehouses, lakes, and standalone ML systems. The guide proposes evaluating platforms across query performance, scalability, data integration, BI connectivity, total cost of ownership, and governance and security, with attention to MPP, columnar storage, and decoupled compute and storage. It contrasts traditional warehouses, data lakes, and lakehouses: warehouses favor structured SQL analytics, lakes offer native-format flexibility but weaker quality and performance guarantees, and lakehouses combine these capabilities using open formats such as Delta Lake and Apache Iceberg. The stated conclusion is that a lakehouse can provide a single governed foundation for SQL, BI, streaming, ML, and AI workloads, while teams should select tools according to workload, scale, budget, and future AI needs.
Databricks StaffAgentic AI vs Generative AI: Comparing Autonomy, Workflows, and Use Cases
Agentic AI and generative AI differ primarily in whether a system autonomously pursues a multi-step goal or produces content in response to a prompt. The post defines agentic systems through a perceive-plan-act cycle: agents maintain memory and state, decompose goals, call tools or sub-agents, evaluate conditions, and recover from errors, while generative AI typically performs bounded, reactive inference. It presents workflow examples such as sales follow-up and market-intelligence summarization, showing how APIs connect agents to CRMs, databases, communication platforms, and news services while LLMs provide text generation or reasoning at individual steps. RAG can ground generative outputs in external knowledge, but agentic deployments add operational concerns around repeated inference loops, human oversight, provenance logging, and access controls. The conclusion recommends choosing by task structure: generative AI for single-turn creation or summarization, agentic AI for autonomous coordination, and both together for complex enterprise workflows.
Databricks StaffUnlocking SAP business context in Databricks with semantic metadata Delta Sharing
SAP Business Data Cloud now offers generally available semantic metadata synchronization with Databricks Unity Catalog for mounted SAP BDC Delta Shares. When a table is accessed, Unity Catalog receives table-level business-friendly display names, descriptions, context, and governance tags, while SAP BDC remains the single source of truth and subsequent changes are reflected. The capability builds on BDC Connect and Delta Sharing, allowing governed SAP data products to be discovered and combined with other enterprise sources without recreating business context or governance separately. Column descriptions, table relationships such as primary and foreign keys, and PersonalData namespace tags give Databricks AI Assistant and AI/BI Genie explicit context for natural-language questions and join-ready queries. The stated result is more understandable, discoverable, and AI-ready SAP data, with automated classification signals supporting compliance, access control, and responsible AI.
Akram Chetibi, Katie Cummiskey, Moe Derakhshani, Abhijit ChakankarThe marketing activation gap has a fix: Databricks and Stitch partner to turn data infrastructure into marketing performance
Databricks and Stitch are partnering to connect enterprise data infrastructure with marketing execution, addressing a gap that leaves campaigns dependent on stale segments, delayed data, and disconnected tools. The partnership positions Stitch as a marketing implementation layer that structures Databricks data for real-time segmentation, personalization, AI-driven decisioning, and self-service analytics while building applications and agents directly on the platform. Its work spans campaign-ready architecture, full-stack marketing applications, Genie-based access for nontechnical users, AI-powered campaign operations, and migrations from legacy platforms. Examples include real-time transaction data reaching customer marketing at a convenience-store brand, measurable campaign results within weeks for a medical testing company, and a global QSR rebuilding campaign workflows on Databricks as AI tools improve.
Michael Burton, Bobby Tichy, Katy YuanAlert fatigue is a business risk
Enterprise security operations centers may receive tens of thousands of alerts daily, making prioritization necessary and leaving lower-priority signals uninvestigated. Alert fatigue is presented as a data architecture problem: fragmented endpoint, network, identity, and cloud telemetry, combined with proprietary SIEM collection-and-discard practices, limits correlation and overwhelms analysts. Lakewatch proposes an open lakehouse foundation that unifies security, IT, and business telemetry, applies automated OCSF normalization, and uses Agent Bricks for data wrangling and alert triage. Databricks Genie is positioned as a natural-language AI security agent whose autonomous agents can hunt, summarize, and neutralize threats, while Unity Catalog logs queries and actions for audit and forensic purposes. Lakewatch is currently available in Private Preview.
Taylor KainBackstage with Lakebase
Backstage with Lakebase describes a proof of concept that moves Backstage’s stateful internal developer portal from standard Postgres to Databricks Lakebase, a serverless Postgres surface in the Databricks Data + AI Platform. Connection required an app-config.yaml change, PgSearchEngine instead of the default in-memory search, and short-lived OAuth JWT credentials generated with databricks postgres generate-database-credential; Knex migrations then ran cleanly. Using copy-on-write branches, the team cloned a roughly 63 MB catalog in 1.09 seconds and recovered 32 deleted entities in 3.78 seconds, while observing that point-in-time recovery snaps backward to the nearest WAL record. The post argues that cheap, isolated database state can replace much mocking and staging iteration with real-data development, branch-based testing, schema validation in CI, and automated cleanup, while noting that VS Code/Cursor synchronization tooling remains underway.
Cameron Casher, Kevin HartmanShipping faster isn't learning faster
Product organizations can ship features in days while taking weeks to understand their behavioral impact, because fragmented analytics stacks depend on analysts, BI expertise, or SQL skills. The post argues that this architectural bottleneck slows the insight-to-ship cycle, causing roadmap decisions to rely on instinct, anecdotes, and lagging indicators. It presents Databricks AI/BI Genie as a conversational interface to event-level behavioral data, with experiment integration, cohort analysis, and product-specific growth-metric definitions. According to the post, product leaders can ask questions in plain language without filing analyst requests, while governed data access supports faster follow-up and feature-impact decisions. It reports that Genie users across 3,300+ Databricks customers cited a 49% productivity gain, 41% faster speed to market, and 5x faster ad-hoc analysis, though these are reported customer results.
Madelyn MullenWhy your OEE dashboard is lying to you
Manufacturing OEE dashboards can look healthy while omitting recurring downtime that supervisors recognize, because relevant facts are split across SCADA, MES logs, maintenance tickets, and shift reports. OEE combines Availability, Performance, and Quality, but extracting those inputs often requires SQL or an analyst, delaying root-cause analysis after throughput drops. Databricks Genie is presented as a conversational AI layer over a unified data platform that leaves MES and SCADA in place while allowing business leaders to ask plain-language questions such as OEE by line against maintenance windows. Its semantic-layer awareness maps terms to actual fields, while governed access and logged, source-cited answers support different visibility levels and traceability. The proposed shift is from static reporting to faster operational questioning about production status, forecast risk, line performance, and quality signals.
Caitlin Gordon