Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
- 564 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
Unity Catalog Pages: a governed home for your business knowledge in Genie Ontology
Business concepts like active customer or completed trip are often defined differently across teams, scattered in Slack threads and spreadsheets, causing AI agents to guess confidently when answering questions. Databricks introduces Unity Catalog Pages as a governed home within Genie Ontology where data stewards with Genie Code define the authoritative meaning of those terms and entities. Pages complement existing semantics — metric views how to calculate, domains where to look, and certification what to trust — by capturing what each concept means. Genie Code can bootstrap Pages by extracting definitions from Confluence, Slack, Google Docs, or GitHub via MCP-connected tools, turning weeks of work into minutes. When Genie One uses a Page to ground an answer, the Page is cited as a clickable source so users can inspect the definition and verify the reasoning.
Database Branching: A Developer's Guide to Git-Style Workflows
Database branching creates isolated database environments from a known parent state using copy-on-write, where branches share unchanged data and only store modifications. For a large parent database like one that is 40 GB, two branches may require just 5.6 MB of additional storage instead of 80 GB with traditional full copies. Developers can test schema migrations like adding a NOT NULL column to a table with millions of rows against realistic data and constraints, CI pipelines provision per-pull-request branches for isolated tests, and AI agents experiment across short-lived environments without risking the parent database. Safe operation relies on protecting parent branches from writes and deletions, restricting sensitive data in disposable branches, automating creation and cleanup, and keeping environments reproducible through version-controlled configuration. Databricks Lakebase provides database branching, enabling teams to adopt Git-style workflows for development, CI/CD, and AI agent use cases.
The Genie One MCP is now Generally Available
Databricks has announced general availability of the Genie One Model Context Protocol (MCP) server, now a managed service within Unity Gateway with centralized governance, fine-grained policies, and audit logging across every invocation. The MCP provides a single governed interface for any AI agent to access structured and unstructured data from Genie One, grounded in governed business context from Genie Ontology, eliminating agent sprawl and inconsistent answers. It exposes tools for asking questions, getting query results, checking incremental progress, and steering responses, with an optional MCP App for real-time interactive visualizations and Genie Ontology citations. Connected agent clients automatically interpret domain semantics via Genie Ontology across relational data and document repositories without custom connectors. Use cases include enriching slide creation with richer data, bringing customer usage context to outreach agents, and integrating business definitions into developer workflows, as noted by GetYourGuide.
Genie One MCP: Give any AI Agent the Right Business Context
Business leaders often have access to plenty of data but still cannot get a reliable answer to a simple question like why net revenue declined 8% at their largest account last week. The answer may span sales, finance, promotions, inventory, and account data, with each source potentially correct in isolation yet different in definitions, detail, relationships, and authority. Without the right business context, general-purpose agents cannot reliably determine what metrics mean, which sources are authoritative, how data should connect, or what each user is allowed to see. Genie One MCP gives governed context to any MCP-compatible AI agent, connecting assistants like ChatGPT, Claude, Microsoft Copilot, and coding agents to Genie One. This lets them answer business questions using approved definitions, trusted data relationships, and permission-aware access controls, with traceable, permission-aware answers grounded in governed business context.
What is AI analytics? Why it only works on governed data
AI analytics applies AI and machine learning to data analysis, enabling natural-language queries and automated insights without manual query-building, shifting from retrospective dashboards toward agentic investigations. Unlike traditional BI, which requires users to navigate predefined dashboards and filters, AI analytics systems understand intent, determine relevant data and business context, and can conduct multi-step investigations on behalf of the user. A key challenge is that AI analytics only works when grounded in governed data: without shared metric definitions, authoritative sources, and permission-aware execution, the system produces untrustworthy results. Trust in AI analytics is a system property, not a model property; it requires governed data, shared business context, permission enforcement, and verifiability of queries and calculations. Better AI models do not eliminate the need for business context; as AI becomes more autonomous, semantic definitions and governance become more critical.
Data Ontology defined: The context layer your AI agents are missing
The Post argues that AI agents are breaking a long-standing assumption in enterprise data: that a human analyst can always resolve ambiguity. Richard Tomlinson explains that while analysts rely on tribal knowledge—like which revenue table is trusted or what ‘active customer’ means—agents lack this context. This gap causes inconsistent answers and erodes trust in self-service tools. The conversation examines why previous ontology efforts stalled and what must be true for an ontology to support agents. The ontology is positioned as the missing context layer that companies have not yet built.
RADAR: Catch gray failures with anomaly detection
Gray failures are partial customer-facing problems that can leave health checks green while a subset of users quietly fails. RADAR (Reliability Anomaly Detection, Alerting, and Root-cause analysis) addresses this gap by collecting error counts and distinct affected users, segmented by error code and region, then applying the unsupervised streaming SPOT model. SPOT learns normal behavior from the previous 14 days using a single risk parameter, while downstream stages enrich, filter, deduplicate, and route alerts before providing anomaly details and AI/BI Genie-backed dashboards for investigation. Running RADAR internally reduced incident-discovery time by 95% with over 90% precision and no human needed to spot the pattern. The post also describes a Databricks scaffold that maps metrics, detection, alerting, and root-cause analysis to platform components and can be assembled by an AI agent from a metric, scaffold, and prompt.
Enabling secure, productive work on personal devices
Personal devices let employees use Slack, approve requests, and access internal apps, but they combine corporate data with private content that Databricks cannot inspect or control. The proposed mobile security model layers Mobile Device Management (MDM), identity and context-aware authentication, Zero Trust Network Access (ZTNA), and application management. On iOS, Account-Driven User Enrollment creates an encrypted workspace without full device management, while Android Work Profile provides comparable separation; per-app VPN keeps work traffic apart from personal traffic. Access requires strong authentication, a managed device, and a trusted tunnel, with continuous posture checks able to block unhealthy devices. Managed apps, configurations, tenant controls, and sensitivity tiers contain data where supported, while transparency about privacy encourages enrollment and supports mobile apps such as Genie.
Modernizing the Trade Lifecycle With Governed Data and AI
Capital-markets firms are revisiting the trade lifecycle as data volumes grow, AI moves toward production, settlement cycles shorten, and scrutiny increases. Fragmented data across research, trading, risk, operations, and surveillance can slow decision-making, complicate execution analysis, limit risk visibility, and hinder reconstruction of events. The proposed response starts with decisions such as execution-cost divergence, shock impacts, and exception or settlement-fail rates, then identifies the data domains needed for a few high-value workflows rather than migrating everything at once. It emphasizes a governed, reusable foundation connecting real-time and historical data, analytics, and AI, with access controls, lineage, traceability, and evaluation. Databricks components named include Unity Catalog for governance, discovery, and lineage, Agent Bricks for domain-specific agents, and AI/BI for governed exploration; the stated conclusion is that this operating-model approach can support faster decisions, resilience, and scalable use cases.
Database for AI Agents: 5 Evaluation Criteria
The post defines a database for AI agents as infrastructure for short-term, episodic, procedural, and operational memory, rather than merely a place to retrieve documents. It proposes five evaluation criteria for moving agent workloads from prototypes to production: per-agent branch isolation, serverless scale-to-zero, hybrid search, ACID guarantees, and unified platform access. The technical rationale is that agents perform repeated reads and writes, resume tasks, share state concurrently, and need current data across vector, keyword, and metadata retrieval; zero-copy branches, sub-second wakeups, and single-query hybrid search address specific operational constraints. It concludes that a database missing more than one minimum bar becomes a production risk, while standard Postgres alone supplies ACID but not branching, scale-to-zero, or unified operational and analytical access.
Building a knowledge assistant over code
Building a knowledge assistant over code requires more careful chunking than indexing ordinary business documents, because code has nested structure, cross-file dependencies, and multiple file formats. The comparison built three Databricks Knowledge Assistants over the Casper’s Kitchens repository: fixed-size character chunks, LangChain language-aware splitting, and AST-based splitting with structure and metadata headers. The AST approach used ASTChunk to preserve units such as functions, classes, or statement blocks and prepend file paths plus class/function hierarchies to chunks. MLflow GenAI evaluation ran 46 questions with retrieval sufficiency, groundedness, and custom answer-correctness judges; AST chunking achieved 89% retrieval sufficiency and 70% fully correct answers, versus 85% and 59% for the naive baseline. The advantage was strongest for disambiguating structurally similar code, while value lookups and complete definition retrieval were comparable across methods.
Google Cloud Use Cases: How Enterprises Run Data & AI on GCP with Databricks
The guide presents Google Cloud as a unified platform for enterprise data engineering, analytics, machine learning, and generative AI, with managed services intended to reduce infrastructure and integration work. It focuses on practical architectures, including medallion ETL pipelines that store Bronze, Silver, and Gold Delta Lake tables in Google Cloud Storage and expose Gold data to BigQuery through an optimized connector. For streaming workloads, Cloud Pub/Sub supplies ingestion while Apache Spark Structured Streaming processes continuous micro-batches; together with Delta Lake ACID transactions, this supports exactly-once results despite duplicates or out-of-order messages. The text also describes Cloud Run and Cloud Functions for containerized or event-driven workloads, and identifies retrieval-augmented generation applications on Vertex AI as a common starting point for generative AI. It concludes that integrated service combinations, rather than isolated tools, are the central strategic advantage for organizations building evolving data and AI platforms on Google Cloud.
Databricks recognized as a Gartner® Peer Insights™ Customers' Choice for Analytics and BI
Databricks says it was recognized as a Customers’ Choice in Gartner Peer Insights’ Voice of the Customer for Analytics and Business Intelligence Platforms, based on verified reviews from customers using platforms in production. The post reports a 4.8 out of 5 overall rating and 94% willingness to recommend, while noting that the designation reflects customer adoption and overall experience meeting or exceeding market averages. It presents Databricks AI/BI, AI/BI Dashboards, Genie Spaces, and Genie as a unified architecture combining governed live data, AI, business intelligence, and Unity Catalog controls without moving or duplicating datasets. The described use cases include conversational exploration, dashboard and visualization development, consolidating fragmented reporting, and expanding governed analytics access, with the company positioning these capabilities as a foundation for agentic analytics.
How Databricks Helps Baseball Teams Gain an Edge with Data & AI
Baseball teams can use Databricks to turn Statcast and team-derived pitch-level data into game-day plans for hitters, bullpen changes, pinch-hit choices, advance scouting, and roster decisions. Genie, on Unity Catalog, queries governed Delta tables and reusable Unity Catalog SQL functions for tendencies by count, handedness, pitch type, location, and base-runner state. A Multi-Agent Supervisor built with Agent Bricks and deployed on Model Serving simulates matchup pockets, compares reliever arsenals with hitter profiles, and produces explanations that staff distill into printed decision cards. For advance work, AI Search indexes pitcher and hitter embeddings to find similar arms, while Lakebase Postgres keeps reports, tags, and decisions close to the source data for shared operational memory.
What is a Relational Data Model?
The relational data model organizes structured data in tables of rows and columns, with SQL used to access and manage it. Introduced by Edgar F. Codd in 1970, it uses schemas, domains, keys, constraints, and relationships to define and connect data. A relation schema specifies a table's structure, while a relation instance is the data held at a particular moment; primary, candidate, super, foreign, and composite keys support identification and referential integrity, and one-to-one, one-to-many, and many-to-many relationships model associations between tables. Functional dependencies inform normalization, whose 1NF, 2NF, and 3NF stages reduce redundancy and insertion, update, and deletion anomalies, though heavier normalization can require more joins. The model is used in banking, e-commerce, healthcare, education, and enterprise systems, where constraints and ACID transactions help preserve accuracy and reliability.
What is Intelligent Document Processing?
Intelligent document processing (IDP) uses AI to extract, classify, validate, and structure information from PDFs, images, emails, forms, and other structured, semi-structured, or unstructured documents. Its pipeline combines document ingestion and preprocessing with OCR, document classification, NLP, layout analysis, data validation, human review for uncertain results, and delivery into downstream systems such as ERP, CRM, or accounting software. The post explains how embeddings support classification, how continuous learning adapts models to new layouts and corrections, and how metrics such as accuracy, throughput, confidence scores, and model drift support optimization. IDP can reduce manual effort, improve accuracy, lower costs, scale document-heavy workflows, and supply structured data for analytics, automation, AI agents, and retrieval-augmented generation, while high document variability and edge cases still make human-in-the-loop review relevant.
What are Analytic Applications?
Analytic applications are packaged business intelligence systems designed to measure, analyze and improve performance for defined business domains. They combine data integration, modeling, visualization, reporting and workflow support so technical and non-technical users can turn operational data into decisions. Typically, they connect ERP, CRM and supply chain systems with a data warehouse or lakehouse, process historical and real-time data, and deliver dashboards, reports, scorecards or embedded insights through preconfigured schemas, connectors, governance controls and business logic. Unlike general-purpose BI tools built for open-ended exploration, they support standardized, domain-specific workflows such as sales performance management, financial planning, risk assessment and inventory optimization. Organizations can start with descriptive analytics and adopt predictive or prescriptive capabilities as data maturity increases, while choosing standalone or embedded deployment based on workflow needs and data readiness.
Building an A/B testing analysis framework for mobile gaming on Databricks
HARDlight needed a more consistent way to analyse mobile game A/B tests as experimentation expanded, because manual workflows and differing statistical approaches slowed decisions and weakened trust. It built a Databricks-native framework that standardises experiment definitions, telemetry, modelling, and governed outputs, with notebooks materialising effect estimates, uncertainties, and segment-level impacts into a unified analytics model. Databricks AI/BI then presents daily refreshed, progressively detailed dashboards, beginning with an LLM-generated summary and extending through KPIs, diagnostics, revenue, engagement, monetisation, and segment analysis. At experiment completion, a frozen snapshot preserves outcomes, decisions, and recommended actions. The framework reduced manual effort by more than eight hours weekly, saves roughly one day per experiment, and enabled a targeted two-times increase in monthly testing capacity without added headcount.
Tevogen Bio’s Journey to Streamlining Life
Tevogen Bio faced drug development costing upwards of $3 billion and requiring 10–12 years; selecting targets for its single-HLA-restricted SARS-COV2 product took 18–24 months of wet-lab work. Tevogen.AI partnered with Microsoft and Databricks to build and curate a multi-terabyte protein-sequence library and parallelize processing for machine-learning training. Databricks Data + AI Platform, Medallion Architecture, Unity Catalog, and distributed computing cut processing from 50 days to 24 hours under governed, fine-grained access controls. The dataset reached 24 million proteins, 16 billion datapoints, ~700 million unique peptides, and ~37 million cross-matched expert articles. An MLOps framework automated training, inference, monitoring, and retention; the alpha PredicTcell model used XGBoost and ESM models and reported 93–97% recall and 38–43% accuracy, while beta PredicTcell and alpha AdapTcell work began.
What is a Cloud
A cloud-based database management system stores, manages, and queries data on third-party cloud infrastructure instead of local or on-premises servers. It retains capabilities such as query optimization, security, transactions, and support for relational SQL, NoSQL, data warehouse, and in-memory workloads, while operating across distributed nodes. Organizations can self-manage database software on cloud virtual machines or use DBaaS, where the provider manages the software and infrastructure; connections typically use SQL, APIs, or web consoles. Virtualized resources can scale, and providers may handle patching, backups, maintenance, and failover, while replication across availability zones supports recovery and availability. The comparison with on-premises systems covers ownership, scalability, cost models, maintenance, accessibility, control, compliance, total cost, and emerging capabilities including AI integration, vector search, autonomous operation, multicloud, hybrid, and serverless architectures.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.