Loading…
Databricks
Data and AI platform for data engineering, analytics, machine learning, and generative AI.
- 542 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
How to ground Genie Agents in both structured data and documents without losing governance
Databricks Genie Agents can combine structured tables and unstructured documents, but the central challenge is preserving data governance when one agent can access both. The proposed architecture runs agents with the end user’s credentials, using Unity Catalog identity synchronization, object privileges, ABAC policies, row filters, and column masks to enforce permissions at query time. Automatic Identity Management syncs users and groups from Microsoft Entra ID and Okta, while just-in-time provisioning carries current memberships into Databricks. Structured assets include tables and metric views, and governed tags can attach protection across newly tagged data; documents are governed through Unity Catalog volumes and their grants. In the Brickstore example, APAC and AMER managers ask the same question yet receive different, authorized results, with regional rows, masked email values, and shared metric definitions; the source recommends testing by impersonating users and planning volume boundaries.
Innocent until combined: Blocking the lethal trifecta with Omnigent Contextual Policies
An AI agent can exfiltrate private data when one session combines access to confidential information, attacker-controlled content, and an external communication channel, even though each capability is individually authorized. Omnigent's contextual policy tracks those three legs as session state: private data, untrusted content, and exfiltration, with human-defined tool or argument-based classifications. It allows ordinary actions until both prerequisite legs are lit, then denies the outbound call that would complete the trifecta. In the support assistant example, a user-supplied ticket instructs the agent to read internal Q3 revenue figures and email them externally; without the policy, the confidential number leaves in a reply, while the protected run blocks that email. A password-reset ticket still proceeds because it lights only the untrusted-content leg, and an empty internal lookup does not light the private-data leg.
Introducing FILE type: a native column type for multimodal data
The beta of FILE introduces a native column type for storing unstructured data such as documents, images, audio, and video alongside structured table data. FILE applies fine-grained governance, row- and column-level controls, ABAC, and lifecycle synchronization, so deleting a row also deletes the corresponding binary in object storage. Its lightweight pointers fetch file content only when a query requires it, while SQL, Python UDFs, built-in AI functions, and materialized views support multimodal processing. The example uses sampled dashcam frames and object detection to identify stops with no hazard, then joins extracted results with trip metadata; the announcement also describes connectors, incremental pipelines, and an open effort around Parquet, Delta Lake, Apache Iceberg, and Apache Spark.
Managing AI Coding Costs at Scale
AI coding tools have improved velocity metrics at Databricks, but broad deployment creates exponentially growing costs that can threaten the efficiency gains they deliver. The post presents a cost-management playbook aimed at preserving broad, low-friction access while keeping aggregate spending within a roughly fixed per-user envelope. It emphasizes the coding-model efficiency frontier, automated evaluations of internal workloads, model-flexible harnesses or meta-harnesses, and request- or task-level routing to the cheapest capable model. Databricks reports that Unity AI Gateway Smart Router reduced average task cost by more than 30% while roughly matching the most expensive model’s quality, and that harness and caching changes cut generated tokens and associated costs by almost 50% without observed quality degradation. The proposed AI Gateway centralizes model access, budget policies, tool configuration, and coding-session traces for ongoing optimization.
What is an AI Assistant?
AI assistants are software systems that use language models, data retrieval, and reasoning to understand natural-language requests and return information or perform tasks. For enterprise data teams, they can generate SQL, build dashboards, troubleshoot pipelines, and automate repetitive work, but their usefulness depends on integration with data catalogs and governance rather than generic text generation alone. The described pipeline captures input, parses intent and context, retrieves knowledge, generates an output, and either delivers it for review or executes an action; advanced systems incorporate tool and API use plus feedback from corrections and approvals. The article categorizes assistants as voice, conversational, general-purpose, domain-specific, data and developer, or autonomous agents, while noting risks including hallucinations, privacy exposure, integration complexity, over-reliance, and bias. It presents assistants as augmenting human work through faster, more accessible, and more scalable decision support, not replacing human oversight.
What are Agentic Workflows?
Agentic workflows are AI-driven processes that let one or more agents plan, execute, and refine multi-step tasks toward a defined goal with minimal human intervention. Unlike chatbots and fixed automation, they evaluate context and intermediate results at runtime, gather data, select tools dynamically, and adapt when outcomes are incomplete, incorrect, ambiguous, or unexpected. The described pattern combines agents and LLMs with APIs, databases, code executors, prompts, feedback mechanisms, memory and state management, and, for complex work, coordinated specialized agents. Production reliability depends on clear success criteria, action guardrails, human review points for high-stakes actions, audit trails, governance, and production-grade infrastructure. Examples include customer-service resolution, fraud investigation, supply-chain optimization, and event-driven document processing across systems.
What is Tool Calling?
Tool calling lets an AI model interact with external tools, APIs, and systems to retrieve information or perform actions beyond its training data, turning a conversational model into an agent that can act. The process is an application-mediated loop: the model recognizes when outside help is needed, selects a tool from schema-described options, emits a structured request—typically JSON—and receives the result as context for its response or next step. Available tools can support information retrieval, code execution, process automation, and smart-device or IoT monitoring, while complex tasks may chain multiple calls into a workflow. The piece also distinguishes tool calling from function calling, describing the latter as a subset, and presents MCP as a standardization layer for describing, discovering, and invoking tools, with Databricks Agent Bricks offering governed integrations and agent-development support.
Introducing OfficeQA Pro V2: A New Benchmark for Enterprise Grounded-Reasoning
OfficeQA Pro V2 benchmarks whether AI agents generalize grounded reasoning to unfamiliar enterprise document collections rather than specialize in one corpus or task distribution. It contains 90 questions grounded in roughly 120,000 pages of U.S. Treasury accounting records spanning 1793–2024, including dense tables, charts, revised values, and changing reporting conventions. Questions were generated with asynth from seeded inputs, corpus evidence searches, traceable reasoning, and quality gates, then evaluated with deterministic exact match at 0.0% tolerance. Out-of-the-box frontier agents averaged 37.5% in one matched comparison, while competition systems averaged 41.1% and the winning team reached 63.3%; Databricks Genie raised matched-model accuracy by 24.0 percentage points on average, although failures remained in parsing, temporal reconciliation, and entity or category interpretation.
Kimi K3 from Moonshot AI is now available on Databricks through Unity AI Gateway
Kimi K3 from Moonshot AI is now available on Databricks through the Foundation Model API and Unity AI Gateway, giving customers native access to an open-weight model alongside proprietary models from Anthropic, OpenAI, and Google Gemini. The announcement positions Kimi K3 as a frontier-quality option for coding, agentic reasoning, and document understanding, with deployment flexibility, customizability, and zero data retention coverage. Databricks says the model runs against governed enterprise data through Unity Catalog, while Unity AI Gateway provides permissions, safety guardrails, audit trails, spend controls, routing, and a standardized REST API within Databricks’ security perimeter. The post cites a score of 57 and rank of #4 in Artificial Analysis’ Intelligence Index, internal benchmarks showing performance at the level of leading proprietary models, and 50–72% lower cost per task; availability includes US hosting for AWS and GCP workspaces.
Unity AI Gateway is Generally Available
Unity AI Gateway is generally available as a centralized way for enterprises to manage AI costs, security, governance, and model choice across agents and other AI assets. It positions Unity AI Gateway alongside Unity Catalog: Unity Catalog supplies identity, permissions, lineage, and auditing, while the gateway applies runtime guardrails and contextual policies across AI interactions. The gateway centralizes usage data in Unity Catalog, offering dashboards, Genie-powered analysis, granular cost attribution, budgets, hard spend caps, and a single-query API for frontier models from providers including Anthropic, OpenAI, Gemini, Kimi, and GLM. Smart Routing is in beta and dynamically selects models using quality, cost, performance, availability, and budget; the announcement says customers already use the gateway at scale, including more than one quadrillion tokens passing through it in the past year.
Databricks joins the Open Secure AI Alliance to advance AI safety and security
Databricks announces that it is a founding member of the Open Secure AI Alliance, formed with NVIDIA and more than 75 organizations to advance openly shared AI safety, AI security, and AI-enabled cyber defense research. The alliance’s scope extends beyond model weights to an open execution stack covering runtimes, guardrails, agent harnesses, risk frameworks, and governance. Databricks contributes Omnigent, an Apache 2.0 meta-harness supporting 13+ harnesses with contextual policies, spend caps, and sandbox isolation; DASF 3.0, which maps 97 risks across 13 components to 73 controls; DAGF; and BlackIce, a containerized toolkit bundling 14 red-teaming tools. Its Lakewatch Security Lakehouse applies the same openness to governed, agentic detection and response over portable data formats.
The New Monday Morning Report: How Generative AI can deliver the insights your executives need.
Retail and CPG joint planning meetings often spend time reconciling fragmented, stale, and inconsistent data instead of deciding how to adjust spend, inventory, or forecasts. The proposed Monday Morning Brief replaces the manually stitched weekly deck with an overnight agent-generated brief built from continuously refreshed point-of-sale, shipment, and inventory data alongside trade, promotion, forecast, and external signals. It uses Delta Sharing for a shared view, Genie Ontology for business context, Unity AI Gateway for permissions, guardrails, logging, rate limits, and human approval, and supports models across AWS, Azure, and Google Cloud. The agent scans item-store combinations, ranks material watchouts, answers cited natural-language follow-ups, and can draft actions such as purchase-order changes, while humans retain approval for consequential decisions. The proposed progression from report to ritual to intelligent decision system includes a 90-minute workshop, a day-30 pilot scope, and a first live Monday targeted for day 90.
Ingest semi-structured data faster and more efficiently with Variant - Now Generally Available
Databricks announces general availability of the Variant data type for ingesting semi-structured JSON, XML, and CSV while avoiding the usual choice between flexible storage and fast, schematized queries. Variant Shredding, also generally available, stores common fields as columns in Parquet files, while Predictive Optimization uses workload and query patterns to identify important fields, collect statistics, and improve file skipping. More than 5,000 teams write Variant, with users executing over 500 million Variant queries monthly across more than 160 TB of data; shredding delivers nearly four-times-faster reads than unshredded Variant and 30-times-faster reads than JSON strings. Auto Loader and Zerobus can ingest Variant into Delta or Iceberg, and future plans include Liquid Clustering by Variant fields, expanded SQL functions, and additional integrations.
Databricks Completes Acquisition of Panther: Accelerating the Security Lakehouse Era
Databricks says it has completed its acquisition of Panther, an AI SOC platform, to accelerate its security lakehouse strategy. The combination pairs Lakewatch’s open, governed foundation for collecting, retaining, and analyzing petabyte-scale security telemetry with Panther’s operational SOC workflows and more than 100 out-of-the-box integrations. Panther adds detections-as-code, CI/CD-based authoring and deployment, and AI-native triage and investigation that correlate cloud, identity, SaaS, IT, and business data. The announcement presents the combined platform as a way to retain high-fidelity telemetry, preserve data ownership through open standards including OCSF, Spark, Unity Catalog, Delta, Parquet, and SQL, and automate investigations, detection refinement, and response workflows for modern security operations.
Backstage with Lakebase, part 3
This third part describes joining Backstage’s live operational Postgres catalog with analytical billing data so teams can query resource ownership and cloud spend together. Lakebase isolates compute per workload, allowing portal traffic and FinOps queries to use the same underlying storage; benchmarked catalog queries ran at 55–65 ms end-to-end and searches at two to four ms. Because Lakehouse Federation’s Postgres connector accepts static credentials while Lakebase applications use OAuth JWTs, the proof-of-concept creates a separate SCRAM-SHA-256 Postgres role for federation. This enables a single query to join Backstage resource names with Unity Catalog’s system.billing.usage without data movement, while branch billing remains visible per branch and endpoint. The operational considerations include password rotation, read-only grants, branch TTLs, and scale-to-zero endpoints that stop billing when idle.
Foundations for an AI-forward healthcare organization
Healthcare organizations adopting AI face fragmented data, mismatched governance, and no repeatable operating model, rather than a shortage of ideas or vendors. The piece defines an AI-forward organization as one where AI can be built, trusted, and scaled through a foundation of unified data, visible guardrails, and an operating model that helps teams prioritize and move pilots into production. It describes patient identifiers differing across source systems, requiring manual reconciliation and creating recurring integration costs. Governance must avoid both untrusted outputs and approval processes so rigid that nothing leaves the sandbox, while self-service users need controlled access to clinical, operational, and financial data. The source says modern tooling can centralize permissions and enable a governed first-use case in days rather than quarters when scope and data are ready.
Agentic media buying cannot scale without the right foundation. See how buyers and sellers get there on Databricks.
Media buying remains slowed by fragmented coordination across emails, spreadsheets, PDFs, and phone calls, leaving teams to research inventory, compare pricing, negotiate, and issue orders manually. The post presents an agentic workflow in which buyer and seller agents discover inventory, negotiate prices, and book deals through IAB Tech Lab’s AAMP standards, including AdCOM, OpenDirect, OpenRTB deals, and registry-based discovery and trust. Its Databricks implementation runs self-contained applications on Databricks Apps, with CrewAI agent crews using Databricks Foundation Model APIs and Claude models, Lakebase providing transactional Postgres state, and MLflow tracing capturing decisions and tool calls. The example processes a $200,000 Q3 Brand Launch across CTV and Linear TV with a $38 CPM ceiling, and is packaged as a Databricks Automation Bundle accelerator deployable with a single command.
Convert proprietary code to open ANSI SQL with Genie Code
Databricks introduces an agentic converter in Genie Code to translate proprietary warehouse dialects into open ANSI SQL, initially supporting T-SQL, Snowflake, Redshift, Oracle, BigQuery, and Teradata. Migration projects provide a workspace hub for source files, conversion status, complexity scoring, dependency lineage, and collaboration, helping teams prioritize work and identify scripts that can move independently. When launched, swarms of subagents convert files in parallel, iteratively fixing errors and validating syntax and semantic intent; in the proof-of-concept, six of eight files converted successfully. Two files required review because stored procedures needed three-part Unity Catalog names, which could be fixed manually or encoded as a custom Genie Code skill for reuse across the codebase. The post also notes that Databricks supports multi-statement transactions, temporary tables, and stored procedures, while planned additions include legacy ETL sources, new target dialects, data migration, and reconciliation.
NBCUniversal’s Seamless Migration: Unlocking Scalable Analytics with Databricks
NBCUniversal sought to modernize its data infrastructure as growing data volumes, evolving analytics demands, and slot-based compute reservations created cost, contention, and flexibility concerns. Working with EXL and Databricks, it migrated workloads to a Databricks lakehouse architecture that unifies data engineering, analytics, machine learning, and business intelligence. The phased, risk-based program assessed workloads and dependencies, designed workspaces and governance with Unity Catalog, and used four custom accelerators for SQL translation, DAG migration, data transfer, and validation. Job-specific compute, autoscaling, Delta Lake layouts, MLflow, and Lakeflow Jobs supported workload-specific execution and operations. The reported outcome was a 30% reduction in data infrastructure costs, elastic scaling for high-traffic events, unified ML operations, and the onboarding of approximately 300 analysts to Databricks SQL.
Quality care is the mission. Finance protects the margin.
Healthcare finance teams must protect margins amid rising medical costs, denied or underpaid claims, complex payment arrangements, and cash trapped in receivables, while fragmented systems and delayed information increase decision risk. The post presents ontology as a way to preserve the business meaning behind figures, linking numbers to service lines, payers, contracts, and changing context rather than treating accuracy alone as correctness. Databricks Genie is described as a governed, data-smart AI coworker that answers finance questions with sourced figures, traces them to their origins, respects permissions, and keeps a person in the loop. It applies this model to care costs exceeding reimbursement, revenue lost to denials and underpayments, and aging or unbilled receivables, with the intended actions of renegotiating rates, preventing claim errors, accelerating collections, and protecting margin.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.