Observability for any agent, anywhere: Production-ready tracing with OpenTelemetry & Unity Catalog on Databricks

Summary

Databricks supports writing OpenTelemetry (OTel) traces directly to Unity Catalog, where real-time telemetry is stored in Delta tables for governed analytics and retention. AI traces capture prompts, tool calls, responses, latency, and execution paths, enabling debugging, evaluation, and monitoring, while lakehouse storage also allows SQL queries, dashboards, joins with business data, and PII controls. The managed, serverless ingestion layer uses Zerobus Ingest to accept OTLP over gRPC from collectors and REST integrations, streaming spans, logs, and metrics to Unity Catalog without intermediate message buses. A LangGraph support manager assistant demonstrates instrumentation with mlflow.langchain.autolog() and an @MLflow.trace root span, while Genie is invoked through MCP for data-driven questions. The resulting traces can be searched in MLflow, evaluated at scale, and monitored continuously, with FAQ details stating 200 QPS starting throughput, no storage limit, and Unity Catalog governance options for access control, masking, and filtering.

Context

AI traces contain valuable prompts, tool calls, responses, latency, and execution paths, but keeping them only in observability systems limits long-term retention, SQL analysis, business-data joins, evaluation, and governance. Sending raw prompts to third-party SaaS platforms can also create InfoSec friction because of sensitive data and PII.

Approach / What changed

Use Databricks-managed, serverless Zerobus Ingest to receive OpenTelemetry spans, logs, and metrics over OTLP/gRPC or REST and store them in Unity Catalog Delta tables. Configure the tables with MLflow, instrument agents with compatible libraries such as mlflow.langchain.autolog() and @MLflow.trace, then use MLflow, SQL, dashboards, and downstream evaluation workflows.

Takeaways

• Unity Catalog stores OpenTelemetry spans, logs, and metrics in dedicated tables, with derived MLflow views for unified traces, metadata, and annotations.
• Zerobus Ingest provides a single-sink telemetry path that bypasses intermediate message buses such as Kafka and starts at a stated ingestion limit of 200 QPS; storage has no limit according to the FAQ.
• Unity Catalog provides fine-grained access controls, column masking, and row filtering for trace data, but the feature does not apply special PII handling automatically.

Related reading