Loading…
Topic
OpenTelemetry
16 posts about OpenTelemetry. Every summary links to the original.
- 16 posts
- 9 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Evaluating AI agents: moving past the vibe check
Manual vibe checks for AI agents miss silent failures like retry loops, token waste, and hallucinated parameters while plausible final answers mask them. The post outlines a structured evaluation pipeline using the open-source agent-eval toolkit, starting with critical user journeys (CUJs) that define success in concrete, testable clauses. Synthetic user simulations (via ADK) generate multi-turn scenarios for initial trace collection, which developers grade with a ‘red pen’ before automating checks. Four layers of verification are recommended: unit tests, deterministic metrics, LLM-as-a-judge with boolean rubrics (avoiding unreliable 1-to-5 scales), and integration tests, all standardized on OpenInference/OpenTelemetry traces. The rollout progresses from Day 0 manual review through Day 1 automated checks to Day 2 CI/CD gates and shared dashboards, with real production traces fed back into the golden dataset.
Sentry JavaScript SDK v11 adds tracing for all JS platforms
Sentry JavaScript SDK v11 replaces OpenTelemetry-based tracing with a code transformer from the apm-js-collab packages that injects diagnostics channels into supported modules at load time. The transform wraps target functions in tracingChannel() calls, and Sentry integrations subscribe to those channels to create native spans directly from the events. This approach works both at runtime via Module.registerHooks and at build time through bundler plugins for Vite, esbuild, webpack, and others, with no risk of double-wrapping. The previous OpenTelemetry instrumentation required Node-specific loader hooks, could not instrument bundled dependencies, and had ordering issues with ESM imports, limiting support to Node.js. v11 extends automatic tracing to Cloudflare Workers, Deno, Bun, and bundled server deployments while offering three OpenTelemetry integration modes: Sentry only, OpenTelemetry-compatible, or custom OTel with OTLP export.
Introducing Cloudflare Traces: follow requests through our entire platform
Cloudflare Traces, in open beta, extends tracing beyond Workers to the full request path, capturing spans for security, transformations, cache, routing, Workers, and origin handling in one timeline. Users set a baseline sampling rate and use Trace Rules to override it for specific traffic, enabling targeted debugging without increasing volume across the domain. The feature accepts and forwards W3C traceparent headers for distributed tracing and exports spans via OTLP to any compatible backend, with no instrumentation required. It reveals why a request was blocked, which rule acted, whether the URL was rewritten, and where time was spent between cache, origin, and the application. Pricing, effective December 1, 2026, is based on ingestion and retention: free tier 0.5 GB daily with 7-day retention, paid plans 50 GB ingestion and up to 365-day retention.
Behind the Scenes: How the OpenTelemetry Plugin Maps Your Microservices in Real-Time
Outdated architecture diagrams and static code analysis can miss how services are connected at runtime, so the OpenTelemetry plugin builds a Service Map from live telemetry. With a lightweight local OpenTelemetry backend, the plugin supplies standard OTel environment variables, receives application spans, and continuously updates an internal architecture model that the IDE renders on demand. Its stream-processing algorithm classifies spans through semantic attributes, identifies services, matches HTTP CLIENT and SERVER spans across propagated trace context, and infers database or messaging dependencies while retaining unmatched spans in memory. Because spans can arrive out of order and traces have no definitive finish signal, the map is incrementally reconstructed rather than built from complete traces; the result is a language- and vendor-agnostic view for OpenTelemetry-instrumented applications across several JetBrains IDEs.
How we eliminated $1 million a year of wasted AI agent spend in one hour
Databricks engineers investigated whether silent failures in MCP tool calls were driving hidden AI-agent costs as usage increased. Unity Gateway emitted OpenTelemetry traces for every invocation, recording tool names, arguments, errors, token counts, latency, and session IDs in a table that Genie One could query with plain-English questions. In a 24-hour sample, the analysis identified seven Jira and Google Drive/Docs bugs responsible for an estimated $499,000 annually in wasted tokens and 12,023 hours of agent wait time, or $1.2 million in lost productivity. The most costly patterns included under-specified interfaces, incompatible argument shapes, and cryptic or misleading errors that triggered retries and recovery loops. Engineers fixed the issues with a coding agent, completing the find-quantify-fix cycle in about an hour; the post recommends designing tools to tolerate reasonable LLM inputs and continuously tracing failures.
How Uken Games reduces observability costs by 87% with ClickHouse
Uken Games replaced Datadog with an open-source observability stack built around ClickHouse to monitor backend services supporting mobile games with millions of players. The migration addressed rising Datadog costs and vendor lock-in while targeting pragmatic feature parity across metrics, traces, and logs. OpenTelemetry collectors run in two layers: lightweight agents on ECS instances pre-filter traces, while an autoscaling gateway batches and buffers writes to SigNoz on ClickHouse; managed Prometheus handles metrics, CloudWatch retains logs, and Grafana provides dashboards and alerts. Uken samples ordinary traces, retains errors and high-latency requests, computes alert metrics without sampling, and limits retention to two weeks. The resulting system stores all traces on one ClickHouse node using about 170 GB, operates without migration downtime, and costs about $12,000 annually—an 87% reduction.
Agent Observability: Monitoring Decisions, Not Requests
Agent observability captures the sequence of decisions an agent makes between model calls, not just request-level metrics, and is distinct from monitoring (health) and evaluation (quality). Unlike deterministic database systems, an agent can succeed at every step yet fail overall—for example, looping forever on a failed fix while spans report success. The primary artifact is a trace, a parent-child tree of spans covering agent invocations, model calls, tool executions, and memory reads and writes. OpenTelemetry GenAI conventions define span types like create_agent and execute_tool, but the standards are still in development and change frequently; vendor support includes Datadog, Honeycomb, and frameworks like LangChain and CrewAI. A reasoning trace layer on top of operational telemetry explains why an agent chose that path; in multi-agent systems the trace must span the full coordination chain, not one agent’s output.
Close the Database Blind Spot with MongoDB Atlas Observability
MongoDB Atlas introduces OpenTelemetry Metrics Sink, a push-based stream that delivers high-fidelity Atlas metrics to OpenTelemetry-compatible platforms, enabling database telemetry to flow alongside application and infrastructure metrics. This closes the gap where database telemetry was often disconnected from application and infrastructure monitoring, forcing operators to jump between dashboards. The push-based delivery eliminates the complexity of pull-based collection, custom exporters, and polling infrastructure, reducing operational overhead and avoiding API rate limits. It supports configuration via the Atlas UI or Terraform, works with custom OTel endpoints, Dynatrace, and New Relic, and is available for dedicated M10+ clusters; a test metric is emitted on creation or update to verify it is working. Combined with Atlas Log Integration and upcoming Events Integration, teams gain unified visibility into database health alongside other signals, accelerating incident response and root cause analysis.
Driving the Agent Quality Flywheel from Your Coding Agent
The quality-flywheel skill addresses the gap between prompt tweaks that improve a few examples and evidence that an agent improved without regressions. A coding agent orchestrates dataset preparation, inference, trace grading, failure analysis, and optimization, using Google's adaptive AutoRaters or custom metrics while keeping the optimizer and evaluator independent. In a travel-concierge example, the revision_honored rubric found that 21% of cases ignored mid-conversation trip changes, exceeding the stated action threshold. A separate software-bug-assistant cycle raised tool-disclosure responses from 0% to 96% across 15 cases, while production traces and online monitors extend the same loop beyond synthetic scenarios. Human approval remains required, and adaptive scores are treated as directional signals rather than absolute grades.
Observability for any agent, anywhere: Production-ready tracing with OpenTelemetry & Unity Catalog on Databricks
Databricks supports writing OpenTelemetry (OTel) traces directly to Unity Catalog, where real-time telemetry is stored in Delta tables for governed analytics and retention. AI traces capture prompts, tool calls, responses, latency, and execution paths, enabling debugging, evaluation, and monitoring, while lakehouse storage also allows SQL queries, dashboards, joins with business data, and PII controls. The managed, serverless ingestion layer uses Zerobus Ingest to accept OTLP over gRPC from collectors and REST integrations, streaming spans, logs, and metrics to Unity Catalog without intermediate message buses. A LangGraph support manager assistant demonstrates instrumentation with mlflow.langchain.autolog() and an @MLflow.trace root span, while Genie is invoked through MCP for data-driven questions. The resulting traces can be searched in MLflow, evaluated at scale, and monitored continuously, with FAQ details stating 200 QPS starting throughput, no storage limit, and Unity Catalog governance options for access control, masking, and filtering.
Triage and Fix with Confidence: heroku run and OTel on Heroku Fir
Heroku Fir addresses production troubleshooting when a release causes failures, such as a background worker breaking after a database schema change, without requiring access to a live formation dyno. Its interactive heroku run command launches a dedicated one-off dyno with the application code and environment variables, relaying terminal input and output for tasks such as rake db:migrate; using it requires a public SSH key and a deployed application. The platform’s native OpenTelemetry (OTel) signals add router request-duration summaries and percentiles, represent heroku.router.connect and heroku.router.service as floats, and attach heroku.app.name to application signals. Together with service.name filtering for first-party services, these capabilities support identifying affected applications, applying isolated fixes, and verifying latency and errors through an observability platform.
OpenTelemetry, Kubernetes, and Fir: Putting it All Together
Modern cloud-native applications span distributed microservices and dynamic environments, making performance diagnosis and failure analysis dependent on observability. The post explains how Fir, Heroku’s next-generation platform, combines Kubernetes infrastructure with OpenTelemetry to provide standardized visibility while hiding Kubernetes complexity behind Heroku’s interface. Fir runs on AWS EKS, uses OCI images and Cloud Native Buildpacks, and includes a Heroku OpenTelemetry Collector that gathers telemetry from applications, the Heroku router, and the Heroku API. OpenTelemetry’s APIs, SDKs, Collector, and OTLP support logs, metrics, and traces, with drains routing data to dashboards, CLI logs, or external observability platforms. The text also distinguishes OTLP gRPC and HTTP transports, noting gRPC efficiency and features versus HTTP’s firewall compatibility, and presents Fir’s open standards as a way to support scaling, hybrid environments, and reduce lock-in.
OpenTelemetry Basics on Heroku Fir
Heroku Fir includes OpenTelemetry as a built-in observability capability, addressing the need to monitor increasingly distributed and complex applications. The post introduces OpenTelemetry’s logs, metrics, and traces, while noting that events and profiles are under development. It distinguishes application-embedded SDKs, which generate and process telemetry, from standalone collectors that receive, process, and export data, although Fir acts as the central hub for telemetry pipelines. The walkthrough uses the Go getting-started app in a Fir space, creates Grafana Cloud credentials and environment variables, converts OTLP headers to JSON, and adds an HTTP Heroku Telemetry Drain with the CLI. After deployment and browser refreshes, application metrics appear in Grafana Cloud, followed by traces and logs.
Heroku CLI v10: Support for Next Generation Heroku Platform
Heroku CLI v10.0.0 updates the command-line tool for the next-generation Fir platform while introducing breaking changes and a Node.js 20 runtime. Fir users must replace several legacy command forms, including logs and ps:stop/ps:restart arguments, and use heroku run:inside because heroku run, ps:exec, ps:copy, ps:forward, and ps:socks do not work with Fir apps. New Fir support includes heroku telemetry commands for OpenTelemetry, a generation selector for spaces:create, generation details in space commands, Fir-aware pipeline and buildpack output, and real-time log streaming with --tail. The release also upgrades oclif to v4.14.36, updates GitHub workflows to Node 20, fixes log color rendering, and is available through heroku update.
Logbook: November 7 to November 14, 2022
The November 7–14, 2022 logbook presents updates intended to make Elixir applications easier to deploy and monitor in production, alongside documentation and storage guidance. Mark’s contribution explains how to set up Github Actions for Elixir CI, including the work needed to automate deployment workflows. A separate guide uses OpenTelemetry to monitor a production Elixir deployment and trace an N+1 query performance problem, while the Django section reports community progress toward more comprehensive Fly documentation. The logbook also points to a MinIO deployment guide for self-hosted, S3-compatible object storage, noting that colocating storage in the same datacenter as an application can matter although using an S3 host such as AWS S3 is described as easier in some cases.
Logbook - 2022-05-05
Fly.io’s 2022-05-05 Logbook reports a broad set of infrastructure, product, security, documentation, and bug-fix updates intended to run customer apps faster and more reliably in VMs close to users worldwide. Remote builders now use regional Docker Hub mirrors, clean unused RUN --mount=type=cache build caches, and perform cleanup at startup and shutdown; the new behavior is default for new builders, while existing builders must be destroyed to receive it. WireGuard gateway improvements address load issues, add regional capacity, and support future redundancy, while another IAD gateway is intended to improve connectivity for customer 6pn networks and CI, especially GitHub Actions. The update also adds API security logging for SOC2, re-enables OpenTelemetry, speeds the proxy build pipeline, changes default persistent volume size from 10GB to 3GB, and ships a dashboard, payment retry feature, VSCode fly.toml validation and documentation changes, alongside GraphQL performance work and fixes for monitoring, invoices, encryption, and Homebrew updates.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.