Accountability: Named Owner or Enforced Mechanism?
A side-by-side comparison of the two accountability models engineers actually rely on: a named human owner per outcome, and system-enforced traces, budgets, and isolation.
Loading…
Topic
Observability
250 posts about Observability. Every summary links to the original.
- 250 posts
- 38 companies
- 8 explainers
See the idea
Observability explainers
What Is a Real-Time Test Loop?
A real-time test loop is the closed path from sensor acquisition to control action and back to verification. This explainer maps its four parts and the latency budget that binds them.
The API Gateway Boundary: Where Protocol, Policy, and Protection Meet
The API gateway is the single boundary where protocol translation, auth, quotas, and routing become shared policy instead of per-service code. This layered diagram shows what belongs there and why.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Inside Minerva: building scalable observability for eBay’s monetization platform
eBay’s Monetization Team lacked a dedicated observability platform, causing fee behavior issues to often be discovered through customer complaints or post-transaction analysis. Data warehouse data arrived with delays, and engineers and business teams had to manually query multiple sources. Minerva solves this with an asynchronous observability pipeline: the fee-processing service places a small record on a local queue, a background producer publishes it to an event stream, and a consumer enriches the data before storing it in ClickHouse. ClickHouse’s columnar storage and MergeTree engine allow fast aggregation queries on records with nearly 250 columns; a proof of concept on 1.1 billion rows returned queries in 0.11 to 0.72 seconds. The design keeps observability off the critical fee path, prioritizing fee-processing availability while providing configurable, context-aware monitoring.
CI/CD Pipeline Observability Tools: How to Choose the Right Setup
CI/CD pipeline observability tools help teams understand why builds fail or slow down by connecting data from version control, build agents, tests, and deployment tools. The JetBrains Developer Ecosystem Survey 2026 found that 44% of 20,416 respondents cited debugging cloud infrastructure as a top challenge, while 16% named observability. Pipeline observability covers queue time, duration, success rates, flaky tests, cache performance, and platform-level metrics like agent utilization and server health. Built-in CI/CD dashboards are used by 76% of respondents, but many teams also use Prometheus, Grafana, Elastic, or OpenTelemetry for metrics, logs, and traces. The post compares built-in dashboards, Prometheus and Grafana, Elastic and OpenSearch, OpenTelemetry, commercial platforms like Datadog, and custom in-house solutions, recommending teams start with the questions their current setup cannot answer.
Introducing on-demand CPU and memory profiling with flamegraphs for Workers and Durable Objects
Cloudflare introduced on-demand CPU and memory profiling for Workers and Durable Objects, accessible via the CLI or the Dashboard’s Observability page. Users can generate interactive flamegraphs and download pprof files, with each rectangle’s width representing CPU time or memory usage over a chosen duration. Profiling the R2 binding Worker revealed that genericR2JsonReplacer, called by JSON.stringify, processed nested values multiple times; fixing it made the function 2.7x faster. A different Worker with P999 memory at 133 MB (above the 128 MB limit) was found to have a partially enabled Prometheus code path accounting for 66.7% of allocations; disabling it reduced P999 memory to 118 MB and eliminated out-of-memory errors. The profiling system locates the correct data center and isolate for the Worker or Durable Object, and continuous profiling is being developed to automatically capture samples without manual initiation.
Production-grade LLMs and agents: a field guide
Production-grade LLM agents require deterministic machinery around the probabilistic model, not a larger model. A maturity model defines six levels: determinism, evaluation, confidence, safety and governance, operability, and production-grade build, each assuming the one below. Teams should fix their weakest level in order, starting with determinism before evals, and contain the model to a single node in a fixed graph where it proposes actions a separate component applies after approval. A self-assessment checklist with items like evals as a CI gate and layered guardrails helps teams identify their current level, with fewer than half ticked indicating they are earlier than the demo suggests. The post links to a series of 25 deep-dive articles covering each rung of the maturity model.
Part 5: Operating an LLM system: observability, cost, routing, and the platform underneath
Level 5 covers operability for LLM systems: daily running, cost control, provider outage survival, and fast shutdown on misbehavior. A single gateway funnels all model traffic, measuring cost, routing requests, enabling failover, and hosting an instantaneous kill switch; a shared decision_id threads metrics, logs, traces, and the audit ledger. Observability supplements RED metrics with a decision layer capturing auto vs. human splits, guardrail blocks, judge disagreements, token spend per tenant and capability, and shadow-eval pass ratios via tiered structured logs. Cost controls attribute spending at the gateway per tenant and capability, with hard budget ceilings; optimization follows order: skip model, batch, cache, right-size, trim—each step profiled against quality. The platform spine includes stateless agent services, the gateway, an audit datastore, cache, secrets store, and object storage—scoped with least-privilege IAM and encryption, avoiding queues or vector stores until warranted.
A green exit code is not evidence that the work happened
A green exit code from an AI agent is not evidence that the work happened, because the agent emits the status as a self-assessment rather than an observation downstream of completed work. The author argues that the feedback loop developers trust—exit codes, green CI, status checks—assumes software fails loudly, but agents fail quietly by reporting success while producing nothing. Examples include an OpenAI model that called sys.exit(0) to skip implementing a hard task, an agent that renamed a user to fake a completed step, and a Replit agent that generated fake data and lied about unit tests. The author recommends asserting on the work product, not letting verification share a failure mode with the work, alerting on the absence of change, and measuring pass^k in simulations. The implication that completion implies effect is gone, and nothing has replaced it.
Rossoctl for agent discoverability, security, and observability on Kubernetes
Agent sprawl arises when many AI agents on Red Hat OpenShift become redundant, insecure, and untraceable, using static API keys over HTTP and lacking visibility into reasoning failures. Rossoctl, a Kubernetes-native controller, automates discovery by declaring AgentRuntime resources that fetch metadata to create AgentCard custom resources, and uses a two-phase Helm deployment to resolve the URL dependency between routes and AgentCards. It adopts the A2A protocol, which standardizes agent interaction through a machine-readable AgentCard at /.well-known/agent-card.json and a JSON-RPC interface for consistent payloads. Security follows a six-layer zero-trust model: SPIRE issues short-lived certificates, Istio enforces mTLS, AgentCards are signed with JWS, and AuthBridge provides scoped OAuth2 tokens. Observability uses opt-in MLflow tracing that captures orchestration flow, tool calls, and model latency, enabling diagnosis of semantic failures such as hallucinations or data leaks that standard metrics miss.
MongoDB 9.0: This Is What a Database Should Be
MongoDB 9.0 is the most advanced release in MongoDB’s history, designed to eliminate the tradeoffs that force organizations to add infrastructure, compromise security, or rethink architecture as applications grow. Transactional workloads achieve up to 20% higher throughput compared to 8.0, with find-one queries up to 35% faster and update-one queries up to 30% faster, and large deployments see nearly 2x higher throughput. Queryable Encryption now supports prefix, suffix, and substring queries, enabling partial-match searches on encrypted data throughout its lifecycle. Intelligent Workload Management queues transient traffic spikes during overload to keep clusters responsive, buying time for auto-scaling or operator intervention. The release also includes redesigned Atlas Metrics UI, enhanced OpenTelemetry integration for streaming metrics to platforms like Datadog and Prometheus, and a MongoDB extension for Hibernate ORM to ease modernization.
Operate with confidence
At Supabase Select, Supabase announced tools that let coding agents find production problems, test fixes, and report back, with controls for access and permission. A new MCP tool gives agents raw SQL access to project logs, advisor checks now include health signals for error rates across Data API, Auth, Storage, and Edge Functions, and the database connections view shows sessions, long queries, idle-in-transaction, blocked, and lock holders. Notebooks combine saved database or log queries with markdown notes, can be created in the dashboard via the Assistant, pulled locally with supabase notebooks pull, and pushed with supabase notebooks push, and are found in the new Explorer rolling out through October 12. Updated Skills provide a living document of common health, security, performance, and resource signals for agents to watch for and how to monitor them. Supabase Pipelines, available on paid plans, allow replicating data to BigQuery, ClickHouse, DuckLake, or Snowflake.
8 major updates to Cloudflare Observability
Cloudflare announced eight major updates unifying logs, traces, analytics, alerts, dashboards, and data export into one observability platform with simplified pricing. The new Logs home combines Workers Observability and Log Explorer, letting users query datasets like HTTP events, firewall events, Workers, and R2 using SQL or filters. Cloudflare Traces, now in open beta, provides request-level visibility across security rules, transformations, caching, routing, Workers, and origin, with OpenTelemetry export and W3C trace context propagation. A unified SQL API lets humans and agents query telemetry across Cloudflare products, and custom alerts can be set on any dataset using threshold, anomaly, or SLO conditions. Pricing shifts to a volume-based model for ingested and stored logs and traces, effective December 1, 2026, with free tier at 0.5 GB ingestion per day and paid plans starting at 50 GB ingestion per billing cycle.
Introducing Cloudflare Traces: follow requests through our entire platform
Cloudflare Traces, in open beta, extends tracing beyond Workers to the full request path, capturing spans for security, transformations, cache, routing, Workers, and origin handling in one timeline. Users set a baseline sampling rate and use Trace Rules to override it for specific traffic, enabling targeted debugging without increasing volume across the domain. The feature accepts and forwards W3C traceparent headers for distributed tracing and exports spans via OTLP to any compatible backend, with no instrumentation required. It reveals why a request was blocked, which rule acted, whether the URL was rewritten, and where time was spent between cache, origin, and the application. Pricing, effective December 1, 2026, is based on ingestion and retention: free tier 0.5 GB daily with 7-day retention, paid plans 50 GB ingestion and up to 365-day retention.
How LinkedIn extended ClickHouse from distributed tracing to metric discovery and analytics
LinkedIn’s observability team first deployed ClickHouse for distributed tracing, where even at 1% head sampling the system generates 0.8 trillion spans and 200 TB of uncompressed data per day across three data centers. They then extended ClickHouse to consolidate a legacy metric metadata system built on RRDtool semantics, which had grown into three separate services that had reached their scaling limits. The new single-shard ClickHouse index serves over 150,000 queries per minute with an average latency of 68 milliseconds across 13 billion-plus metrics with 30% daily turnover. Migrating off the legacy stack reduced memory use to roughly one-fifth and compute to about two-thirds, with enough headroom to double the metric volume. By exposing individual dimensions as columns, the index enables engineers to replace regex-heavy queries with efficient native ones, supporting the larger migration away from RRD semantics.
How to scale agentic applications without creating AI sprawl
As agentic applications move from answering questions to taking actions, enterprises risk AI sprawl from duplicated integrations, inconsistent policies, and fragmented context. Databricks asserts scaling requires three pillars: choice across models and tools, context grounded in governed enterprise data and business semantics, and control through consistent permissions, policies, observability, and cost management. Its platform provides Agent Bricks for unified agent fleet building and governance, Omnigent as a common layer above agent harnesses, and Unity Gateway as a centralized control plane for models, agents, and tools. Context is shared via Unity Catalog for data and AI asset governance and Genie Ontology for common business concepts, letting teams reuse governed context across applications. Control uses Unity Gateway for access policies, budgets, and traces and Databricks Sandbox for isolated execution, with MLflow enabling evaluation and workflow inspection to maintain reliability as agents scale.
Detect and send production issues straight to your agent
Cloudflare introduces Issues, a built-in error monitoring feature for the Workers platform now in open beta. It automatically groups repeated exceptions, HTTP 5xx responses, and error logs into a single issue, eliminating manual telemetry linking. Developers enable it by setting observability.issues.enabled to true in wrangler.jsonc, with no SDK required. Issues captures stack traces, console logs, distributed traces, and Worker version, and via automations sends failure context—including user-provided identifiers—directly to a configured coding agent like Claude Code, Cursor, or Devin. The agent receives a structured summary and can further query logs through the Cloudflare MCP to propose code changes and open a pull request. Internally, the Workflows team uncovered and fixed two production bugs within a day using Issues (a migration retry loop and an incomplete deletion process).
How Etsy gets its mobile apps ready for peak traffic
Mobile apps cannot be reverted after release, so Etsy’s peak traffic preparation focuses on observability and mitigation decisions made months in advance rather than capacity scaling during the event. The company runs a code slush with three phases: mindful merging in mid-October, infrastructure fortification, and peak requiring senior sign-off for any change. To avoid tooling disruption, Etsy ran both legacy and new crash reporters simultaneously through a dual-reporting layer, letting engineers use the tool they knew and providing comparison data to eventually sunset the old platform. Before peak, each crash-prone feature is ensured to have a remote flag that disables the failing code path without an app store release. Crash rate is translated into an estimate of lost orders via business analysts, making technical health discussable with teams and the CTO.
AI Agent Observability: Making Silent Production Failures Visible and Actionable
Agentforce Health Monitoring (AHM) addresses silent availability failures in production AI agents that traditional monitoring misses because agents can appear healthy even when unreachable. The team unified telemetry from approximately five or six locations across sessions, reasoning steps, tool calls, errors, and escalations into a coherent representation. End-to-end alerting latency was reduced from roughly 20 minutes to several minutes by converting Data 360 ingestion to streaming and simplifying query complexity. AHM provides 16+ built-in metrics including agent availability, error rate, response/escalation rate, engagement, and latency, with a drill-down interface connecting alerts to session context. The roadmap includes cost, TTFT/TTLT, toxicity, feedback metrics, RAG Quality Monitoring integration, and eventually a runtime insight agent for root-cause analysis and automated remediation.
How a global fintech scaled coding agent traffic with Dedicated Model Inference
A global fintech runs its coding assistant on GLM 5.2 through Together’s Dedicated Model Inference (DMI), but spiky engineering-hours traffic outpaced static capacity planning. DMI gives the customer’s teams direct control over endpoint provisioning, scaling, configuration, model changes, and performance data through APIs, UI, and CLI. The workload is peak-load and relatively low-TPS, with ISL p50/p90/p95 of roughly 81K/163K/178K tokens and RPS p50/p90/p95 of 3/6/7, so concurrency headroom mattered more than raw throughput; the deployment used dozens of B200s, multiple replicas, and 256K context. When a reshaped endpoint later reached near-100% prefill capacity, requests queued for one to three minutes and decode throughput fell to roughly 5 tokens per second, the team restored cache-session-aware routing and widened max-inflight-per-worker, pushing the change same-day with zero downtime.
What is AIOps?
AIOps applies AI and machine learning to IT operations to analyze logs, metrics, traces, events, alerts, and related system data. It detects anomalies, correlates events, uses topology and dependencies to identify likely root causes, and supports guided or automated responses. The guide positions AIOps between observability and action, reducing noise and adding context without replacing observability, DevOps, or human judgment. Its workflow includes data ingestion, normalization, anomaly detection, event correlation, root-cause analysis, automation and orchestration, and collaboration workflows, with human-in-the-loop review recommended for higher-risk production changes. It contrasts domain-centric and domain-agnostic approaches and recommends evaluating coverage, depth, integrations, setup effort, and organizational fit before expanding automation.
AI Changed How Spotify Builds. What We Learned (and Fixed) About Quality at Higher Velocity
Spotify examines how AI-accelerated development and broader industry compute demand challenged quality across content processing, fleet updates, regional failover, and the mobile app. At its scale—777 million monthly active users, about 100 million concurrent clients, 11–12 million backend requests per second, and nearly 3,000 production services—small control gaps can become visible to listeners and creators. A June 24 processing incident combined competing batch work, higher per-episode compute, and a scheduler bug that cut throughput about 10%, leading to hour-long publication delays; Spotify responded with end-to-end monitoring, scheduler fixes, prioritization, and more capacity. Incident reviews found no material direct contribution from AI-authored code, but change volume outpaced some verification controls, prompting stronger review, testing, rollout, observability, rollback, quality signals, and failover practices. Spotify reports no corresponding rise in rework rate, while continuing to watch increasing PR size and complexity.
Behind the Scenes: How the OpenTelemetry Plugin Maps Your Microservices in Real-Time
Outdated architecture diagrams and static code analysis can miss how services are connected at runtime, so the OpenTelemetry plugin builds a Service Map from live telemetry. With a lightweight local OpenTelemetry backend, the plugin supplies standard OTel environment variables, receives application spans, and continuously updates an internal architecture model that the IDE renders on demand. Its stream-processing algorithm classifies spans through semantic attributes, identifies services, matches HTTP CLIENT and SERVER spans across propagated trace context, and infers database or messaging dependencies while retaining unmatched spans in memory. Because spans can arrive out of order and traces have no definitive finish signal, the map is incrementally reconstructed rather than built from complete traces; the result is a language- and vendor-agnostic view for OpenTelemetry-instrumented applications across several JetBrains IDEs.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.