---
title: "Observability"
description: "251 posts about Observability, summarised, each linking to the original."
---

# Observability
> 251 posts about Observability, summarised, each linking to the original.

## Articles

### [IBM and UC Berkeley Diagnose Why Enterprise Agents Fail Using IT-Bench and MAST](https://yomu.fyi/post/ibm-and-uc-berkeley-diagnose-why-enterprise-agents-fail-using-it-bench.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Ayhan Sebin, Rohan Arora, Saurabh Jha
- Published: Feb 18, 2026
- Type: Benchmark

IBM Research and UC Berkeley analyze how agentic LLM systems fail in enterprise IT automation, including incident triage, logs and metrics queries, and Kubernetes actions across long-horizon tool loops. To move beyond benchmark success rates, the analysis applies MAST (Multi-Agent System Failure Taxonomy) to execution traces, converting failures into structured signatures across system design, inter-agent misalignment, and task verification. It covers 310 ITBench SRE traces from Gemini-3-Flash, Kimi-K2, and GPT-OSS-120B, whose mean recalls were 75.5%, 28.6%, and 12.4%, respectively. Gemini averaged 2.6 failure modes per failed trace, compared with 4.7 for Kimi and 5.3 for GPT-OSS-120B, while FM-3.3, Incorrect Verification, was identified as the strongest predictor across models; the analysis recommends external verification, explicit termination controls, and clarification branches.


### [Building AI-Powered Observability with Heroku Managed Inference and Agents](https://yomu.fyi/post/building-ai-powered-observability-with-heroku-managed-inference-and-ag.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Karunasri (Karuna) Garigipati
- Published: Feb 13, 2026
- Type: Tutorial

Production incident debugging often requires switching among an IDE, Splunk, Sentry, and PagerDuty while translating natural-language questions into platform-specific queries. The described system connects AI coding assistants to observability platforms through the Model Context Protocol (MCP), exposing typed tools for Splunk searches, Honeycomb queries, and Sentry event lookups. Heroku Managed Inference and Agents provides an MCP gateway for SSE transport, allowing servers to run as STDIO processes and be deployed with a Heroku app, an inference add-on, credentials, and an inference token configured in tools such as Cursor or Claude. The post reports ranked production error results, processing more than 175,000 events in minutes for a suspicious-account review, and using natural-language questions to reduce syntax barriers and keep telemetry in the IDE.


### [Engineering at scale, with purpose.](https://yomu.fyi/post/engineering-at-scale-with-purpose.md)
- Company: [Thumbtack](https://yomu.fyi/company/thumbtack.md)
- Author: Thumbtack People Team
- Published: Feb 5, 2026
- Type: Explainer

Senior software engineer Brett Shouse outlines site reliability engineering initiatives and infrastructure modernization efforts underway at Thumbtack. The engineering organization is currently leading an operational project to migrate multiple self-hosted observability services onto a single, unified SaaS platform. This architectural transition consolidates application logs, distributed traces, and system metrics into one accessible view tailored for engineers, customer support staff, and company executives. Moving away from legacy self-hosted monitoring systems reduces systems administration overhead, mitigates alert fatigue, and lowers direct operational infrastructure costs. Concurrently, the reliability team tackles technical debt accumulated from rapid organizational growth by establishing structured incident response processes and automating repetitive operational toil.


### [Introducing SyGra Studio](https://yomu.fyi/post/introducing-sygra-studio.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Surajit Dasgupta, Bidyapati Pradhan, Amit Kumar Saha, Vipul Mittal, Sriram Puttagunta
- Published: Feb 5, 2026
- Type: Announcement

SyGra 2.0.0 introduces Studio, an interactive environment for building, previewing, executing, and debugging synthetic data generation flows on a visual canvas. Users can configure connectors and models, preview source rows, reference automatically created state variables in prompts, define structured outputs, and inspect the generated YAML or JSON graph configuration. Studio streams node-level execution status, token usage, latency, cost, logs, and guardrail outcomes, while storing run history in .executions/runs/\*.json and autosaving drafts. It also runs existing workflows, including a Glaive Code Assistant example with conditional revision loops, tunable dataset and batching inputs, and observable intermediate critiques; generated outputs can support training, evaluation, or annotation workflows.


### [Towards self-driving codebases](https://yomu.fyi/post/towards-self-driving-codebases.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Wilson Lin
- Published: Feb 5, 2026
- Type: Explainer

The post describes research into scaling long-running autonomous coding agents, using a web browser as a complex benchmark and running thousands of agents continuously for one week. An initial single-agent plan failed because the model lost track, stopped prematurely, and stalled on complex details; shared-state coordination then introduced locking contention, while more structured planner, executor, worker, and judge roles improved ownership and throughput. The team built a Rust-based harness on one large Linux VM, logged agent messages and tool activity for replay and analysis, and iterated through designs that eventually favored a more dynamic system with freshness mechanisms and fewer overloaded roles. The research concludes that architecture, prompts, observability, explicit throughput trade-offs, and anti-fragile recovery matter, reporting linear scaling of token throughput with minimal harness overhead while humans retained taste, judgment, and direction.


### [Demystifying user journeys: Revolutionizing troubleshooting with auto tracking](https://yomu.fyi/post/demystifying-user-journeys-revolutionizing-troubleshooting-with-auto-t.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Alex Prokofiev
- Published: Dec 23, 2025
- Type: Problem & solution

Troubleshooting critical issues in the Grab app was hindered by incomplete user-journey data, because manually instrumenting hundreds of UI components left interaction gaps and delayed diagnosis. AutoTrack is a new SDK that records application state, user interactions, and UI screens in real time, creating an end-to-end session chronicle from launch through navigation. It tracks lifecycle transitions and launch mechanisms such as deeplinks and push notifications, while UIViewController, Activity, and Fragment lifecycle events identify the displayed screen. Screen metadata includes the full hierarchy as JSON; after personal information is redacted for GDPR/PDPA compliance, that data is used to train an AI model that generates an HTML representation of the screen and interactions. AutoTrack supports reproducing elusive bugs, automated issue assignment, UI test generation, and analytics-trigger validation, with future work adding network and performance signals, privacy guardrails, and automated triage.


### [Real-time data quality monitoring: Kafka stream contracts with syntactic and semantic test](https://yomu.fyi/post/real-time-data-quality-monitoring-kafka-stream-contracts-with-syntacti.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Yuanzhe Liu
- Published: Nov 26, 2025
- Type: Problem & solution

Kafka stream data can fail syntactically when actual records do not match expected schemas, or semantically when fields violate agreed patterns, ranges, or constants. The described platform addresses this gap with user-defined data contracts covering schema agreements, field-level semantic rules, stream configuration, and ownership metadata. An LLM recommends semantic rules from schemas and anonymized samples, while a transformation engine converts them into Test Runner configurations and inverse SQL queries. A long-running FlinkSQL Test Runner reads Kafka with its own consumer group, publishes issue events and stores them in AWS S3; Genchi sends Slack alerts and Coban shows violating fields and sample records. Deployed across 100+ critical topics, the solution provides real-time detection, alerting, and visibility, with future work targeting producer tracing, cross-field tests, completeness, freshness, and DTO change detection.


### [Treating Data as Code at Two Sigma](https://yomu.fyi/post/treating-data-as-code-at-two-sigma.md)
- Company: [Two Sigma](https://yomu.fyi/company/two-sigma.md)
- Author: Emily Majewski
- Published: Nov 13, 2025
- Type: Problem & solution

Scaling research and trading platforms across thousands of data sources created bottlenecks at Two Sigma, where reliance on database snapshots and fragmented infrastructure slowed dataset delivery to data scientists. To address operational costs and architectural complexity, the data engineering team adopted software development principles by treating data as code. The organization migrated to Google BigQuery's serverless architecture, standardized SQL transformations using dbt, and defined declarative pipelines with Terraform under continuous integration workflows. Internal tooling was introduced to track directed acyclic graph dependencies, automate anomaly detection, and streamline data discovery. These shifts eliminated manual data movement routines, reduced operational overhead, and enabled formal data contracts to safeguard downstream consumers and emerging language model integrations.


### [Introducing Cursor for Enterprise](https://yomu.fyi/post/introducing-cursor-for-enterprise.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Cursor
- Published: Oct 31, 2025
- Type: Announcement

Cursor is introducing enterprise features intended to make its coding agents more capable, transparent, and secure for organizations already using the product. Hooks expose the agent loop to custom scripts that can log actions and tool calls, enforce compliance or command allowlists, scrub sensitive material, connect external systems, and trigger automations; they can be distributed through MDM or Cursor’s cloud option. Team Rules provide shared context and can recommend or require standards such as API schemas, conventions, and workflows, while upgraded analytics report activity, adoption, AI lines of code, directory-group filters, and exportable data with updates every two minutes. The release also adds an Audit Log covering 19 event types and Sandbox Mode, which restricts network and file access by default, with administrators controlling availability and team-wide Git and network access.


### [MongoDB Community Edition to Atlas: A Migration Masterclass With BharatPE](https://yomu.fyi/post/mongodb-community-edition-to-atlas-a-migration-masterclass-with-bharat.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Nick Bell
- Published: Sep 21, 2025
- Type: Problem & solution

BharatPE, an Indian fintech processing over ₹12,000 crore monthly across 450 cities, operated a 45-terabyte self-hosted MongoDB Community Edition deployment. Managing three sharded three-node clusters resulted in uneven data distribution, high maintenance costs, scaling friction, and disaster recovery gaps under strict regulatory requirements. To modernize infrastructure, the database operations team partnered with MongoDB to execute a structured five-stage migration to MongoDB Atlas. The initiative utilized mongosync for secure data transition alongside environment mirroring and automated integrity validation scripts. Transitioning to MongoDB Atlas yielded a 40 percent improvement in query response times and provided automated failover backed by a 99.995 percent uptime SLA.


### [Triage and Fix with Confidence: heroku run and OTel on Heroku Fir](https://yomu.fyi/post/triage-and-fix-with-confidence-heroku-run-and-otel-on-heroku-fir.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Su Glasgo
- Published: Sep 8, 2025
- Type: Explainer

Heroku Fir addresses production troubleshooting when a release causes failures, such as a background worker breaking after a database schema change, without requiring access to a live formation dyno. Its interactive heroku run command launches a dedicated one-off dyno with the application code and environment variables, relaying terminal input and output for tasks such as rake db:migrate; using it requires a public SSH key and a deployed application. The platform’s native OpenTelemetry (OTel) signals add router request-duration summaries and percentiles, represent heroku.router.connect and heroku.router.service as floats, and attach heroku.app.name to application signals. Together with service.name filtering for first-party services, these capabilities support identifying affected applications, applying isolated fixes, and verifying latency and errors through an observability platform.


### [OpenTelemetry Basics on Heroku Fir](https://yomu.fyi/post/opentelemetry-basics-on-heroku-fir.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Alex Arnell
- Published: May 27, 2025
- Type: Tutorial

Heroku Fir includes OpenTelemetry as a built-in observability capability, addressing the need to monitor increasingly distributed and complex applications. The post introduces OpenTelemetry’s logs, metrics, and traces, while noting that events and profiles are under development. It distinguishes application-embedded SDKs, which generate and process telemetry, from standalone collectors that receive, process, and export data, although Fir acts as the central hub for telemetry pipelines. The walkthrough uses the Go getting-started app in a Fir space, creates Grafana Cloud credentials and environment variables, converts OTLP headers to JSON, and adds an HTTP Heroku Telemetry Drain with the CLI. After deployment and browser refreshes, application metrics appear in Grafana Cloud, followed by traces and logs.


### [How to Add the Moesif API Observability Add-On to Your Heroku Applications](https://yomu.fyi/post/how-to-add-the-moesif-api-observability-add-on-to-your-heroku-applicat.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Matt Tanner
- Published: Apr 16, 2025
- Type: Tutorial

Heroku applications can add the Moesif API Observability add-on to analyze API traffic and support monetization. The setup uses either the Heroku CLI or Dashboard: the CLI command is heroku addons:create moesif, while the UI path goes through Resources, Find more add-ons, and Metrics and Analytics. Provisioning creates a MOESIF\_APPLICATION\_ID config variable, and the application then requires the Moesif SDK initialized with that value. The guide also covers copying the variable into a local .env file with Heroku Local and excluding that file from Git through .gitignore. Once configured, API events should flow into Moesif for live logs, time-series metrics, funnels, retention, alerting, and billing features, while Heroku handles add-on billing and user management.


### [Improving Hugo stability and addressing oncall challenges through automation](https://yomu.fyi/post/improving-hugo-stability-and-addressing-oncall-challenges-through-auto.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shuguang Xiang
- Published: Mar 20, 2025
- Type: Problem & solution

Hugo, Grab’s data-ingestion platform for a data lake with more than 4,000 user-onboarded pipelines, faces failures that can leave users with limited visibility and on-call teams handling ad-hoc diagnosis, unclear resolution steps, and DPIs without clear RCA. The proposed automation architecture combines Signal, Diagnosis, RCA Table, Auto-resolution, Data Health API, and Data Health Workbench modules to monitor table health, identify root causes and assignees, execute fixes, and expose status to users and other platforms. Signals come from Airflow failure callbacks and Genchi’s SLA and data-completeness checks; diagnosis avoids parsing extensive logs, runs analysers in parallel according to signal type, and supports asynchronous custom handlers such as retry with backoff for transient errors. Reported outcomes include improved data visibility, faster issue resolution, reduced Hugo on-call workload, scalable handling of transient failures, and more precise data-contract incident management, while next steps include runtime optimisation, richer workbench details, and Flink diagnostics.


### [Building a Spark observability product with StarRocks: Real-time and historical performance analysis](https://yomu.fyi/post/building-a-spark-observability-product-with-starrocks-real-time-and-hi.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Huong Vuong
- Published: Mar 6, 2025
- Type: Problem & solution

Grab’s Iris project replaces a fragmented Spark observability stack with a centralised platform for real-time and historical analysis. The previous setup split data between Grafana, Superset, InfluxDB, and a data lake, creating usability, access-control, pipeline, metadata, and high-cardinality query challenges. The new architecture uses StarRocks as a unified store, ingesting Kafka directly, serving a custom Iris web application and Superset, and backing up to S3. Its relational data model links worker metrics through worker\_uuid and Spark event metrics through app\_id, while materialised views and dynamic partitioning support responsive queries and data retention as volumes grow. The resulting system consolidates metrics and metadata, simplifies ingestion and offline processing, and supports job-level showback/chargeback; the roadmap includes recommendations, advanced analytics, machine learning, and scalability improvements.


### [Enhance your monitoring by integrating Stripe events with AWS CloudWatch Log Groups](https://yomu.fyi/post/enhance-your-monitoring-by-integrating-stripe-events-with-aws-cloudwat.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Andrew Robinson
- Published: Dec 9, 2024
- Type: Tutorial

Stripe events notify account owners about changes such as successful charges, failed invoice payments, and available reconciliation reports. Because Stripe retains events for 13 months but exposes older events than 30 days only as summaries, Amazon EventBridge can route them into an AWS account and Amazon CloudWatch Log Groups can provide longer-term monitoring and analysis. The setup uses CloudWatch metric filters to match event types such as invoice.payment\_failed, convert matches into custom metrics, and support alarms, Amazon SNS notifications, dashboards, and Logs Insights queries. For example, a failed-invoice alarm can use a five-minute observation period, Sum statistics, and a threshold based on expected failure volume or anomaly detection. The resulting setup supports near-real-time visibility into successful charges and failed payments, historical trend analysis, troubleshooting, and responses to unusual activity.


### [How we seamlessly migrated high volume real-time streaming traffic from one service to another with zero data loss and duplication](https://yomu.fyi/post/how-we-seamlessly-migrated-high-volume-real-time-streaming-traffic-fro.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Md Riyadh
- Published: Dec 5, 2024
- Type: Problem & solution

Grab migrated its stream-processing (write) functionality from an existing service into a separate service while handling 20,000 reads per second from 16 Kafka streams and writing to output streams and DynamoDB tables. The migration required gradual stream-by-stream rollout, production validation, and a no-downtime switchover; feature flags were rejected because deployment timing could cause loss, duplication, or inconsistent values. The team moved processing into shared /commons code, created temporary validation sinks, and scheduled each switchover with a shared map\[string\]time.Time, allowing both services to change writers at the same time. During each cutover, both services consumed streams, the old service stopped writing to resources, and monitoring continued before old consumption was removed; the result was zero data loss or duplication, no downtime, and completion in about three weeks. It was temporary and tested, but rolling back either service could create inconsistency if systems fell out of sync.


### [Avoiding silent errors in your Stripe integration](https://yomu.fyi/post/avoiding-silent-errors-in-your-stripe-integration.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Abe Haskins
- Published: Nov 21, 2024
- Type: Tutorial

Stripe integrations can appear healthy while API and webhook failures accumulate unnoticed when applications lack robust logging and alerting. The post demonstrates Stripe Workbench, a Dashboard view that stores available information by default, obfuscates sensitive data, and provides searchable logs without code changes or additional software. In the example, about 50% of daily API calls fail; the Errors tab groups failures and connects each type to sample requests and related logs, revealing repeated card-verification errors and a request associated with an “Attacker Engineering Test Key” whose Go user-agent conflicted with the stated .NET backend. The investigation identifies a team testing attack-like API calls in production and recommends stopping production-key testing, avoiding test requests in production, and using Sandboxes.


### [Tinder API Style Guide — Part 1](https://yomu.fyi/post/tinder-api-style-guide-part-1.md)
- Company: [Tinder](https://yomu.fyi/company/tinder.md)
- Author: Tinder
- Published: Sep 6, 2024
- Type: Explainer

Tinder’s rapid growth and expansion into domain-centric teams produced inconsistent API practices across services, including mixed error conventions, fragmented schemas, uneven versioning, unclear URIs, and weak API contracts. Part 1 presents standards for the RESTful APIs used across the platform, beginning with the URI pattern https://service-mesh-host-name/\[version\]/\[service-name\]/\[resource\]/\[sub-resource\] and guidance on concise, plural, kebab-cased paths. It defines path-prefix versioning, retaining the previous version until clients migrate and creating new versions only for breaking changes such as removing fields, changing types, or renaming fields. The guide also covers HTTP methods, standard and custom request headers, the X-Tinder- naming convention, and request tracing through MDC-populated identifiers and centralized logging. Request and response bodies, status codes, and enforcement tools are reserved for Part 2.


### [Simple error handling strategies with Stripe Workbench](https://yomu.fyi/post/simple-error-handling-strategies-with-stripe-workbench.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Ben Smith
- Published: Sep 2, 2024
- Type: Tutorial

Stripe Workbench addresses the difficulty of finding and understanding errors produced by Stripe API invocations. It gives developers a view of recent errors, groups repeated failures, shows their frequency, and links summaries to request logs and actionable resolution guidance. The article describes Stripe error categories such as StripeCardError, StripeInvalidRequestError, and StripeConnectionError, then demonstrates Node.js try/catch handling that branches on the error type. It also covers webhook responses for payment\_intent.payment\_failed events, including retrieving event.data.object.last\_payment\_error, and explains how objects such as PaymentIntents, Setup Intents, Invoices, Setup Attempts, Payouts, and Refunds retain failure information for later inspection. Combined, these mechanisms support targeted responses, troubleshooting through detailed request and response data, and more reliable Stripe integrations.


[Newer posts](https://yomu.fyi/topic/observability/page/10.md) · [Older posts](https://yomu.fyi/topic/observability/page/12.md)