Loading…
Topic
198 posts about Monitoring. Every summary links to the original.
See the idea
Anomaly detection identifies meaningful departures from expected behavior. This explainer shows how signals, baselines, context, and alert routing turn unusual measurements into engineer-ready findings.
Autoscaling trades responsiveness, stability, and cost: this explainer shows how signal choice, scaling direction, and capacity limits shape the right control loop for a workload.
Airflow is a workflow orchestration platform that schedules, monitors, and manages data pipelines by defining tasks as code, executing them in dependency order, and tracking their success or failure.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Supabase has launched Log Drains for its Pro tier, allowing users to send Supabase logs to their own logging backend and investigate them alongside the rest of their stack. The feature addresses the separation between Supabase logs and external observability tools by exporting events from Postgres, API Gateway, Auth, Storage, Edge Functions, and Realtime. Logs are sent over HTTP in batches of up to 250 entries or every second, whichever comes first, with Gzip compression when supported by the destination. Supported destinations include Sentry, Grafana Loki, Datadog, AWS S3, Axiom, and generic HTTP endpoints; users configure drains in the dashboard and can create multiple drains, with pricing of $60 per drain per project, $0.20 per million events, and $0.09 per GB of egress.
ClickHouse 26.2 introduces an embedded distribution of the ClickStack observability UI packaged directly inside the ClickHouse binary. Designed for local exploration and instance inspection, this distribution allows developers to visualize internal query logs, traces, and system metrics without deploying external services. To preserve build integrity and keep the added binary footprint around 4.1 to 4.2 MB, the team avoided a Node.js dependency by embedding a pre-built static Next.js application using CMake-generated C++ structs. Supporting components such as MongoDB, Express, and the Pyodide WebAssembly runtime were omitted, disabling alerting and persistent dashboards in favor of simplicity. Users can access the interface directly over HTTP to monitor local databases and query performance out of the box.
IBM Research and UC Berkeley analyze how agentic LLM systems fail in enterprise IT automation, including incident triage, logs and metrics queries, and Kubernetes actions across long-horizon tool loops. To move beyond benchmark success rates, the analysis applies MAST (Multi-Agent System Failure Taxonomy) to execution traces, converting failures into structured signatures across system design, inter-agent misalignment, and task verification. It covers 310 ITBench SRE traces from Gemini-3-Flash, Kimi-K2, and GPT-OSS-120B, whose mean recalls were 75.5%, 28.6%, and 12.4%, respectively. Gemini averaged 2.6 failure modes per failed trace, compared with 4.7 for Kimi and 5.3 for GPT-OSS-120B, while FM-3.3, Incorrect Verification, was identified as the strongest predictor across models; the analysis recommends external verification, explicit termination controls, and clarification branches.
Supabase reports a 3-hour-and-42-minute outage on February 12–13, 2026, affecting every service in its us-east-2 region, including Postgres, Auth, Data APIs, Edge Functions, Storage, and Realtime. A deployment of an internal monitoring service enabled AWS VPC Block Public Access in block-bidirectional mode, blocking internet gateway traffic across production VPCs in the region. The incident response was delayed because alerts pointed toward shared services, the relevant CloudTrail event was inconspicuous, pre-production lacked us-east-2, and infrastructure specialists were not involved initially. Supabase restored connectivity by rolling back and removing the monitoring stack, then audited regions, added AWS Organizations policies and IaC safeguards, and planned account isolation, external connectivity probes, production-parity testing, and improved escalation.
Ashmann Syngle, a backend software engineer on Thumbtack's Pricing team, focuses on systems that connect local service professionals with homeowners. To improve matching and revenue operations, the team recently deployed new pricing features, launched experiments, and partnered closely with Data Science and Monetization Experience groups. Current engineering efforts center on infrastructure enhancements, specifically upgrading alerting and monitoring across owned services to raise system reliability and operational efficiency. Because the team maintains complex monetization systems within a codebase that has evolved over many years, engineers conduct deep system analyses to evaluate edge cases before implementing foundational platform modifications. Thumbtack supports these initiatives through a virtual-first operational model complemented by regular in-person offsites.
The 2025 BFCM live globe was implemented inside a browser-based pinball machine to represent real-time merchant sales and celebrate milestones. Built with three.js and react-three-fiber, the experience runs at 120fps, uses Server-Sent Events for order arcs and live metrics, includes a 128x32 shader-driven dot-matrix display, and supports AR through react-three/xr. For performance, the team simulated pinball physics in 2D with Rapier2D and used raycasts for ramps and elevation, reducing the physics step from 2ms to 0.2ms. The scene also includes interactive mini games, infrastructure statistics, Easter eggs, and milestone-triggered reactions, while Apache FlinkSQL pipelines process event streams behind the commerce visualizations. The globe was also displayed on the Exosphere in Las Vegas for the third consecutive year.
Shopify’s BFCM readiness program addresses the challenge of operating critical commerce infrastructure at traffic levels informed by 2024 records, while preparing APIs with no prior holiday-traffic history. Starting in March, teams combined capacity planning on Google Cloud, an infrastructure roadmap, risk assessments, chaos-engineering Game Days, and a Resiliency Matrix covering failure modes, recovery objectives, runbooks, and on-call coverage. Genghis load tests and Toxiproxy network faults ran across three GCP regions, while five April–October scale tests exercised browsing, checkout, analytics, webhooks, regional failover, and cascading failures; testing reached 146 million requests per minute and a p99 target of 200 million. The work exposed and addressed Kafka partition, memory, timeout, routing, queueing, and failover issues, while the operational plan adds monitoring, 24/7 incident coverage, merchant communications, live tuning, and a post-BFCM review.
Transitioning from relational databases to MongoDB often leads developers to replicate SQL habits, causing severe performance bottlenecks, brittle schemas, and operational challenges. Effective document design requires balancing referencing and embedding based on query access patterns and update frequencies, rather than placing every entity into separate collections or excessively nesting large documents. Database performance depends on matching indexes to specific query shapes and field orders, which developers can systematically verify by inspecting execution plans using explain plans. Utilizing the built-in aggregation framework allows filtering, grouping, and calculations to run natively inside the database, eliminating slow, manual data transformations in application code. Establishing early monitoring for latency, replication lag, memory usage, and cluster failover resilience ensures stability before adopting advanced tools like Atlas Search and Vector Search.
Widespread enterprise SaaS adoption has introduced notable security blind spots due to inconsistent controls, visibility, and configurations across disparate vendor applications. While traditional compliance frameworks such as SOC 2 and ISO 27001 evaluate provider organizational security, they typically fail to address customer-facing application capabilities. To resolve this discrepancy, MongoDB collaborated with the Cloud Security Alliance and GuidePoint Security to develop the SaaS Security Capability Framework. This framework establishes actionable technical controls across six critical operational domains, including identity and access management, machine-readable logging, configuration management, and incident notifications. By standardizing customer-side capabilities within the shared responsibility model, the framework simplifies vendor risk assessments and provides engineering teams with structured implementation benchmarks.
Stripe Workflows can automate additional compliance safeguards by flagging transactions that need review before order processing. The described pattern starts when a Payment intent succeeded event occurs, retrieves the customer, and checks whether the customer’s custom_status metadata is not “Approved” while the payment amount exceeds $5,000. When these conditions are met, the workflow emails one or more designated team members with instructions for assessing the order. The examples motivating review include unusually high order values, unfamiliar sales regions, and behavioral patterns commonly associated with fraud, such as a new customer order above a monetary threshold. Workflows uses a visual building interface in the Stripe dashboard to sequence actions across multiple Stripe products, helping enforce business rules, reduce manual effort, and support risk management.
Product data reconciliation across internal databases, ERPs, and platforms such as Stripe becomes harder to maintain as catalogs and system relationships grow, affecting financial reporting, customer experience, and operations. The post compares real-time reconciliation, using tools such as Kafka, Flink, Delta Live Tables, and Debezium for more immediate consistency, with batch approaches built around Spark, AWS Glue, Snowflake, and dbt for periodic, large-scale checks at lower cost and complexity. It also presents validation options including Great Expectations, Apache NiFi, Talend, and Airbyte, and recommends defining data ownership, applying field-specific conflict rules, and routing critical mismatches to manual review. Soft deletes and tombstones preserve deletion history, while observability through synchronization latency, conflict rates, resolution patterns, and regular audits helps assess reconciliation health as product catalogs scale.
Maintaining consistent product data between an internal catalog and Stripe becomes difficult when outages, throttling, high volumes, and changes from both systems create reconciliation failures. The post compares a naive one-way flow, in which a Lambda calls the Stripe API with no persistent synchronization record, with a queue-based design that durably stores changes, controls processing rate, retries failures with exponential backoff, and sends persistent failures to dead-letter queues. For bidirectional synchronization, it contrasts periodic ETL, using extraction, transformation, and loading, with an event-driven model based on Stripe Event Destinations, EventBridge, SQS, and symmetrical events from internal systems. The conclusion is that durable, observable queues and event-driven processing provide buffering, auditability, near-real-time propagation, independent change processing, and more graceful scaling for distributed reconciliation.
Stripe Workflows can automate responses to early fraud warnings so potentially fraudulent charges receive refunds when disputing them would be more costly. The workflow starts when an early fraud warning is created in Radar, retrieves the related charge using its Charge ID, and checks whether the amount is less than 15 USD. If the condition is met, it creates a refund; if it is not met, an optional branch can email a team member for manual review. Workflows uses a visual builder in the Stripe dashboard, where triggers, actions, and conditions run sequentially across multiple Stripe products, while Stripe Radar provides real-time fraud protection and Radar for Fraud Teams adds customization and deeper insights.
Stripe webhook handlers that treat events as signals and fetch the latest resource can preserve correctness when payloads are stale, partial, duplicated, or out of order, but bursts of events can drive excessive API traffic. The post describes Stripe’s general limit as 100 read requests per second and notes that exceeding it produces 429 responses. Its solution places Hookdeck Event Gateway between Stripe and the application: Hookdeck authenticates and queues incoming webhooks, throttles delivery, and lets the handler retrieve the current Stripe resource at a controlled pace. The Express.js flow verifies the Hookdeck signature, checks the event and resource ID, fetches the invoice with the Stripe SDK, and supports queue monitoring, alerts, and retries for backpressure.
Stripe introduces Stripe Workflows, a Dashboard-based system for orchestrating Stripe products and APIs through responsive, end-to-end automation. It addresses the complexity of coordinating multiple Stripe products by replacing custom event-handling and dependency-management code with visual workflows that support real-time triggers, branching logic, dynamic fields, and more than 600 event triggers. Workflows includes execution monitoring with event-payload and step-level tracing, 30-day default log retention, exponential-backoff retries, built-in idempotency, and recursion protection capped at a depth of five. The post describes applications including reducing chargeback risk, routing high-value transactions for compliance review, and creating custom-priced subscriptions from finalized quotes without additional infrastructure.
A 2023 summer internship on Ramp’s growth engineering team changed the author’s view of software engineering and led to a return as a new graduate engineer. The team accelerated outbound sales through fast experiments, metric ownership, and decisions about whether promising tactics deserved scalable systems, rather than following the thorough specification process typical of regular projects. His work included reducing third-party data-enrichment costs, building AI-driven outbound-sales tools, adding observability and service monitors, and conducting competitive intelligence projects. He returned to join the new procurement product line, where he helped match card transactions to purchase orders and led additional projects. The post also recommends rapidly building system context through code review and on-call shadowing, while full-time engineers should weigh velocity against maintainability and take initiative on less-defined problems.
During a fall internship at Ramp, a backend software engineering intern worked on the reimbursements team and describes projects that improved reimbursement flows while providing full-stack experience. One early change clarified multi-currency mileage reimbursements by showing both the expense currency and the payout currency, while later work overhauled mileage support with live navigation storage, frequent-trip recommendations, commute deductions, and automatic e-receipts. The post also discusses using LLMs to predict merchant categories from receipt text, enhancing merchant matching, and adding monitoring and alerts for OCR categorization errors. The reflection emphasizes measuring feature usage and outcomes, writing and revising specifications before implementation, leaving reusable code, and using Ramp’s transparent documentation and collaborative support. The internship concluded with a return to the University of Waterloo planned for 2025.
Pay-as-you-go services need a way to warn customers when consumption reaches a configured limit, but building usage aggregation, monitoring, and notification workflows adds operational complexity. This tutorial uses Stripe Billing Alerts to define a customer-specific usage threshold with a customer_id, meter_id, gte value, and recurrence, then sends billing.alert.triggered events through Stripe Event Destinations to an Amazon EventBridge partner event source. It tests the flow with Stripe Meter Events and shows AWS CLI rules routing the event to Amazon SNS, as well as a Lambda example that retrieves the customer's email from Stripe and sends mail with Amazon SES. The result is a minimal-code notification path using AWS services, with sandbox setup required because some APIs and features may not be available in conventional test mode.
The article presents a production-grade approach to securing Stripe API keys in AWS, motivated by the risks and operational limits of basic secret storage for payment processing. It compares AWS Parameter Store with Secrets Manager, emphasizing built-in rotation, CloudTrail audit trails, IAM integration, larger secrets, and cross-account access despite higher cost. The implementation uses environment-specific secret paths, tags, and IAM conditions to isolate development, staging, and production credentials, with separate rotation schedules. For zero-downtime rotation, applications refresh cached credentials after authentication failures while both old and new keys remain valid during a transition. The design also covers CloudWatch monitoring, emergency rotation procedures, cross-region replication, region-specific schedules, and cost considerations, concluding that ongoing review is necessary for secure and reliable payment operations.
The guide presents ten practices for building production-ready Stripe integrations, covering payment forms, subscriptions, and operational concerns. It recommends verifying webhook signatures with the Stripe-Signature header and endpoint secret, recording event IDs for idempotency, and returning a fast 2xx response while processing asynchronously. Test and live API keys, webhook endpoints, logs, and monitoring should remain separate, while error handling should distinguish card, request, and API failures and use exponential backoff for retries. The guide also recommends testing successful, declined, insufficient-funds, and 3DS scenarios with Stripe test cards, alongside simulated webhook events. Its conclusion emphasizes edge-case planning, logging, monitoring, and ongoing review as foundations for reliable, secure payment processing that can scale with business needs.
The best engineering reads, once a week.
