From Custom to Open: Scalable Network Probing and HTTP/3 Readiness with Prometheus

Summary

Slack encountered a lack of client-side observability when introducing HTTP/3 at its edge. Because HTTP/3 runs over QUIC on UDP instead of TCP, neither existing commercial SaaS tools nor the internal Prometheus Blackbox Exporter could probe the new endpoints. To address this gap, an engineering intern integrated the open-source quic-go library into Prometheus Blackbox Exporter to build a native, configurable HTTP/3 probe. Slack deployed an internal implementation using the upstream functionality while waiting for pull request reviews, ultimately landing the contribution upstream. The resulting setup provides a unified view of HTTP/1.1, HTTP/2, and HTTP/3 metrics within Grafana, enabling reliable alerting and faster correlation with other telemetry.

Context

Slack began rolling out HTTP/3 at the edge but faced an observability gap: HTTP/3 relies on QUIC over UDP rather than TCP, rendering existing SaaS tools and the Prometheus Blackbox Exporter incapable of probing HTTP/3 endpoints to track regressions or round-trip measurements.

Approach / What changed

Slack intern Sebastian Feliciano integrated the quic-go library into the Prometheus Blackbox Exporter to construct an HTTP/3-capable client adhering to existing configuration patterns. While the contribution was upstreamed to the open-source repository, Slack deployed an in-house system using the new capabilities to probe HTTP/3 endpoints immediately.

Takeaways

• Prometheus Blackbox Exporter lacked native QUIC support, preventing network probing of HTTP/3 endpoints over UDP.
• Slack selected quic-go to construct a configurable HTTP/3 transport within the Prometheus Blackbox Exporter architecture.
• Integrating HTTP/3 probes allowed Slack to combine HTTP/1.1, HTTP/2, and HTTP/3 metrics into a single Grafana dashboard for improved alerting and debugging.

Related reading

• Slack · May 5, 2026

  From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines

  Slack's data platform accumulated over 700 SSH-based Airflow operators executing jobs directly on AWS Elastic MapReduce clusters, creating broad security surfaces and operational instability. Stateful SSH connections frequently failed when Kubernetes pods restarted, leaving orphaned zombie processes and preventing infrastructure modernization such as migrating EMR clusters to child accounts. To eliminate SSH access, Slack adopted a REST-based architecture using Quarry, a gateway providing unified REST APIs and token authentication across compute engines. For non-Hadoop CLI workloads, Slack leveraged YARN Distributed Shell to execute arbitrary shell scripts stored in Amazon S3 within managed, isolated YARN containers. The phased migration achieved zero downtime across eight data regions, unlocking reliable monitoring, simplified onboarding, and future initiatives like Spark on Kubernetes.

  Mahendran Vasagam

• Grab · Apr 23, 2024

  Ensuring data reliability and observability in risk systems

  Grab's in-house risk management platform, GrabDefence, relies on ingesting large volumes of upstream data to drive real-time heuristic rules and data science models. To prevent data discrepancies and missing information from degrading fraud detection, the team implemented a real-time data observability pipeline. The solution uses Apache Flink SQL alongside custom table functions like JSONEXPLOAD to flatten nested JSON structures and aggregate metrics over 5-minute tumbling windows. Processed metrics are sent to Datadog, where counters are grouped by source stream and evaluated using anomaly detection algorithms. Alerts are routed directly to Slack, reducing anomaly detection latency from days or weeks down to within the same day or hour.

  Yi Ni Ong

• Grab · Jul 7, 2021

  Reshaping Chat Support for Our Users

  Grab transitioned from voice hotlines and third-party tools to an in-house native chat support system integrated into their CRM. The team validated the platform through an MVP and user shadowing to address session disconnections, agent context switching, and routing bottlenecks. To optimize support operations at scale, they introduced dynamic queue limits based on Little's law, machine learning autocomplete suggestions for agents, and duration timers with visual nudges. These enhancements reduced chat waiting times by 30%, unresponsive users by 7%, and overall chat handling duration by 22%.

  Elisa Monacchi

• Slack · Dec 1, 2025

  Streamlining Security Investigations with Agents

  Slack's Security Engineering team handles billions of daily security events and needed a reliable way to streamline on-call alert triage. An initial prototype relying on a single 300-word prompt produced inconsistent results and frequently reached spurious conclusions without properly challenging assumptions. To gain precise control, the team decomposed the workflow into chained model invocations with structured JSON outputs organized across three agent personas: a Director, four domain experts, and a Critic. Domain experts gather raw evidence through tool calls, the Critic evaluates finding quality and synthesizes a timeline, and the Director steers investigation phases using tiered model costs. The multi-agent system enables engineers to supervise investigations via a real-time dashboard while uncovering emergent issues like credential exposures across process ancestry chains.

  Dominic Marks