Fly's Prometheus Metrics

Summary

Fly describes the Prometheus-style metrics infrastructure behind its globally distributed micro-VM platform and explains why it favors metrics over traditional monitoring checks. The system collects counters and other measurements through HTTP /metrics endpoints, stores them in a clustered VictoriaMetrics (“Vicky”) deployment, and uses Telegraf, Vector, exporters, and Consul to connect sources and manage discovery. Within Firecracker instances, custom init, the Nomad driver, and a vsock carry node-exporter-type JSON statistics from guests for collection, while applications can expose their own exporters. Deployments can declare an exporter port and /metrics path in fly.toml, after which Fly routes those application metrics into Vicky alongside platform data. The resulting Prometheus API is available for Grafana queries, dashboards, and alerts, with labels such as app, region, host, and instance; the post argues that the same historical data supports anomaly detection, trending, and planning.

Takeaways

1. 1

   VictoriaMetrics replaced the initial simple Prometheus setup after scaling it with Thanos created too much operational hassle, according to the excerpt.

2. 2

   A Fly application can expose a Prometheus exporter by declaring its port and /metrics path in fly.toml; deployment configuration then sends those metrics to the Vicky cluster.

3. 3

   Metrics-based monitoring uses historical time-series data to define anomaly rules later, unlike checks that require scripts for predefined probes.