Fly Behind The Scenes: Fresh Logging

Summary

Fly replaced its centralized Graylog logging platform after seven servers could not process peak volumes of 20,000–30,000 logs per second, causing drops. Vector, written in Rust, now runs on every server, parses and transforms logs from journald and other applications, and sends them directly to Elasticsearch, removing Graylog as a middleman. The pipeline adopts the Elasticsearch Common Schema (ECS), with Fly-specific fields for app names, allocation IDs, and regions, while Vector configurations are distributed through the existing configuration management system. The migration took about two weeks alongside other work; logs are now available quickly enough to support failed-allocation queries that previously often raced processing, and future sinks may include Papertrail, Honeycomb, or Kafka.

Takeaways

1. 1

   Vector distributes log capture and transformation across servers, while Elasticsearch handles indexing and retrieval after Graylog was removed from the path.

2. 2

   The Elasticsearch Common Schema standardizes fields across applications, including source IP data, and supports Fly-specific fields such as app names, allocation IDs, and regions.

3. 3

   Vector configurations can be validated and unit tested locally before deployment; configuration changes are then distributed to all servers through the configuration management system.