Multiple Logs for Resiliency

Summary

An unexpected network outage can make an application misbehave while also removing crucial log data from the normal Fly.io pipeline. The post traces that pipeline from application STDOUT through Fly.io, NATS, the Log Shipper, Vector, and a third-party provider, then presents two ways to add redundancy. One strategy broadcasts framework logs to a separate file on each machine, preferably on a persistent volume with rotation; a Rails example uses ActiveSupport::Logger.broadcast. The other runs a separate app that subscribes directly to NATS, parses messages, and writes selected fields to disk, with a Fly authentication token and provisions for reconnection and error handling. File copies are more failsafe but raw and harder to fetch, while a custom shipper can produce more usable data but is less bulletproof.

Takeaways

1. 1

   Per-machine log files provide the most failsafe redundancy, but they are raw and must be fetched from each machine; persistent volumes and rotation help preserve them without filling available storage.

2. 2

   A Rails implementation can use ActiveSupport::Logger.broadcast to send entries to STDOUT and a separate production log file.

3. 3

   A custom shipper can subscribe to the logs.> NATS subject, parse JSON messages, and write timestamp, instance, region, level, and message fields to disk, but it needs safeguards for write failures and lost NATS connections.