Python Async Workers on Fly Machines

Summary

A Python web app needed to build and install libraries without blocking HTTP requests or risking backend failures when resource-heavy builds overlapped. The initial backend implementation sometimes ran for more than 30 minutes and could kill the backend, while always-on Celery workers left capacity idle. The proposed proof of concept stores job parameters in Redis, starts a Fly Machine through the Machines API with Redis key environment variables, executes a Python function, and writes the result back. A status endpoint polls the Machine state and returns PENDING until completion, after which cleanup deletes the Machine and Redis entries. The design pays for worker resources only during jobs, but the example requires JSON-serializable arguments and results and is presented as a simple implementation rather than a full job system.

Takeaways

1. 1

   Fly Machines are started only when a background job is requested, avoiding the cost of keeping dedicated workers available during idle periods.

2. 2

   The worker imports a module and function named in Redis, runs the supplied args and kwargs, and stores a JSON result with a SUCCESS status.

3. 3

   The proof of concept limits job arguments and results to JSON-serializable values and uses a separate Docker image for the worker.