How to Secure the Celery Task Queue with Fernet

Summary

Celery jobs sent through a Redis broker can expose their contents because Redis does not currently support SSL and Heroku add-ons communicate over the public web. The proposed solution encrypts jobs before they travel between dynos and Redis by wrapping Kombu’s default serializers with Fernet symmetric encryption from Cryptography. The open-source kombu-fernet-serializers library supplies these encrypted serializers through Setuptools entry points; setup requires adding the package, storing a Fernet key in KOMBU_FERNET_KEY, and configuring Celery to use fernet_json with MIMETYPE in CELERY_ACCEPT_CONTENT. The example creates a Celery app with REDIS_URL as its broker and uses the serializer settings, retaining Redis’s straightforward configuration while addressing sensitive information in transit.

Takeaways

1. 1

   Redis does not currently support SSL, so Celery job contents sent through Heroku add-ons can travel unencrypted between dynos and Redis.

2. 2

   Kombu custom serializers control how Celery jobs become byte strings for the broker and how they are converted back, enabling Fernet encryption around the default serializers.

3. 3

   The example configures CELERY_TASK_SERIALIZER='fernet_json' and includes MIMETYPE in CELERY_ACCEPT_CONTENT while using REDIS_URL as the broker.