Sitar-agent: Building a reliable dynamic configuration sidecar at scale

Summary

Airbnb modernized sitar-agent, a Kubernetes sidecar that delivers dynamic configuration updates to thousands of polyglot service instances without requiring redeployments. The architecture uses periodic AWS S3 snapshots to enable fast, decoupled pod startups and maintains local file-based storage for the main container to read configs with in-memory caching. Key design decisions include retaining an isolated sidecar over an in-process library and optimizing pull-based polling with server-side caching.

Takeaways

1. 1

   Preloading hourly configuration snapshots from Amazon S3 on pod startup prevents thundering herd load spikes and allows services to bootstrap even if the configuration service is down.

2. 2

   Retaining a sidecar container instead of an in-process library preserves resource isolation and eliminates the need to rewrite agent synchronization logic across multiple programming languages.

3. 3

   Pull-based polling load was minimized using a short 10-second server cache and resume tokens that skip redundant database scans on cache misses.