Docker lazy loading at Grab: Accelerating container startup times

Summary

Large Docker images for Airflow and Spark Connect caused minute-scale downloads, delaying cold starts and weakening autoscaling responsiveness. The post describes testing eStargz and Seekable OCI (SOCI) with containerd remote snapshotters, which expose registry-backed layers through a virtual lowerdir and fetch file content on demand. In production, SOCI delivered 30–40% faster startup times for Airflow and Spark Connect, while benchmark results showed SOCI matching standard images’ 5.0-second Airflow application startup; eStargz took 25.0 seconds. Configuration tuning increased download concurrency, unpack concurrency, and chunk size, reducing fresh-node image download time from 60 seconds to 24, a 60% improvement. Grab chose SOCI because Bottlerocket supports it natively, while noting that lazy loading redistributes rather than removes download work and requires workload-specific tuning.

Takeaways

1. 1

   SOCI matched standard images’ 5.0-second Airflow application startup in benchmarking, while eStargz took 25.0 seconds.

2. 2

   Increasing SOCI’s max_concurrent_downloads_per_image from 5 to 10, max_concurrent_unpacks_per_image from 3 to 10, and chunk size from 8MB to 16MB reduced fresh-node image download time from 60 seconds to 24 seconds.

3. 3

   Production SOCI deployment produced 30–40% faster startup times for Airflow and Spark Connect; lazy loading shifts image downloads into runtime rather than eliminating them.