Profile-guided optimisation (PGO) on Grab services

Summary

Profile-guided optimisation (PGO) improves Go application performance by feeding production CPU profile data into subsequent compiler builds. Teams evaluated PGO across multiple Grab services, including TalariaDB and an orchestrator within the Catwalk service, using Go 1.20 and Go 1.21 builds. TalariaDB saw substantial resource savings after compiling Docker images with a 360-second pprof profile, resulting in at least a 10% drop in CPU usage, a 30% drop in memory consumption, and a 38% reduction in persistent volume usage for event ingestion queues. Conversely, testing on Catwalk showed only a 5% performance gain after adjusting the profiling duration from 59 seconds to 6 minutes, an outcome deemed insufficient to justify pipeline modifications. Broader adoption across monorepo services remains constrained until the central build process supports PGO arguments and automated profile retrieval.

Takeaways

1. 1

   Compiling TalariaDB with a 360-second PGO profile reduced CPU usage by at least 10%, decreased memory usage by 30%, and increased the ingested event count per CPU unit from 1.1 million to 1.7 million.

2. 2

   Profiling duration directly impacts optimisation quality; a 59-second profile increased CPU usage on the Catwalk service, whereas a 6-minute profile reduced CPU usage.

3. 3

   Services hosted in monorepos require build systems and CI pipelines that natively support passing and attaching pprof files before PGO can be broadly adopted.