# Profile-guided optimisation (PGO) on Grab services

[Grab](https://yomu.fyi/company/grab) · Yonghao Hu · Jun 5, 2024

**Type:** Benchmark

## Summary

Profile-guided optimisation (PGO) improves Go application performance by feeding production CPU profile data into subsequent compiler builds. Teams evaluated PGO across multiple Grab services, including TalariaDB and an orchestrator within the Catwalk service, using Go 1.20 and Go 1.21 builds. TalariaDB saw substantial resource savings after compiling Docker images with a 360-second pprof profile, resulting in at least a 10% drop in CPU usage, a 30% drop in memory consumption, and a 38% reduction in persistent volume usage for event ingestion queues. Conversely, testing on Catwalk showed only a 5% performance gain after adjusting the profiling duration from 59 seconds to 6 minutes, an outcome deemed insufficient to justify pipeline modifications. Broader adoption across monorepo services remains constrained until the central build process supports PGO arguments and automated profile retrieval.

## Context

Evaluating the feasibility, performance gains, and operational overhead of adopting Go's profile-guided optimisation across production services at Grab.

## Approach / What changed

Capturing 360-second pprof CPU profiles from running services and passing them into compiler builds via the \`-PGO\` build flag inside Dockerfiles and CI pipelines.

## Takeaways

- Compiling TalariaDB with a 360-second PGO profile reduced CPU usage by at least 10%, decreased memory usage by 30%, and increased the ingested event count per CPU unit from 1.1 million to 1.7 million.
- Profiling duration directly impacts optimisation quality; a 59-second profile increased CPU usage on the Catwalk service, whereas a 6-minute profile reduced CPU usage.
- Services hosted in monorepos require build systems and CI pipelines that natively support passing and attaching pprof files before PGO can be broadly adopted.

**Tags:** [CI/CD](https://yomu.fyi/topic/ci-cd), [Docker](https://yomu.fyi/topic/docker), [Go](https://yomu.fyi/topic/go), [Performance](https://yomu.fyi/topic/performance)

[Read original post](https://engineering.grab.com/profile-guided-optimisation)