How We Built a Logging Stack at Grab

Summary

Grab needed a scalable logging platform to replace slow, fragmented systems that hindered debugging across their growing service fleet. Generating 25TB of daily logs, the team built a horizontally scalable Elasticsearch cluster configured via Ansible and monitored with Datadog. Although the initial proof of concept assigned all node roles (ingest, coordinator, master, and data) to every machine, operating at scale introduced major challenges with JVM heap exhaustion and cluster stability. The team resolved memory pressure and performance bottlenecks by tuning circuit breakers, lowering field data cache limits, adjusting shard allocations based on segment memory, and disabling translog compression during shard transfers.

Takeaways

1. 1

   Sorting by large fields like document IDs can load tens of gigabytes into the field data cache, causing JVM out-of-memory crashes unless field cache limits and circuit breakers are strictly tuned.

2. 2

   Disabling translog compression reduced the transfer time of a 50GB shard from one hour to 20 minutes because underlying Lucene segments are already compressed.

3. 3

   Combining master, data, ingest, and coordinator roles on single nodes in large clusters creates stability and troubleshooting issues, such as searches exhausting field data cache and crashing master nodes.