# How We Built a Logging Stack at Grab

[Grab](https://yomu.fyi/company/grab) · Daniel Kasen · Jul 31, 2019

**Type:** Problem & solution

## Summary

Grab needed a scalable logging platform to replace slow, fragmented systems that hindered debugging across their growing service fleet. Generating 25TB of daily logs, the team built a horizontally scalable Elasticsearch cluster configured via Ansible and monitored with Datadog. Although the initial proof of concept assigned all node roles (ingest, coordinator, master, and data) to every machine, operating at scale introduced major challenges with JVM heap exhaustion and cluster stability. The team resolved memory pressure and performance bottlenecks by tuning circuit breakers, lowering field data cache limits, adjusting shard allocations based on segment memory, and disabling translog compression during shard transfers.

## Context

Grab lacked centralized log visualisations and fast search metrics, relying on slow cloud storage retrievals, timing-out SaaS tools, or manual SRE log fetching while handling 25TB of daily logging data.

## Approach / What changed

The engineering team deployed an Elasticsearch cluster using Ansible for provisioning, adopted ingest nodes for preprocessing, established Datadog dashboards for JVM and cluster monitoring, and tuned Elasticsearch memory and compression configurations.

## Takeaways

- Sorting by large fields like document IDs can load tens of gigabytes into the field data cache, causing JVM out-of-memory crashes unless field cache limits and circuit breakers are strictly tuned.
- Disabling translog compression reduced the transfer time of a 50GB shard from one hour to 20 minutes because underlying Lucene segments are already compressed.
- Combining master, data, ingest, and coordinator roles on single nodes in large clusters creates stability and troubleshooting issues, such as searches exhausting field data cache and crashing master nodes.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Elasticsearch](https://yomu.fyi/topic/elasticsearch), [Monitoring](https://yomu.fyi/topic/monitoring), [Performance](https://yomu.fyi/topic/performance), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://engineering.grab.com/how-built-logging-stack)