How We Built a Logging Stack at Grab

Summary

Grab needed a scalable logging platform to replace slow, fragmented systems that hindered debugging across their growing service fleet. Generating 25TB of daily logs, the team built a horizontally scalable Elasticsearch cluster configured via Ansible and monitored with Datadog. Although the initial proof of concept assigned all node roles (ingest, coordinator, master, and data) to every machine, operating at scale introduced major challenges with JVM heap exhaustion and cluster stability. The team resolved memory pressure and performance bottlenecks by tuning circuit breakers, lowering field data cache limits, adjusting shard allocations based on segment memory, and disabling translog compression during shard transfers.

Context

Grab lacked centralized log visualisations and fast search metrics, relying on slow cloud storage retrievals, timing-out SaaS tools, or manual SRE log fetching while handling 25TB of daily logging data.

Approach / What changed

The engineering team deployed an Elasticsearch cluster using Ansible for provisioning, adopted ingest nodes for preprocessing, established Datadog dashboards for JVM and cluster monitoring, and tuned Elasticsearch memory and compression configurations.

Takeaways

• Sorting by large fields like document IDs can load tens of gigabytes into the field data cache, causing JVM out-of-memory crashes unless field cache limits and circuit breakers are strictly tuned.
• Disabling translog compression reduced the transfer time of a 50GB shard from one hour to 20 minutes because underlying Lucene segments are already compressed.
• Combining master, data, ingest, and coordinator roles on single nodes in large clusters creates stability and troubleshooting issues, such as searches exhausting field data cache and crashing master nodes.

Related reading

• Grab · Mar 5, 2019

  Structured Logging: The Best Friend You’ll Want When Things Go Wrong

  Grab redesigned its backend logging approach to address mounting vendor costs, query language limitations, and debugging difficulties across a growing microservices ecosystem. Most services previously emitted syslog-style key-value logs almost entirely at the INFO level, which made volume reduction difficult and lacked causal ordering and automated correlation. The engineering team migrated to a self-managed Elastic stack backend and built a structured logging library in Go from the ground up. This framework introduces dynamic log-level adjustment at runtime, automatic trace-based log correlation via Grab-Kit, and the Common Grab Log Schema to enforce consistent JSON formatting without Elasticsearch indexing conflicts.

  Aditya Praharaj

• Grab · Mar 3, 2019

  How We Simplified Our Data Ingestion & Transformation Process

  Grab evolved its real-time data ingestion pipeline after an initial architecture built on Spark Streaming and Python encountered operational complexity, node failures, and data loss from S3 eventual consistency. Because the streaming workload primarily handled event partitioning and ORC file generation, the team consolidated these tasks directly into an existing Golang processing service. They implemented sharded concurrent maps for high-throughput partitioning and optimized heap allocations to resolve memory bottlenecks. This refactor removed intermediate Avro conversions and intermediate storage hops. The simplified Go pipeline eliminated data loss and reduced processing lag from up to 13 minutes down to approximately 1 minute.

  Yichao Wang

• Grab · Sep 23, 2020

  Our Journey to Continuous Delivery at Grab (Part 1)

  Around the end of 2018, Grab's backend architecture consisted of roughly 270 services managed through fragmented, manual deployment workflows. Engineers copied release parameters between build logs, wiki pages, Slack bots, and multiple Jenkins jobs, leading to high operational friction and an average of 10 business days between production updates for a service. To streamline delivery, Grab built Conveyor, an internal automation platform built on top of open-source Spinnaker. Conveyor introduced a custom user interface focused on pipeline visibility and a pipeline-as-code DSL called Artificer using Jsonnet files in the monorepository. The platform automatically registers build artifacts with commit metadata to eliminate manual parameter entry and automatically provisions integration, staging, and production pipelines.

  Sylvain Bougerel

• Grab · Feb 4, 2016

  Grab You Some Post-Mortem Reports

  Grab uses a Service-Oriented Architecture to deploy features quickly, but unfamiliarity across teams makes cross-service production debugging difficult. Historical incident reports lacked context, diagnostic details, impact data, and timelines, leaving outside engineers unable to learn from past outages. To address this, Grab established a four-pillar framework for post-mortem reports covering chronology, context, empowerment, and solutions. The approach mandates blameless, educational write-ups that categorize post-incident improvements across people, product, and process dimensions. Final reports undergo peer reviews by engineers from external teams to ensure clarity and remove bias.

  Lian Yuanlin