Structured Logging: The Best Friend You’ll Want When Things Go Wrong

Summary

Grab redesigned its backend logging approach to address mounting vendor costs, query language limitations, and debugging difficulties across a growing microservices ecosystem. Most services previously emitted syslog-style key-value logs almost entirely at the INFO level, which made volume reduction difficult and lacked causal ordering and automated correlation. The engineering team migrated to a self-managed Elastic stack backend and built a structured logging library in Go from the ground up. This framework introduces dynamic log-level adjustment at runtime, automatic trace-based log correlation via Grab-Kit, and the Common Grab Log Schema to enforce consistent JSON formatting without Elasticsearch indexing conflicts.

Takeaways

1. 1

   Recording log write times instead of generation times jumbles the order of events emitted within microseconds, breaking causal ordering.

2. 2

   Emitting conflicting data types under the same JSON field name triggers indexing failures in Elasticsearch mappings.

3. 3

   Dynamic log levels controlled via configuration management allow systems to run at WARN or INFO in production and switch to DEBUG during incidents without redeploying.