# Structured Logging: The Best Friend You’ll Want When Things Go Wrong

[Grab](https://yomu.fyi/company/grab) · Aditya Praharaj · Mar 5, 2019

**Type:** Problem & solution

## Summary

Grab redesigned its backend logging approach to address mounting vendor costs, query language limitations, and debugging difficulties across a growing microservices ecosystem. Most services previously emitted syslog-style key-value logs almost entirely at the INFO level, which made volume reduction difficult and lacked causal ordering and automated correlation. The engineering team migrated to a self-managed Elastic stack backend and built a structured logging library in Go from the ground up. This framework introduces dynamic log-level adjustment at runtime, automatic trace-based log correlation via Grab-Kit, and the Common Grab Log Schema to enforce consistent JSON formatting without Elasticsearch indexing conflicts.

## Context

Rapid growth of backend services at Grab led to high logging vendor costs, limited query capabilities, pervasive use of only the INFO log level, and issues with causal ordering and cross-service log correlation.

## Approach / What changed

Grab built a custom Go logging library integrated with a self-managed Elastic stack, featuring dynamic runtime log levels, automated trace ID correlation, and the Common Grab Log Schema (CGLS) for deterministic JSON field typing.

## Takeaways

- Recording log write times instead of generation times jumbles the order of events emitted within microseconds, breaking causal ordering.
- Emitting conflicting data types under the same JSON field name triggers indexing failures in Elasticsearch mappings.
- Dynamic log levels controlled via configuration management allow systems to run at WARN or INFO in production and switch to DEBUG during incidents without redeploying.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Elasticsearch](https://yomu.fyi/topic/elasticsearch), [Go](https://yomu.fyi/topic/go), [Microservices](https://yomu.fyi/topic/microservices), [Observability](https://yomu.fyi/topic/observability)

[Read original post](https://engineering.grab.com/structured-logging)