Building a Hyper Self-Service, Distributed Tracing and Feedback System for Rule & Machine Learning (ML) Predictions

Summary

Grab's Trust, Identity, Safety, and Security team processes billions of daily rule and machine learning decisions for fraud detection, safety, and identity checks. Earlier logging approaches using plain text Kibana logs and the ActionTrace library lacked structured formats, dynamic entity customization, and fine-grained access controls. To resolve these limitations, the team built Archivist, a centralized tracing, statistics, and feedback system. Archivist ingests events through an SDK into Kafka streams, buffers and routes data into Elasticsearch indices and Amazon S3, and provides a role-based user portal. The platform handles 80 million daily logs across roughly 50 business scenarios, reducing scenario onboarding times from days to minutes.

Takeaways

1. 1

   Archivist replaces rigid schemas with a flexible model allowing users to define up to five custom entity types and six custom data fields per event type.

2. 2

   Decoupling teams into separate Kafka streams and Elasticsearch daily file indices simplified data management and cost allocation across multiple tenants.

3. 3

   Unbounded goroutine creation per Elasticsearch document write caused memory exhaustion at 400,000 goroutines, necessitating a hard concurrency limit.