# Building a Hyper Self-Service, Distributed Tracing and Feedback System for Rule & Machine Learning (ML) Predictions

[Grab](https://yomu.fyi/company/grab) · Warren Zhou · May 24, 2021

**Type:** Problem & solution

## Summary

Grab's Trust, Identity, Safety, and Security team processes billions of daily rule and machine learning decisions for fraud detection, safety, and identity checks. Earlier logging approaches using plain text Kibana logs and the ActionTrace library lacked structured formats, dynamic entity customization, and fine-grained access controls. To resolve these limitations, the team built Archivist, a centralized tracing, statistics, and feedback system. Archivist ingests events through an SDK into Kafka streams, buffers and routes data into Elasticsearch indices and Amazon S3, and provides a role-based user portal. The platform handles 80 million daily logs across roughly 50 business scenarios, reducing scenario onboarding times from days to minutes.

## Context

Grab's security and safety services make billions of predictions daily, but unstructured Kibana logs and rigid ActionTrace schemas caused high maintenance overhead, lacked access control, and slowed scenario onboarding for operations and data science teams.

## Approach / What changed

Grab built Archivist, an architecture using an SDK to push events to Kafka streams, which an event processor consumes to write into shared Elasticsearch clusters while simultaneously archiving data into Amazon S3 for Presto queries, complemented by a self-service web portal with role-based access control.

## Takeaways

- Archivist replaces rigid schemas with a flexible model allowing users to define up to five custom entity types and six custom data fields per event type.
- Decoupling teams into separate Kafka streams and Elasticsearch daily file indices simplified data management and cost allocation across multiple tenants.
- Unbounded goroutine creation per Elasticsearch document write caused memory exhaustion at 400,000 goroutines, necessitating a hard concurrency limit.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Elasticsearch](https://yomu.fyi/topic/elasticsearch), [Kafka](https://yomu.fyi/topic/kafka), [Observability](https://yomu.fyi/topic/observability)

[Read original post](https://engineering.grab.com/building-hyper-self-service-distributed-tracing-feedback-system)