Loading post…
EbayGautam Singh, Ravi Agarwal, Akshay KamathProblem & solution
Inside Minerva: building scalable observability for eBay’s monetization platform
Summary
eBay’s Monetization Team lacked a dedicated observability platform, causing fee behavior issues to often be discovered through customer complaints or post-transaction analysis. Data warehouse data arrived with delays, and engineers and business teams had to manually query multiple sources. Minerva solves this with an asynchronous observability pipeline: the fee-processing service places a small record on a local queue, a background producer publishes it to an event stream, and a consumer enriches the data before storing it in ClickHouse. ClickHouse’s columnar storage and MergeTree engine allow fast aggregation queries on records with nearly 250 columns; a proof of concept on 1.1 billion rows returned queries in 0.11 to 0.72 seconds. The design keeps observability off the critical fee path, prioritizing fee-processing availability while providing configurable, context-aware monitoring.
Context
Prior to Minerva, eBay’s Monetization Team lacked a dedicated observability platform. Fee behavior issues were often discovered through customer complaints or post-transaction analysis. Data warehouse data arrived too late for hour-by-hour monitoring, and investigations required manual queries across multiple data sources.
Approach / What changed
Minerva uses an asynchronous pipeline: the fee-processing service writes a small record (identifiers) to a local bounded queue with a ring-buffer policy. A background producer publishes records to an event stream. A consumer enriches them by querying transactional databases and stores the enriched data in ClickHouse. Dashboards and alerting modules (query execution, use-case processing, notification delivery) evaluate the data without affecting fee processing.
Takeaways
Minerva decouples observability from the fee-processing path using a local bounded queue with a ring-buffer policy, prioritizing fee-processing availability over lossless monitoring delivery.
ClickHouse’s columnar storage and MergeTree engine enable sub-second aggregations on billions of rows with nearly 250 columns, as shown in a proof of concept with 1.1 billion rows (0.11-0.72 seconds per query).
The monitoring pipeline has three modules (query execution, use-case processing, notification delivery), allowing new use cases to be added without changing the fee-processing service.
FAQ
Why does Minerva use a local queue instead of publishing directly?
Publishing directly would couple monitoring availability to fee-processing availability. A slow network call or unavailable broker could increase request latency. Minerva's local queue allows the request to avoid waiting for network publication under normal conditions, and the queue is bounded to contain backpressure risk.
How does Minerva ensure query performance on large datasets?
Minerva uses ClickHouse, a columnar database, and its MergeTree engine. A proof of concept with 1.1 billion rows and 6 vCPUs returned aggregations in under one second. The data model is organized around listing-session and account-event identifiers with up to 250 columns.
What happens if the asynchronous pipeline fails?
A prolonged downstream interruption can create a gap in monitoring data, but it does not affect the original fee calculation. The local queue can flush or overwrite old records when backlog exceeds capacity, prioritizing fee-processing over lossless monitoring.
Source and freshness
- Source
- innovation.ebayinc.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.