Engineering stories behind the Medium Daily Digest Algorithm: Part 2

Summary

Medium’s Daily Digest filtering had become costly and difficult to maintain after a single Bloom Filter instance accumulated billions of items for read, muted, and presentation rules. The system could face up to 15,000 filter questions per feed and more than 15 million per second across thousands of feeds, while Bloom Filters could not delete or enumerate entries and introduced derived-data drift. The team replaced the muted-writer filter with a user-based database query that fetched all muted writers for the current user and cross-referenced them with the story shortlist, then split read filtering between early user-based queries and later pairwise lookups. The resulting implementation removed Bloom Filters entirely, was reported as ten times cheaper in their situation, made costs independent of shortlist size, supported reading-history clearing, and was easier to control and debug.

Takeaways

1. 1

   Bloom Filters supported fast membership checks but could not list or delete entries, so an unmuted writer could remain effectively muted and stored data could drift from the source database.

2. 2

   For most users, querying all muted writers from DynamoDB retrieved fewer than one item per feed on average; users who muted thousands of writers could still be handled in real time, with higher latency.

3. 3

   The new implementation removed Bloom Filters entirely and was reported as ten times cheaper in this situation, while making filtering costs independent of the initial shortlist size and enabling reading-history clearing.