Engineering stories behind the Medium Daily Digest Algorithm: Part 4

Summary

Medium’s Daily Digest team investigated complaints that recommendations had become repetitive, especially for power users, after earlier filtering changes. Data showed some stories were sent up to eight times to one user, while affected users saw digest posts repeated twice on average; the team also had to account for Apple Mail Privacy Protection, which prevents reliable open tracking. They therefore tracked digest email IDs when readers clicked a story, marking emails with at least one click as “definitely opened,” and added a recommendation-stack filter that counts prior occurrences in those digests. Testing supported a threshold of one occurrence: complaining users would lose nearly five stories per digest, whereas free users would lose fewer than one. An A/B test produced a statistically significant increase in reading time, with a larger rise among members; the team reported no further complaints and suggested soft filtering and email-position rules as possible refinements.

Takeaways

1. 1

   Story clicks provided a reliable signal that a digest email had actually been opened, despite Apple Mail Privacy Protection limiting open tracking.

2. 2

   A threshold of one prior occurrence would filter nearly five stories from complaining users’ digests while affecting free users by fewer than one story on average.

3. 3

   The A/B test produced a statistically significant reading-time increase, with a larger increase among members; the team identified soft filtering and email position as possible refinements.