Building a Transformer-Based Category Recommender at Thumbtack

Summary

Thumbtack developed a transformer-based category recommender to personalize email and push notification suggestions across roughly 500 home service categories. Because user activity is sparse while contextual signals like location and climate are rich, the architecture represents search history and context features as sequences of tokens evaluated against candidate category tokens using cross-attention. Initial training with naive cross-entropy suffered from severe popularity bias, skewing recommendations toward head categories and yielding a low tail hit rate. To resolve this issue, the team implemented adaptive mixed sampling, combining in-batch and random negative candidates and keeping only top-scoring hard negatives for cross-entropy loss computation. This method raised tail hit rate from 2.1% to 28.4%, improved overall HR@10 to 54.6%, and yielded a statistically significant 5% lift in requests per user in production A/B testing.

Takeaways

1. 1

   Training with naive cross-entropy on skewed category distributions resulted in popularity bias that degraded tail HR@10 to 2.1%.

2. 2

   A multi-token transformer with cross-attention enables runtime token masking, allowing a single model to support distinct campaign themes like location-based or activity-based recommendations.

3. 3

   Adaptive mixed sampling combining random, in-batch, and top-K hard negatives increased tail HR@10 to 28.4% and produced a 5% lift in requests per user in an A/B test.