Building a Transformer-Based Category Recommender at Thumbtack

Summary

Thumbtack developed a transformer-based category recommender to personalize email and push notification suggestions across roughly 500 home service categories. Because user activity is sparse while contextual signals like location and climate are rich, the architecture represents search history and context features as sequences of tokens evaluated against candidate category tokens using cross-attention. Initial training with naive cross-entropy suffered from severe popularity bias, skewing recommendations toward head categories and yielding a low tail hit rate. To resolve this issue, the team implemented adaptive mixed sampling, combining in-batch and random negative candidates and keeping only top-scoring hard negatives for cross-entropy loss computation. This method raised tail hit rate from 2.1% to 28.4%, improved overall HR@10 to 54.6%, and yielded a statistically significant 5% lift in requests per user in production A/B testing.

Context

Thumbtack needed a recommender system to surface relevant service categories out of roughly 500 options for marketing notifications. Unlike platforms with frequent user activity, home service interactions are sparse, though rich contextual signals like location and climate exist. Initial model training with naive cross-entropy suffered from severe popularity bias, where the model over-indexed on high-volume head categories and ignored long-tail categories.

Approach / What changed

The team designed a multi-token transformer embedding user search history and contextual feature buckets into tokens, scoring candidate categories through cross-attention. To support varied campaign types, tokens are randomly masked during training and runtime. To mitigate popularity bias, Thumbtack implemented adaptive mixed sampling: each batch combines positive targets, in-batch negatives, and random negatives, scores them in one pass, keeps only top-K hard negatives, and calculates cross-entropy loss.

Takeaways

• Training with naive cross-entropy on skewed category distributions resulted in popularity bias that degraded tail HR@10 to 2.1%.
• A multi-token transformer with cross-attention enables runtime token masking, allowing a single model to support distinct campaign themes like location-based or activity-based recommendations.
• Adaptive mixed sampling combining random, in-batch, and top-K hard negatives increased tail HR@10 to 28.4% and produced a 5% lift in requests per user in an A/B test.

Related reading

• Amazon · May 15, 2026

  Making LLMs faster without sacrificing accuracy

  Standard scaling laws optimize parameter count and training data budgets to minimize loss but ignore internal Transformer architecture configurations. To address throughput disparities among equally sized models, researchers developed a scaling framework incorporating hidden size, the multilayer perceptron to attention parameter ratio, and grouped-query attention. The method calibrates a correction surface over standard Chinchilla scaling laws to independently optimize separable architectural parameters for target accuracy. Evaluated across more than 200 models up to three billion parameters, the framework produced the Panda model family for maximum accuracy and the Surefire family for Pareto efficiency. Surefire models matched or exceeded LLaMA-3.2 accuracy while increasing inference throughput by 12% to 47% across vLLM and SGLang serving systems on modern GPUs.

  Tao Yu, Youngsuk Park

• Thumbtack · Apr 29, 2026

  Evaluating AI at Scale: How Thumbtack Approaches Reliability, Safety, and Quality in GenAI