How Generative Recommenders Are Redefining RecSys at Scale

Summary

Generative recommenders (GRs) recast recommendation as next-item or next-action sequence prediction conditioned on user histories, replacing traditional embedding-similarity objectives and potentially unifying retrieval and ranking. The post presents HSTU, which uses timestamp-ordered interleaved items and actions, SiLU-based attention weighting, relative attention bias, and gating, alongside Semantic IDs, which derive a smaller vocabulary through hierarchical clustering of item embeddings. For production, recsys-examples combines TorchRec, DynamicEmb, Megatron-Core, recommender-specific KV and decoding components, while nv-embedding-cache provides tiered GPU, CPU, and remote embedding storage. On an H100, the Semantic ID path was reported as 1.85x higher in online throughput and 46% lower median latency than SGLang under the stated long-context, beam-search workload, and DLRM v3 reached 99,997 queries/sec.

Takeaways

1. 1

   HSTU replaces softmax attention normalization with SiLU-based weighting, adds relative attention bias, and applies elementwise gating before output projection to support efficient long-sequence recommendation.

2. 2

   Semantic IDs create a smaller vocabulary through hierarchical clustering of item embeddings, enabling autoregressive generation, beam search, and ranking from output logits.

3. 3

   In the stated H100 benchmark, Semantic ID-GR serving achieved about 19.7 requests per second versus 10.7 for SGLang, with about 198 ms versus 370 ms median latency.