Scaling Large Language Models for e-Commerce: The Development of a Llama-Based Customized LLM

Summary

eBay developed e-Llama, customized 8B and 70B language models, by adapting Meta’s Llama 3.1 models to e-commerce rather than training entirely from scratch. The approach uses continued pretraining on filtered public listings, product reviews, classifier-selected open-source examples, replay data, and 10% non-English general-domain data to add domain knowledge while limiting catastrophic forgetting. Training ran on 480 NVIDIA H100 GPUs using Megatron-LM, 3D parallelism, flash-attention-2, a 1:1 general-to-e-commerce sampling ratio, a learning-rate maximum set to 10% of the original, and one trillion tokens. The 70B model required about one month and 340,000 GPU-hours. E-commerce benchmarks improved approximately 25% in English and 30% in non-English, while general-domain NLU degraded 1% for e-Llama 70B; subsequent instruction tuning added safety and instruction-following guardrails.

Takeaways

1. 1

   Replay examples from curated, publicly available, open-source, and higher-quality datasets were included to help prevent catastrophic forgetting.

2. 2

   Experiments selected a maximum learning rate equal to 10% of the original and a 1:1 general-to-e-commerce data sampling ratio, with cosine scheduling, warmup, approximately 11.8 million tokens per batch, and 85,000 update steps.

3. 3

   The 70B e-Llama model showed approximately 25% higher English and 30% higher non-English e-commerce benchmark performance, with only 1% degradation on general-domain NLU benchmarks.