Making LLMs faster without sacrificing accuracy

Summary

Standard scaling laws optimize parameter count and training data budgets to minimize loss but ignore internal Transformer architecture configurations. To address throughput disparities among equally sized models, researchers developed a scaling framework incorporating hidden size, the multilayer perceptron to attention parameter ratio, and grouped-query attention. The method calibrates a correction surface over standard Chinchilla scaling laws to independently optimize separable architectural parameters for target accuracy. Evaluated across more than 200 models up to three billion parameters, the framework produced the Panda model family for maximum accuracy and the Surefire family for Pareto efficiency. Surefire models matched or exceeded LLaMA-3.2 accuracy while increasing inference throughput by 12% to 47% across vLLM and SGLang serving systems on modern GPUs.

Context

Existing scaling laws like Chinchilla optimize parameter count and training token budgets for a target loss level but do not account for internal architectural parameters. Consequently, models with identical parameter counts and training data can exhibit inference throughput differences of up to 40%.

Approach / What changed

Researchers augmented the Chinchilla scaling framework by modeling three architectural factors: hidden size, the ratio of MLP parameters to attention parameters, and grouped-query attention. The framework fits standard Chinchilla coefficients to determine a reference loss and calibrates a correction surface across architectural configurations, allowing separable parameters to be optimized independently to find Pareto-optimal designs.

Takeaways

• Two models with identical parameter counts, training datasets, and accuracy can differ by up to 40% in inference throughput depending on internal architectural choices.
• Optimal MLP-to-attention parameter ratios for LLaMA-3.2-style architectures sit near 1.0, indicating that open-weight models with higher ratios such as 4.8 over-allocate parameters to MLP layers.
• Architectural scaling laws calibrated on small models between 80 million and 297 million parameters reliably predict optimal configurations at one-billion and three-billion parameter scales.

Related reading

• huggingface.co · Oct 10, 2020

  Transformer-based Encoder-Decoder Models

  Natural language generation tasks map input sequences to target sequences whose lengths cannot be known in advance and vary by content. Standard deep neural networks struggle with these variable mappings because their inputs and targets require fixed-dimensional vector representations. While recurrent neural networks addressed this challenge by generating target sequences auto-regressively from a compressed context state, transformer-based encoder-decoder architectures became the standard paradigm in modern natural language processing. The framework processes sequence-to-sequence problems by encoding source sequences and auto-regressively decoding target sequences token by token using conditional probability distributions. A step-by-step implementation demonstrates greedy decoding with Hugging Face Transformers, passing cached encoder hidden states alongside previously generated target tokens to iteratively generate German translations from English inputs.

  Patrick von Platen

• huggingface.co · Jul 3, 2020

  The Reformer - Pushing the limits of language modeling

  Standard transformer models hit memory bottlenecks on long sequence modeling tasks due to the quadratic asymptotic memory complexity of global self-attention and oversized positional embedding matrices. The Reformer architecture overcomes these constraints to train sequences of up to half a million tokens using under 8GB of RAM. It re-engineers transformer operations using local and Locality Sensitive Hashing self-attention, chunked feed forward layers, reversible residual layers, and axial positional encodings. In empirical benchmarks using google/reformer-crime-and-punishment, axial positional encodings reduce the model parameter count from over 136 million to approximately 2.58 million by factorizing the positional dimensions. This architectural change cuts inference memory consumption from 959 MB down to 447 MB for evaluated benchmark workloads.