# Large Language Models: A New Moore's Law?

huggingface.co · Julien Simon · Oct 26, 2021

**Type:** Problem & solution

## Summary

Recent advancements in generative artificial intelligence have spurred the rapid growth of large language models like Megatron-Turing NLG 530B, which require vast financial investments and substantial energy consumption. Training these massive parameter architectures demands hundreds of multi-GPU servers and generates significant carbon footprints for relatively modest benchmark gains. Rather than relying on brute-force scaling, engineering teams can adopt smaller, more frugal architectures through knowledge distillation and transfer learning techniques. Practical alternatives include fine-tuning existing pretrained models, deploying on energy-efficient cloud infrastructure, and applying optimizations like pruning, layer fusion, and quantization. These methods deliver low-latency inference and high task accuracy while significantly decreasing hardware requirements, development time, and environmental impact.

## Context

The exponential growth of large language models requires massive hardware investments, high power consumption, and substantial cooling infrastructure, resulting in large carbon emissions and diminishing benchmark returns for real-world enterprise adoption.

## Approach / What changed

Adopt practical machine learning workflows by selecting pretrained models, utilizing smaller knowledge-distilled architectures such as DistilBERT or T0, fine-tuning on specialized datasets, deploying on cloud infrastructure, and applying optimizations such as pruning, layer fusion, and quantization.

## Takeaways

- Replicating massive models like Megatron-Turing NLG 530B requires hundreds of multi-GPU servers consuming up to 6.5 kilowatts each, incurring costs near 100 million dollars.
- Knowledge distillation techniques enable compact models like DistilBERT and T0 to retain high language understanding or outperform larger models like GPT-3 while significantly reducing size and latency.
- Model optimization strategies including pruning, layer fusion, quantization, and specialized hardware like TPUs or Inferentia enable faster inference and lower hardware demands.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance)

[Read original post](https://huggingface.co/blog/large-language-models)