Inference characteristics of Llama

Summary

Llama-2-70B is evaluated against GPT-3.5-turbo on serving cost and latency, with the comparison deliberately excluding GPT-4 and noting fine-tuning as a separate reason to choose Llama. The evaluation serves Llama on two 80-GB A100 GPUs in 16-bit precision, then uses transformer FLOP, memory-bandwidth, and KV-cache analysis to explain the difference between prompt processing and token generation. The measurements indicate that Llama-2 has worse completion-token pricing than GPT-3.5 on two A100s, while its prompt-token pricing is more than three times cheaper; eight A100s might improve pricing but introduce unacceptable latency. The resulting guidance favors GPT-3.5 for most workloads and open-source Llama models for prompt-heavy tasks such as classification or reranking, with batch processing and workloads lacking prompt tokens identified as additional cases where Llama may make sense.

Takeaways

1. 1

   During completion generation, model weights and the KV cache are reread for each token; increasing batch size leaves other memory requirements roughly constant apart from the KV cache.

2. 2

   Llama 2 uses grouped query attention with eight key/value heads or groups, compared with 128 for conventional multi-head attention and one for multi-query attention; the post says this improves performance.

3. 3

   Prompt processing batches matrix multiplication across the sequence dimension, making its per-token memory requirement significantly lower than that of generating completion tokens.