---
title: "Inference characteristics of Llama"
description: "Llama-2-70B is evaluated against GPT-3.5-turbo on serving cost and latency, with the comparison deliberately excluding GPT-4 and noting fine-tuning as a separate reason to choose Llama. The evaluation serves Llama on two 80-GB A100 GPUs in 16-bit precision, then uses transformer FLOP, memory-bandwidth, and KV-cache analysis to explain the difference between prompt processing and token generation. The measurements indicate that Llama-2 has worse completion-token pricing than GPT-3.5 on two A100s, while its prompt-token pricing is more than three times cheaper; eight A100s might improve pricing but introduce unacceptable latency. The resulting guidance favors GPT-3.5 for most workloads and open-source Llama models for prompt-heavy tasks such as classification or reranking, with batch processing and workloads lacking prompt tokens identified as additional cases where Llama may make sense."
---

# Inference characteristics of Llama

[Cursor](https://yomu.fyi/company/cursor) · Aman Sanger · Jul 20, 2023

**Type:** Benchmark

## Summary

Llama-2-70B is evaluated against GPT-3.5-turbo on serving cost and latency, with the comparison deliberately excluding GPT-4 and noting fine-tuning as a separate reason to choose Llama. The evaluation serves Llama on two 80-GB A100 GPUs in 16-bit precision, then uses transformer FLOP, memory-bandwidth, and KV-cache analysis to explain the difference between prompt processing and token generation. The measurements indicate that Llama-2 has worse completion-token pricing than GPT-3.5 on two A100s, while its prompt-token pricing is more than three times cheaper; eight A100s might improve pricing but introduce unacceptable latency. The resulting guidance favors GPT-3.5 for most workloads and open-source Llama models for prompt-heavy tasks such as classification or reranking, with batch processing and workloads lacking prompt tokens identified as additional cases where Llama may make sense.

## Context

The post compares Llama-2-70B with GPT-3.5-turbo specifically on serving cost and latency. It frames the comparison around choosing a cheap language model, while explicitly excluding fine-tuning and GPT-4 from the analysis.

## Approach / What changed

Llama-2-70B is served on two 80-GB A100 GPUs using 16-bit precision and compared with GPT-3.5-turbo at roughly similar latencies. The analysis combines measured latency and pricing data with transformer FLOP, memory-bandwidth, and KV-cache calculations for prompt processing and completion generation.

## Takeaways

- During completion generation, model weights and the KV cache are reread for each token; increasing batch size leaves other memory requirements roughly constant apart from the KV cache.
- Llama 2 uses grouped query attention with eight key/value heads or groups, compared with 128 for conventional multi-head attention and one for multi-query attention; the post says this improves performance.
- Prompt processing batches matrix multiplication across the sequence dimension, making its per-token memory requirement significantly lower than that of generating completion tokens.

**Tags:** [GPU](https://yomu.fyi/topic/gpu), [LLMs](https://yomu.fyi/topic/llm), [Model Serving](https://yomu.fyi/topic/model-serving), [Performance](https://yomu.fyi/topic/performance)

- Source: [Cursor](https://cursor.com/blog/llama-inference)
- Source URL: https://cursor.com/blog/llama-inference
- Ingested by Yomu: 2026-09-02T11:54:54.675Z

[Read original post](https://cursor.com/blog/llama-inference)