LLM Inference Scaling: Techniques for Low-Latency Token Generation
Introduction
A user submits a prompt. The model generates 500 tokens. At 10 tokens/second, that’s 50 seconds of wall-clock time. At 100 tokens/second, it’s 5 seconds. The difference between those two numbers isn’t raw GPU power — it’s inference architecture. The same H100 cluster, configured differently, will produce radically different latency profiles. This article breaks down exactly why, and what production teams do about it.
Why LLM Inference Is Uniquely Hard
Unlike a CNN doing image classification, a large language model generating text is autoregressive — each output token depends on every token generated before it. You cannot parallelize generation across the output sequence. Token 50 must wait for token 49.
This creates two phases with completely different compute characteristics:
Prefill: Processing the input prompt. The entire prompt is available at once, so this is highly parallelizable. It saturates GPU compute (FLOP-bound). A 1,000-token prompt takes roughly the same wall-clock time as a 10-token one on modern hardware.
Decode: Generating output tokens one by one. Each step loads the entire model from GPU memory to compute a single token. This is memory-bandwidth-bound, not compute-bound. The GPU spends most of its time waiting on memory reads.
This asymmetry drives nearly every inference optimization.
KV Cache: The Non-Negotiable Foundation
During attention computation, every transformer layer produces Key (K) and Value (V) tensors for each token. In autoregressive decode, if you recompute these from scratch on every step, you’re doing quadratic redundant work.
The KV cache stores K and V tensors from all previous tokens. Each decode step only needs to compute K/V for the new token, then attend over the cached history. This reduces per-step compute from O(n²) to O(n).
The cost: memory. A single KV cache entry for one token across all layers in a 70B-parameter model consumes roughly 0.5–2 MB depending on precision and architecture. A 4,096-token sequence holds several GB of KV data per concurrent request. At 100 concurrent users, you’re looking at hundreds of gigabytes — before model weights.
PagedAttention (introduced by the vLLM project) solved the fragmentation problem that plagued early KV cache implementations. Instead of pre-allocating a contiguous memory block per request (wasting space when sequences are shorter than the maximum), PagedAttention stores KV cache in fixed-size non-contiguous “pages,” similar to virtual memory in an OS. This improved GPU memory utilization from ~20–40% to over 90% in benchmarks on real workloads as of late 2024.
