Cached key/value tensors from prior tokens so autoregressive inference reuses past attention work—grows with sequence length and dominates VRAM at long contexts.
During generation, recomputing attention for all past tokens each step would be quadratic and slow. A KV cache stores per-layer keys and values for tokens already processed. That is why long chats cost more memory and why systems like PagedAttention (popular in vLLM) manage cache pages like virtual memory to batch many requests.
Ops implication
Budget VRAM beyond weight size—especially for local or self-hosted LLMs and after quantization. High-throughput servers such as vLLM optimize KV memory layout (for example PagedAttention).