Two transformer models with the same parameter count can have very different memory requirements during inference. A major reason is the structure of their attention layers. The number of key-value heads, the representation stored for each token, and the number of earlier tokens retained by each layer determine how much KV cache a request needs.
When serving an existing checkpoint, you do not choose whether its layers use multi-head attention, grouped-query attention, multi-query attention, sliding-window attention, or multi-head latent attention. The trained model has already made those choices. Your task during inference is to inspect the architecture, calculate its memory requirements, and use a compatible cache layout and attention kernel. You can compare checkpoints before deployment, but you cannot replace one attention variant with another and expect identical behavior.
Attention Variants and Memory Efficiency.Photo by Chris Boland. Some rights reserved.
The checkpoint fixes the model configuration; the inference engine does not change it. The configuration and weights define the attention projections, numbers of query and key-value heads, local-attention pattern, and latent dimensions. Replacing MHA with GQA after training modifies the model and may require additional training to recover quality. Replacing full attention with a sliding window would discard context that the trained model expects to see.
Source: Machine Learning Mastery — Read the full article on the original site ↗