Virtual Head Attention
paperYour notes
An attention design from Baidu's PaddlePaddle team that targets three costs at once: KV-cache size, training and prefill FLOPs, and quality. Their argument is that existing designs trade these against each other. GQA keeps quality but caches a lot, MLA compresses the cache but loses quality, and MFA compresses it but adds compute. VHA halves the number of physical query heads and adds two linear operations. Q Premix is a near-identity transform inside each KV group that recovers "virtual" head diversity. Linear Postmix mixes features across heads through a low-rank I + AB⊤ residual. Because both are linear, they fold into the query and output projections at inference, which leaves a plain GQA-2 model. Existing GQA kernels then run unchanged, with group sizes up to 64 that use Tensor Cores fully during decoding.
At four scales (4K context, 50B tokens) VHA beats standard GQA by 0.05–0.81%. It cuts the KV cache 4× against GQA-8 at dense scales and 2× against GQA-4 at the MoE scale, and cuts training and prefill FLOPs 7.9% at 1.7B. On an industrial 30B-A3B MoE trained on 500B tokens, it beats MLA by 0.70 and GQA-4 by 2.60 points on a 12-benchmark base-model suite. In a matched 10B-A1B comparison after long-context mid-training, its average RULER score over 8K–128K is 84.4 against 73.0 for GQA, with two KV heads instead of four. VHA ships in ERNIE 5.5 Flash Base, where it adds cross-head mixing to every attention layer. Accepted at NeurIPS 2026.