Dynamic Memory Compression

Dynamic Memory Compression (ICML 2024): learned per-head adaptive KV-cache compression that retrofits existing LLMs for up to ~4x inference throughput. Highly adopted — it seeded NVIDIA's KV-compression line. Its direct successor, DMS ("Inference-Time Hyper-Scaling with KV Cache Compression", 2025-06), reaches 8x KV compression with ~1K training steps and beats training-free sparse attention, enabling more generated tokens per fixed compute budget. Joint Edinburgh (Nawrot, Ponti) + NVIDIA.