AuthorsVivek Chari, Guanghui Qin, Benjamin Van Durme
TL;DR
KV-DISTILL uses token-level KV cache distillation with a mixed forward–reverse KL objective to reach 86.6% SQuAD accuracy at 25% KV retention on LLAMA-3, only 1.0 points below the uncompressed baseline.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Self-attention has quadratic complexity, and the KV cache grows linearly with sequence length, becoming a primary GPU memory bottleneck.
This limits long-context LLMs; even with uncompressed contexts, models often underutilize them, wasting memory and hurting downstream question answering and summarization.
HOW IT WORKS
KV-DISTILL uses a state selection for cache compression scorer, a LoRA-adapted architecture in LMθ, and a KL-based objective function to distill long KV caches into shorter ones.
You can think of KV-DISTILL like compressing RAM pages into a few hot cache lines, where selected tokens act as packed summaries of the entire context.
This design lets KV-DISTILL preserve next-token distributions nearly losslessly, enabling 10–1000x compression that a plain context window or naive eviction cannot achieve.
DIAGRAM
This diagram shows how KV-DISTILL compresses a fixed context once and reuses the compressed KV cache across many future questions.
DIAGRAM
This diagram shows how KV-DISTILL is trained with mixed KL divergence on instruction data and evaluated on SQuAD, QuALITY, SQuALITY, and GovReport.
PROCESS
State Selection for Cache Compression
KV-DISTILL applies the state selection for cache compression scorer to hidden states X′η, computing importance scores and choosing top k token indices shared across layers.
Architecture
KV-DISTILL routes selected tokens through LoRA-adapted WQ and WO matrices in LMθ, while unselected tokens use frozen weights, packing information into the retained KV rows.
Objective Function
KV-DISTILL computes forward and reverse KL divergences between p and qθ next-token distributions, mixing them with λ to train LMθ on compressed KV caches.
Needle-In-a-Haystack and SQuAD Evaluation
KV-DISTILL evaluates compressed caches on Needle-in-a-Haystack, SQuAD, QuALITY, SQuALITY, and GovReport to measure retrieval, extractive QA, and summarization performance under varying KV retention.
KEY CONTRIBUTIONS
Key-Value Distillation
KV-DISTILL introduces key-value distillation that treats compressed and uncompressed KV caches as student teacher pairs, using a mixed forward reverse KL objective to match next token distributions.
State Selection for Cache Compression
KV-DISTILL proposes a learnable state selection for cache compression scorer over hidden states X′η, enabling question independent token subselection that supports arbitrary KV retention fractions between 0.1 and 80 percent.
Conditional LoRA Architecture
KV-DISTILL designs a conditional LoRA architecture that routes selected tokens through adapted WQ and WO matrices, achieving up to 1000x compression while preserving downstream performance on SQuAD, QuALITY, SQuALITY, and GovReport.
RESULTS
0-Shot Acc.
86.6%
-1.0 over LLAMA3 BASE at 25% KV retention
0-Shot Acc.
86.0%
-1.6 over LLAMA3 BASE at 20% KV retention
0-Shot Acc.
56.6%
-30.0 vs KV-DISTILL on LLAMA3 at 25% KV retention for H2I
0-Shot Acc.
73.3%
-12.7 vs KV-DISTILL on LLAMA3 at 20% KV retention for DODO
On SQuAD, which tests extractive question answering, KV-DISTILL with 25% KV retention reaches 86.6% accuracy on LLAMA-3 versus 87.6% for LLAMA3 BASE, while H2I at 25% retention drops to 56.6% and DODO at 20% retention reaches 73.3%, demonstrating nearly lossless compression in a worst case extractive setting.
BENCHMARK
On SQuAD, which tests extractive question answering, KV-DISTILL with 25% KV retention reaches 86.6% accuracy on LLAMA-3 versus 87.6% for LLAMA3 BASE, while H2I at 25% retention drops to 56.6% and DODO at 20% retention reaches 73.3%, demonstrating nearly lossless compression in a worst case extractive setting.
BENCHMARK
0-Shot Acc. on SQuAD for LLAMA-3 8B under different KV retention methods.
KEY INSIGHT
KV-DISTILL maintains 86.6% SQuAD accuracy on LLAMA-3 at 25% KV retention, only 1.0 points below the 87.6% uncompressed baseline.
This is surprising because aggressive 4x KV compression usually causes large drops, yet KV-DISTILL stays nearly lossless while H2I at 25% retention falls to 56.6%.
WHY IT MATTERS
KV-DISTILL unlocks nearly lossless, question independent KV cache compression, enabling 10–1000x context reduction while preserving long context reasoning.
Builders can now pre compress large static contexts once, reuse compressed KV caches across many queries, and deploy long context LLM applications within strict GPU memory budgets.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Shannan Yan, Jingchen Ni et al.
· 2026
AdaMem organizes dialogue history into Working Memory, Episodic Memory, Persona Memory, and Graph Memory coordinated by a Memory Agent, Research Agent, and Working Agent. On LoCoMo with GPT-4.1-mini, AdaMem achieves 44.65 F1 overall, beating the best baseline LangMem at 41.76 F1 by +2.89.
Answers use this explainer on Memory Papers.
Checking…