KV Cache

AgentKVShift reuses KV cache per retrieved memory unit using Probe Selection, Mean Estimation, and AgentKVShift Correction to adjust stale keys and values. On LoCoMo with AMem and Qwen2.5-3B-Instruct, AgentKVShift achieves 0.319 F1 at recompute ratio 0.1, compared to 0.178 for CacheBlend and 0.125 for ProphetKV.

Agent Memory Below the Prompt stores each agent’s KV state in a block pool, quantizes it via a Q4 pipeline, reloads it with BatchQuantizedKVCache, and reuses it across phases using cross-phase context injection. On Gemma 3 12B, Agent Memory Below the Prompt reduces cold TTFT from 172,096 ms to 1,264 ms at 32K context (136×) compared to FP16 prefix caching baselines like vllm-mlx.

ARKV dynamically combines Per-layer OQ ratio estimation, Token importance scoring, and Tri-state cache assignment to manage KV cache precision under a global memory budget. On LongBench, ARKV reaches 0.972 relative performance versus 0.979 for Origin while achieving 4× KV memory reduction and maintaining ~86% Tokens Per Second.

Fan Duality Model (FDM) uses the Fan Operator, Local-Global Cache, Freeze-Scan Training, and Holographic Reference Beam Decoding to separate wave-like compression from particle-like associative recall. On WikiText-103, Fan Duality Model (FDM) reaches 64.9 perplexity with Freeze-Scan and 62.79 with holographic decoding, while achieving 0.966 MQAR accuracy compared to Transformer at 0.606.

Consolidator attaches a shared slot-local Consolidator, hierarchical router, routed short-term memory, persistent long-term memory, and sliding-attention KV cache to PMNet to learn replay-free STM-to-LTM consolidation. On the two-segment modulo-10 same-address update task, Consolidator reaches 87.02% updated-mapping LTM recall versus 18.32% for forced identity accumulation using only 12.35K trainable parameters.

GradMem combines a WRITE phase, a READ phase, a context encoder Eθ, a self-supervised WRITE objective Lwrite, and a meta-learned initialization M0 to optimize prefix memory tokens via test-time gradient descent while keeping model weights frozen. On associative KV-retrieval with 96 key–value pairs, GradMem with 5 gradient WRITE steps reaches 88.4% exact match versus 12.9% for forward-only RMT with the same 8-vector memory.

Knowledge Packs pre-compute KV Cache Injection, KV–Prefix Equivalence, Banked Routing, and KV Composition to deliver retrieved knowledge and steering via KV states instead of prompt tokens. On HotpotQA, Knowledge Packs’ KV-chat matches RAG at 65.2% EM on Qwen3-8B with 0/500 divergences while eliminating 284 tokens of retrieval text per query.

SleepGate augments transformers with a Conflict-Aware Temporal Tagger, Forgetting Gate, Consolidation Module, and Sleep Trigger that periodically rewrite the KV cache during sleep micro-cycles. On the PI-LLM benchmark, SleepGate achieves 99.5% retrieval accuracy at PI depth 5 and 97.0% at depth 10, while full KV cache, sliding window, H2O, StreamingLLM, and a decay-only ablation all stay below 18% across all depths.

KV-DISTILL combines a learnable state selection scorer, a LoRA-based adapted LMθ, and a KL-type objective function to compress KV caches into shorter, reusable representations. On SQuAD, KV-DISTILL with 25% KV retention achieves 86.6% accuracy on LLAMA-3 versus 87.6% for the uncompressed baseline, while H2I drops to 56.6%.

MEPIC extends vLLM with a Chunk Cache Coordinator, Chunk Matcher, Hybrid KV Manager, Chunk LRU Manager, and Chunk Processor to manage canonical, page-aligned, position-independent KV chunks in HBM. On long-context workloads, MEPIC reduces HBM usage by up to 5.21× and lowers latency by up to 11.48% compared to CacheBlend on Mistral-7B-Instruct-v0.3.

TurboQuant combines MSE Optimal TurboQuant, Inner-product Optimal TurboQuant, QJL, Random Rotation Matrix Π, and Lloyd-Max Quantizer to quantize vectors online with near-optimal distortion-rate guarantees. TurboQuant matches the Shannon lower bound within a factor of √(3π/2)≈2.7 for MSE and achieves absolute quality neutrality for KV cache quantization at 3.5 bits per channel compared to full-precision baselines.

Activation Beacon progressively compresses long contexts by interleaving beacon tokens, using a tailored compression workflow, compression-based auto-regression, and chunk-wise random compression ratio training to store information in activation space. On LongBench, Activation Beacon reaches 34.9 single-doc QA vs 34.8 for Full-FT Llama-2-7B while delivering 2x acceleration and 8x KV cache reduction on 128K-context NeedleBench-style tasks.

MemLong augments OpenLLaMA-3B with a Ret-Mem module, Memory Bank, Retriever, Retrieval Causal Attention, and Dynamic Memory Update to store and retrieve chunk-level K-V caches via dense embeddings. On PG19 at 16k tokens, MemLong with 32K Memory reaches perplexity 9.73 vs 10.37 for MemLong-3B* without memory, and achieves up to +10.2 percentage points over OpenLLaMA on retrieval-augmented in-context learning.