AuthorsPeitian Zhang, Zheng Liu, Shitao Xiao et al.
TL;DR
Activation Beacon compresses long-context KV activations into beacon tokens, preserving LongBench performance while achieving 2x speedup and 8x KV cache reduction at 128K context.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Transformer LLMs face quadratic self attention cost and huge KV cache memory as context grows to 128K and beyond, making inference impractical.
Existing context compression using soft tokens struggles to summarize long documents, cannot exceed the base window, and often requires re-encoding, hurting both efficiency and flexibility.
HOW IT WORKS
Activation Beacon introduces beacon token activations, a progressive compression workflow, and compression-based auto-regression trained with a chunk-wise random compression ratio to store long-context information directly in KV space.
You can think of Activation Beacon as adding compact “checkpoints” in RAM that summarize each chunk, instead of copying the whole document into a few soft prompts every time.
This design lets Activation Beacon handle contexts far beyond the base window, cut KV cache by up to x8, and avoid re-encoding while preserving fine-grained information.
DIAGRAM
This diagram shows how Activation Beacon encodes chunks sequentially, accumulates beacon activations, and reuses them to compress longer-than-window contexts.
DIAGRAM
This diagram shows how Activation Beacon trains with compression-based auto-regression and chunk-wise random compression ratios.
PROCESS
Compression Mechanism
Activation Beacon applies the compression mechanism by partitioning the context into chunks and interleaving beacon tokens at the end of fine-grained units within each chunk.
Encoding and Compression
During encoding and compression, Activation Beacon modifies self attention so beacon tokens get dedicated projection matrices and distill each chunk into their keys and values.
Efficiency Analysis
In efficiency analysis, Activation Beacon restricts attention to local chunks plus accumulated beacons, reducing FLOPs and KV cache by the chosen compression ratio.
Compression-Based Auto-Regression
With compression-based auto-regression, Activation Beacon trains beacon projections and embeddings while freezing LLM parameters, using chunk-wise random compression ratio to support flexible deployment.
KEY CONTRIBUTIONS
Compression Mechanism
Activation Beacon introduces a progressive compression mechanism that directly compresses per-layer KV activations into beacon tokens, enabling inputs far beyond the base window size.
Compression-Based Auto-Regression
Activation Beacon uses compression-based auto-regression to train only beacon projections and embeddings on 1B pre-training tokens plus 30K fine-tuning samples with maximum length 20K.
Chunk-Wise Random Compression Ratio
Activation Beacon employs a chunk-wise random compression ratio sampled from {2,4,8,16,32}, allowing a single model to support x2–x32 compression with near-baseline quality.
RESULTS
Single-Doc
34.9
+0.1 over Full-FT Llama-2-7B
Multi-Doc
27.5
vs 27.5 for Full-FT Llama-2-7B
Few-Shot
61.4
-0.4 vs 61.8 Full-FT Llama-2-7B
Code
57.8
matches 57.8 Full-FT Llama-2-7B
On LongBench (32K context), Activation Beacon matches or slightly exceeds Full-FT Llama-2-7B across Single-Doc QA, Multi-Doc, Few-Shot, and Code while using compressed context. Combined with NeedleBench-style Multi-Needle-in-a-Haystack, this shows Activation Beacon preserves accuracy under x8 compression with 2x speedup and 8x KV cache reduction at 128K.
BENCHMARK
On LongBench (32K context), Activation Beacon matches or slightly exceeds Full-FT Llama-2-7B across Single-Doc QA, Multi-Doc, Few-Shot, and Code while using compressed context. Combined with NeedleBench-style Multi-Needle-in-a-Haystack, this shows Activation Beacon preserves accuracy under x8 compression with 2x speedup and 8x KV cache reduction at 128K.
BENCHMARK
Average Single-Doc score on LongBench Single-Doc tasks at 32K context.
KEY INSIGHT
Activation Beacon with x8 compression on Qwen-2-7B achieves 9.70 accuracy in 1-turn 128K Multi-Needle-in-a-Haystack with 2.445 seconds latency.
This is surprising because heavy compression is expected to lose fine-grained needles, yet Activation Beacon nearly matches the 9.75 accuracy of the uncompressed Full-FT baseline while halving latency.
WHY IT MATTERS
Activation Beacon makes it practical to run 7B–72B LLMs on 128K–256K contexts with x8 KV cache reduction and more than x4 FLOPs savings at 256K.
Builders can now deploy long-context applications like document understanding, few-shot learning, and multi-turn retrieval over 128K inputs without retraining the backbone or sacrificing short-context performance.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Shannan Yan, Jingchen Ni et al.
· 2026
AdaMem organizes dialogue history into Working Memory, Episodic Memory, Persona Memory, and Graph Memory coordinated by a Memory Agent, Research Agent, and Working Agent. On LoCoMo with GPT-4.1-mini, AdaMem achieves 44.65 F1 overall, beating the best baseline LangMem at 41.76 F1 by +2.89.
Answers use this explainer on Memory Papers.
Checking…