AuthorsTao Ge, Jing Hu, Lei Wang et al.
TL;DR
In-context Autoencoder (ICAE) uses LoRA-augmented in-context autoencoding to compress long contexts into 128 memory slots, achieving 4× compression with 99.1% BLEU and 0.017 loss on Llama-7B.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Transformer LLMs suffer when contexts grow long; even with ICAE’s 128 memory slots, BLEU and EM decline once context length exceeds 400–500 tokens.
This degradation means Llama-based systems lose fidelity on long documents, hurting downstream tasks like retrieval-augmented generation and instruction following over multi-page contexts where full text no longer fits efficiently.
HOW IT WORKS
ICAE’s core mechanism combines a LoRA-adapted encoder, memory tokens and slots, and a fixed LLM decoder trained with joint autoencoding and language modeling objectives.
You can think of ICAE like a human using shorthand notes: the encoder writes compressed “memory cards,” and the decoder later reads only those cards instead of the entire book.
This in-context autoencoding lets ICAE represent a 512-token context with just 128 memory slots, enabling 4× compression while keeping Llama’s behavior compatible with compressed working memory rather than a raw context window.
DIAGRAM
This diagram shows how ICAE compresses a context into memory slots and then conditions the LLM decoder on those slots plus a prompt at inference time.
DIAGRAM
This diagram shows how ICAE is pretrained on Pile with AE and LM, then instruction fine tuned on PWC, and how different pretraining setups are compared.
PROCESS
Autoencoding pretraining
ICAE uses the LoRA-adapted encoder and memory tokens to reconstruct Pile contexts from memory slots, tuning the autoencoding objective so the LLM decoder can restore original text.
Text continuation pretraining
ICAE trains on continuation of Pile contexts, letting the LLM decoder generate future tokens from memory slots and improving generalization beyond pure reconstruction.
Instruction fine tuning
ICAE fine tunes on the PWC dataset so memory slots plus prompts lead the LLM decoder to produce desired responses across 240k context prompt response triples.
In-context autoencoding at inference
At test time ICAE encodes a long context into 128 memory slots, concatenates them with a new prompt, and lets the LLM decoder answer using compressed working memory.
KEY CONTRIBUTIONS
In-context Autoencoder for context compression
ICAE introduces a LoRA-adapted encoder and memory slots on top of Llama, achieving 4× compression of 512-token contexts into 128 slots with 99.1% BLEU and 0.017 loss.
Pretraining with AE and LM objectives
ICAE combines autoencoding and text continuation pretraining, where the joint objective yields a 6.4× win lose ratio over a non pretrained ICAE at k=128 on PWC.
Instruction fine tuning with PWC dataset
ICAE fine tunes on the 240k sample PWC dataset, reaching a 74.2% win plus tie rate against GPT 4 when using 128 memory slots on Llama 2 7b chat.
RESULTS
BLEU-4
99.1%
0.7% below perfect reconstruction for 512→128 compression on Llama-7b
Loss
0.017
0.033 lower than k=64 autoencoding loss at 500-token contexts
PPL (memory slot)
9.50
+0.49 over PPL 9.01 with original 512-token context on Llama-7b
Win+tie vs GPT-4
74.2%
18.9 points higher than 55.3% for k=64 ICAE on Llama-2-7b-chat
These metrics come from Pile autoencoding and PWC instruction following, showing that ICAE maintains near lossless reconstruction and modest perplexity increases while compressing contexts 4× and preserving response quality against GPT-4 baselines.
BENCHMARK
These metrics come from Pile autoencoding and PWC instruction following, showing that ICAE maintains near lossless reconstruction and modest perplexity increases while compressing contexts 4× and preserving response quality against GPT-4 baselines.
BENCHMARK
Win+tie rate judged by GPT-4 when conditioning Llama-2-7b-chat on original contexts versus ICAE memory slots.
KEY INSIGHT
ICAE’s 128-slot memory achieves 99.3 BLEU on normal 512-token texts but only 3.5 BLEU on patterned random texts and 0.2 BLEU on completely random texts.
This is surprising because one might expect a compression mechanism to treat all token sequences similarly, yet ICAE behaves more like humans, memorizing meaningful structure far better than random noise.
WHY IT MATTERS
ICAE unlocks a practical way to bolt a working memory module onto existing Llama-family LLMs, compressing 512-token contexts into 128-slot memories with minimal performance loss.
Builders can now cache compressed memory slots for long documents, laws, or textbooks and reuse them across prompts, enabling faster, cheaper long-context reasoning without retraining or modifying the base decoder.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Shannan Yan, Jingchen Ni et al.
· 2026
AdaMem organizes dialogue history into Working Memory, Episodic Memory, Persona Memory, and Graph Memory coordinated by a Memory Agent, Research Agent, and Working Agent. On LoCoMo with GPT-4.1-mini, AdaMem achieves 44.65 F1 overall, beating the best baseline LangMem at 41.76 F1 by +2.89.
Answers use this explainer on Memory Papers.
Checking…