AuthorsKeqin Xie
TL;DR
LPC-SM uses Orthogonal Novelty Transport in a dual-timescale sparse memory block to keep 4096-token training stable with Stage-C LM loss 11.582.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Most long-context language models still rely on attention to handle both local interaction and long-range state, limiting alternative decompositions.
When LPC-SM removes mHC, Stage-A final LM loss jumps from 12.630 to 15.127, showing that monolithic attention-centered blocks can become brittle and hard to optimize.
HOW IT WORKS
LPC-SM builds each block from local attention, dual-timescale memory, predictive correction, Orthogonal Novelty Transport, and multi-head-coupled residual router mHC.
You can think of local attention and fast memory as RAM, slow memory as a disk, and ONT as a smart deduplicating write-back cache.
By explicitly routing novelty through ONT and exposing mismatch via predictive correction, LPC-SM can regulate sparse computation and persistent state in ways a plain context window cannot.
DIAGRAM
This diagram shows how LPC-SM updates fast and slow memory using chunk summaries and Orthogonal Novelty Transport at chunk boundaries.
DIAGRAM
This diagram shows the three-stage training schedule and Stage-A ablations used to evaluate LPC-SM.
PROCESS
Block Structure
LPC-SM first normalizes embeddings and routes them through local attention, dual-timescale memory, predictive correction, and optional mHC inside each block.
Dual-Timescale Memory and ONT
LPC-SM updates fast state every token, forms chunk summaries, and uses Orthogonal Novelty Transport to write novelty into slow memory at chunk boundaries.
Correction and Stopping
LPC-SM predicts hidden states from attention and memory, refines them via predictive correction, and uses a learned stop head to model EOS behavior.
Training Objective
LPC-SM optimizes LM loss plus auxiliary terms for predictive correction, sparsity, memory magnitude, and stopping to keep explicit mechanisms active.
KEY CONTRIBUTIONS
LPC-SM block decomposition
LPC-SM cleanly separates local attention, dual-timescale memory, predictive correction, and mHC routing, enabling a broader division of labor than attention alone.
Orthogonal Novelty Transport
LPC-SM introduces ONT for slow-memory writes, amplifying only orthogonal novelty so memory preserves aligned content while emphasizing genuinely new information.
Adaptive sparse control in continuation
LPC-SM uses learned sparse control that improves Stage-B final LM loss from 12.137 to 10.787 over a fixed sparse controller on OpenWebMath-10k.
RESULTS
Stage-A final LM loss
12.630
+2.497 over w o mHC
Stage-B final LM loss
10.787
+1.350 over fixed sparse control
Stage-C final LM loss
11.582
4096 token continuation stability
Delayed identifier CE
12.031
-2.365 after Stage C continuation
Stage A trains LPC-SM on Dolma3-base, Stage B on OpenWebMath-10k, and Stage C on LongMino continuation. The 12.517% LM loss improvement in Stage B shows LPC-SM's adaptive sparse control materially reshapes computation under mathematical continuation.
BENCHMARK
Stage A trains LPC-SM on Dolma3-base, Stage B on OpenWebMath-10k, and Stage C on LongMino continuation. The 12.517% LM loss improvement in Stage B shows LPC-SM's adaptive sparse control materially reshapes computation under mathematical continuation.
BENCHMARK
Final LM loss for LPC-SM and key Stage-A ablations on Dolma3-base.
BENCHMARK
Final LM loss for LPC-SM with adaptive sparse control versus fixed sparse control on OpenWebMath-10k.
KEY INSIGHT
Disabling ONT in Stage A reduces final LM loss from 12.630 to 11.781, even though ONT is designed to improve slow-memory writes.
This is counterintuitive because ONT explicitly preserves aligned content and amplifies novelty, yet short-budget LM loss suggests that this structured memory constraint can initially hurt optimization.
WHY IT MATTERS
LPC-SM shows that long-context autoregressive modeling can be organized around explicit local attention, dual-timescale memory, predictive correction, and internal control instead of a single attention mechanism.
Builders can now experiment with controllable sparsity, novelty-aware memory writes, and learned stopping inside one coherent block, probing behaviors beyond what extended context windows alone can offer.
Guilin Zhang, Wei Jiang et al.
· 2026
A-MAC scores candidate memories using Utility, Confidence, Novelty, Recency, and Type Prior combined by a learned linear admission policy with Algorithm 1 A-MAC Memory Admission. On the LoCoMo benchmark, A-MAC achieves F1 0.583 and 2644 ms latency, improving F1 by 0.042 and reducing latency by 1187 ms compared to A-mem.
Yi Yu, Liuyi Yao et al.
arXiv 2026 · 2026
Agentic Memory (AgeMem) exposes memory management tools, a three-stage progressive RL strategy, and step-wise GRPO directly inside the agent policy to jointly control long-term and short-term memory. On Qwen3-4B-Instruct, AgeMem attains 54.31% average performance across ALFWorld, SciWorld, PDDL, BabyAI, and HotpotQA, exceeding the best baseline A-Mem at 45.74%.
Can Lv, Heng Chang et al.
· 2026
All-Mem organizes long-term agent memory through Online/Offline Decoupling, Agentic Topology Consolidation, and Topology-Aware Retrieval over a topology-structured memory bank. On LoCoMo, All-Mem reaches 54.63 4o-J versus Mem0’s 48.91, and on LongMemEval-S All-Mem reaches 60.20 4o-J versus Mem0’s 55.80.
Answers use this explainer on Memory Papers.
Checking…