Survey Cognitive Architecture Memory Architecture

Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures

AuthorsParsa Omidi, Xingshuai Huang, Axel Laborieux, Bahareh Nikpour

arXiv 20252025

TL;DR

Memory-Augmented Transformers + a three-axis taxonomy of objectives, memory types, and integration mechanisms + unifies recent architectures into a neuroscience-grounded roadmap for lifelong Transformers.

SharePost on X LinkedIn

Read our summary here, or open the publisher PDF on the next tab.

THE PROBLEM

Transformers Lose Coherence Beyond Fixed Context Windows

Memory-Augmented Transformers highlights that self attention’s quadratic complexity forces fixed context windows, where KV caches must evict or compress older entries, discarding vital information.

This limitation breaks long range reasoning and continual learning, causing knowledge integration failures and degraded coherence when sequences extend to hundreds of thousands or millions of tokens.

HOW IT WORKS

Memory-Augmented Transformers — a three-axis taxonomy from brain to architecture

Memory-Augmented Transformers introduces three taxonomic axes: functional objectives, memory types, and integration techniques, grounded in biological memory architectures and computational principles.

An analogy is a brain inspired operating system where sensory buffers, working memory, and long term stores map to embeddings, caches, and external or parametric memories.

This KEY_MECHANISM of aligning neuroscience principles with a structured taxonomy enables Memory-Augmented Transformers to explain designs that plain context windows and KV caches cannot systematically organize or extend.

DIAGRAM

Three-Dimensional Taxonomy of Memory-Augmented Transformers

This diagram shows how Memory-Augmented Transformers categorizes existing architectures by functional objectives, memory types, and integration techniques.

DIAGRAM

From Biological Memory to Memory-Augmented Transformer Design

This diagram shows how Memory-Augmented Transformers maps sensory, working, and long-term memory principles into concrete Transformer mechanisms and taxonomic axes.

PROCESS

How Memory-Augmented Transformers Structures the Design Space

01
Memory Architectures in Biological Cognitive Systems
Memory-Augmented Transformers first analyzes sensory memory, working memory, and long term memory to extract multi timescale and consolidation principles.
02
Computational Principles from Biological Memory
Memory-Augmented Transformers distills hierarchical resource allocation, attention memory coupling, neuromodulatory gating, replay, and associative retrieval as design heuristics.
03
Taxonomy of Memory-Augmented Transformers
Memory-Augmented Transformers defines three axes functional objectives, memory types, and integration techniques, and populates them with systems like Memformer and ATLAS.
04
Mechanisms of Memory Operations
Memory-Augmented Transformers surveys reading, writing, forgetting, capacity optimization, and self management to highlight shifts toward adaptive test time learning systems.

KEY CONTRIBUTIONS

Key Contributions

01
Comprehensive three-axis taxonomy
Memory-Augmented Transformers introduces a taxonomy over functional objectives, memory types, and integration techniques, covering systems from Transformer XL and Memformer to Titans and ATLAS.
02
Neuroscience grounded design principles
Memory-Augmented Transformers links hierarchical resource allocation, neuromodulatory gating, replay based consolidation, and associative retrieval to concrete Transformer mechanisms.
03
Unified view of memory operations
Memory-Augmented Transformers systematizes read, write, forgetting, capacity management, and self management, revealing a shift from static caches toward adaptive test time learning architectures.

RESULTS

By the Numbers

Context scales

up to 50 million tokens

ARMT demonstrates associative retrieval over 50 million token contexts

Temporal range gain

38% extension

Compressive Transformer boosts temporal range by 38 percent over Transformer XL style caching

Throughput gain

100× throughput

MATTER achieves 100 times throughput improvement over retrieve and read baselines

QA accuracy lift

from 25.8 to 44.3 EM

EMAT raises Natural Questions exact match from 25.8 to 44.3 with external QA memory

Memory-Augmented Transformers aggregates results across long context language modeling, multi hop QA, and retrieval augmented generation benchmarks to show how memory mechanisms extend context, increase throughput, and improve factual QA accuracy.

BENCHMARK

By the Numbers

BENCHMARK

Representative Improvements from Memory-Augmented Transformers’ Surveyed Systems

Selected gains in context length, temporal range, throughput, and QA accuracy for systems categorized by Memory-Augmented Transformers.

KEY INSIGHT

The Counterintuitive Finding

Memory-Augmented Transformers shows that systems like Compressive Transformer can extend temporal range by 38 percent while holding compute roughly constant through learned compression.

This challenges the assumption that longer context always requires proportionally more computation, suggesting hierarchical and compressed memories can scale without quadratic cost.

WHY IT MATTERS

What this unlocks for the field

Memory-Augmented Transformers enables a principled way to choose between parameter encoded, state based, explicit, and hybrid memories for specific objectives like OOD adaptation or reasoning.

Builders can now systematically combine attention fusion, gated control, and associative retrieval to design Transformers that approach lifelong, cognitively inspired behavior instead of ad hoc context window tweaks.

~14 min read← Back to papers

Related papers

SurveyAgent Memory

Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations

Dongming Jiang, Yi Li et al.

arXiv 2026 · 2026

Anatomy of Agentic Memory organizes agentic memory into four structures using components like Lightweight Semantic Memory, Entity-Centric and Personalized Memory, Episodic and Reflective Memory, and Structured and Hierarchical Memory. Anatomy of Agentic Memory then reports comparative results such as Nemori’s 0.781 semantic judge score on LoCoMo versus SimpleMem’s 0.298, and latency differences like 1.129s for Nemori versus 32.372s for MemoryOS.

arXiv:2602.19320 Read explainer

SurveyBenchmarkAgent MemoryLong-Term MemoryMemory Architecture

A Survey on the Security of Long-Term Memory in LLM Agents: Toward Mnemonic Sovereignty

Zehao Lin, Chunyu Li, Kai Chen

· 2026

Mnemonic Sovereignty analyzes long term Write, Store, Retrieve, Execute, Share, and Forget Rollback phases against integrity, confidentiality, availability, and governance objectives for agent memory. Mnemonic Sovereignty’s lifecycle matrix shows most of the ~70 works cluster on write and retrieve integrity, leaving store, availability, and governance primitives like write gate validation and post deletion verification almost entirely unexplored.

arXiv:2604.16548 Read explainer

SurveyRAGAgent Memory

Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontiers

Pengfei Du

· 2026

Memory for Autonomous LLM Agents decomposes agent memory into a POMDP-grounded write–manage–read loop, a three-dimensional taxonomy, and five mechanism families spanning context compression, retrieval stores, reflection, hierarchical virtual context, and policy-learned management. Memory for Autonomous LLM Agents synthesizes results like Voyager’s 15.3× tech-tree speedup and MemoryArena’s 80%→45% drop to show that memory architecture often matters more than backbone choice.

arXiv:2603.07670 Read explainer

Questions about this paper?

Answers use this explainer on Memory Papers.

Checking…