Memory Papers
Memory Architecture

CAMELoT: Towards Large Language Models with Training-Free Consolidated Associative Memory

AuthorsZexue He, Leonid Karlinsky, Donghyun Kim et al.

arXiv 20242024
arXivPDF

TL;DR

CAMELoT adds training‑free consolidated associative memory with novelty–recency balancing to LLaMA2‑7B, cutting Arxiv long‑context perplexity from 5.12 to 3.60 (29.7% reduction).

SharePost on XLinkedIn

Read our summary here, or open the publisher PDF on the next tab.

THE PROBLEM

Transformers Break on Long Contexts with 29.7% Perplexity Gap on Arxiv

Standard attention LLMs have quadratic L² cost, so LLaMa2‑7B reaches 5.12 average perplexity on Arxiv when limited to finite context windows.

This constraint means long document modeling and few‑shot ICL over long prompts are bottlenecked by context size, hurting downstream tasks like question answering and long‑range language modeling.

HOW IT WORKS

CAMELOT: Consolidated Associative Memory Enhanced Long Transformer

CAMELoT introduces Associative Memory (AM) modules with Read Operation, Augment Operation, and Write Operation attached to each attention layer of a frozen LLaMa2‑7B.

You can think of CAMELoT like a cache hierarchy: the transformer context window is fast RAM, while the consolidated AM acts as a compressed disk of past key–value states, refreshed by recency.

By consolidating token representations into non‑parametric distributions and retrieving nearest modes, CAMELoT approximates full‑context attention far beyond the native window without retraining.

DIAGRAM

Token Level Read–Augment–Write Sequence in CAMELoT

This diagram shows how CAMELoT processes each token by reading from associative memory, augmenting attention, and writing back with novelty–recency aware consolidation.

DIAGRAM

Long Context Evaluation Pipeline for CAMELoT

This diagram shows how CAMELoT evaluates long documents by windowing text, updating associative memory, and computing perplexity across Wiki‑103, Arxiv, and PG‑19.

PROCESS

How CAMELoT Handles a Long Document Modeling Session

  1. 01

    Read Operation

    CAMELoT takes layer keys and values for each window token and uses the Read Operation to find the strongest associated memory slot per token via cosine similarity.

  2. 02

    Augment Operation

    CAMELoT prepends retrieved memory keys and values as prefixes, doubling effective context length and running causal attention over Kr and V r plus current tokens.

  3. 03

    Write Operation

    CAMELoT updates memory slots by consolidation when similarity exceeds threshold R, averaging Ki and Vi into Kmem and Vmem with counts cµ.

  4. 04

    Novelty and Recency

    CAMELoT detects novel concepts when similarity is below R, rewrites the oldest slot by age τµ, and increments ages for untouched slots to maintain recency.

KEY CONTRIBUTIONS

Key Contributions

  • 01

    Associative Memory Enabled LLM

    CAMELoT couples an Associative Memory module with Read Operation, Augment Operation, and Write Operation to any frozen attention LLM without retraining, approximating unlimited context attention.

  • 02

    Novelty and Recency Aware Consolidation

    CAMELoT introduces a training free Write Operation that consolidates token representations into non parametric distributions, balancing novelty via threshold R=0.93 and recency via age τµ.

  • 03

    Long Context Perplexity Reduction

    CAMELoT reduces average perplexity on Arxiv from 5.12 to 3.60 and on PG 19 from 8.58 to 7.16 using 10k memory slots and LLaMa2 7B without fine tuning.

RESULTS

By the Numbers

PG-19 Avg Perplexity

7.16

-1.42 vs LLaMa2-7B (8.58)

Arxiv Avg Perplexity

3.60

-1.52 vs LLaMa2-7B (5.12)

Wikitext-103 Avg Perplexity

14.13

-0.96 vs LLaMa2-7B (15.09)

TREC-50 Accuracy

0.44

+0.29 over LLaMa2-7B 0-shot (0.15) with 20k prompt length

Table 1 reports long context causal language modeling perplexity on PG 19, Arxiv, and Wikitext 103, while Table 2 reports TREC 50 few shot ICL accuracy. These results show CAMELoT achieves up to 29.7% perplexity reduction on Arxiv and large ICL gains without retraining.

BENCHMARK

By the Numbers

Table 1 reports long context causal language modeling perplexity on PG 19, Arxiv, and Wikitext 103, while Table 2 reports TREC 50 few shot ICL accuracy. These results show CAMELoT achieves up to 29.7% perplexity reduction on Arxiv and large ICL gains without retraining.

BENCHMARK

Language Modeling Perplexity on Arxiv (Average over Input Lengths)

Average test perplexity on Arxiv for LLaMa2-7B and memory augmented baselines.

BENCHMARK

Test Perplexity on PG-19 by Input Length

PG-19 test perplexity for CAMELoT and LLaMa2-7B across different input lengths.

KEY INSIGHT

The Counterintuitive Finding

CAMELoT maintains PG 19 perplexity around 7.1 even with tiny input length 128, versus 14.4 for LLaMa2 7B at the same length.

This is surprising because shorter context windows usually hurt performance, yet CAMELoT leverages associative memory to keep long range information without increasing visible context.

WHY IT MATTERS

What this unlocks for the field

CAMELoT unlocks training free, plug and play long context handling for frozen LLaMa2 7B using associative memory with novelty and recency aware consolidation.

Builders can now deploy long document modeling and large few shot prompts on limited hardware by running short window attention while CAMELoT approximates full context via memory.

~12 min read← Back to papers

Related papers

Memory Architecture

A Control Architecture for Training-Free Memory Use

Yanzhen Lu, Muchen Jiang et al.

· 2026

TAG routes low-confidence steps to uncertainty-based routing, filters them with guarded acceptance with rollback, chooses between bank selection across rule and exemplar memory, and prunes via evidence-based retirement inside a unified control loop. On SVAMP and ASDiv, TAG reaches 81.0% and 85.2% accuracy, improving over the 74.0% and 77.5% no-memory baselines while a compute-matched Retry baseline stays flat.

arXiv:2604.18206Read explainer

Questions about this paper?

Paper: CAMELoT: Towards Large Language Models with Training-Free Consolidated Associative Memory

Answers use this explainer on Memory Papers.

Checking…