AuthorsXian Gao, Jinpeng Wang, Jiacheng Ruan et al.
TL;DR
MEMO uses query-conditioned evidence units plus a trained memory manager to mix text and visual layouts, reaching 73.91 F1 on 2Wiki under a 128-token budget.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
LLM agents accumulate long interaction histories, but a fixed context budget forces important and irrelevant content to compete at nearly uniform token cost.
On multi-hop QA benchmarks like 2WikiMultiHopQA, purely textual or purely visual memory under a 128-token budget yields only 56.26 and 35.89 F1, hurting downstream reasoning.
HOW IT WORKS
MEMO’s core mechanism chains a query-conditioned evidence extractor, deterministic unit-materialization, a trained memory manager, and working memory building into a single readout pipeline.
You can think of MEMO like a librarian that clips key passages into cards, then decides whether to file each card as text, lay it out visually, or keep both, under a shared shelf budget.
This evidence-level multimodal organization lets MEMO preserve precise facts as text while using visual layouts for structure, something a plain context window of text tokens alone cannot achieve.
DIAGRAM
This diagram shows how MEMO processes a query to extract evidence units, assign modalities, and build a text plus visual working memory under a shared budget.
DIAGRAM
This diagram shows how MEMO trains the evidence extractor and memory manager using benchmark evidence and frozen reader feedback, then evaluates on downstream tasks.
PROCESS
Query-Conditioned Evidence Extraction
MEMO uses the query-conditioned evidence extractor to select source identifiers and character-level spans from memory records, producing compact evidence candidates tailored to the query.
Unit-Materialization Step
MEMO deterministically copies content around each span, forming evidence units with fields like id, content xi, key spans hi, source si, and priority rule ci.
Joint Evidence Selection and Presentation
MEMO’s memory manager assigns each evidence unit an action from text, image, dual, or drop, and chooses visual templates such as cards, timelines, tables, or checklists under a shared budget.
Working Memory Building
MEMO’s working memory building module assembles a text packet T and visual page V from the planned units, verifies Ctext plus Cvis within budget B, and feeds this working memory to the frozen reader.
KEY CONTRIBUTIONS
Query-conditioned working memory construction
MEMO formulates memory readout as building query-specific working memory under a shared budget, integrating query-conditioned evidence extraction, memory manager decisions, and working memory building into one optimization framework.
Multimodal evidence units
MEMO introduces traceable evidence units with source links, key spans, and priority rules, enabling the memory manager to assign text, image, dual, or drop actions while preserving critical semantics.
Systematic multimodal evaluation
MEMO is evaluated on HotpotQA, 2WikiMultiHopQA, LoCoMo, and ALFWorld with readers like Qwen3-VL-32B, achieving 73.91 F1 on 2Wiki under a 128-token budget and using only 83.93 tokens on average without truncation.
RESULTS
F1
73.91%
+17.65 over Text-only on 2Wiki with Qwen3-VL-32B under 128-token budget
EM
62.20%
+5.78 over MemAgent on 2Wiki with Qwen3-VL-32B under 128-token budget
Overall F1
63.84%
+1.84 over MemAgent with Qwen3-VL-32B under 128-token budget
Tokens
83.93
50.5% fewer than MemAgent’s 169.55 tokens without truncation for InternVL3.5-8B
On 2WikiMultiHopQA, which tests multi-hop document reasoning, MEMO reaches 73.91 F1 under a 128-token budget with Qwen3-VL-32B, compared to 56.26 for Text-only and 35.89 for Visual-only memory. These results show that MEMO’s evidence-level multimodal organization yields higher answer quality while staying within strict context limits.
BENCHMARK
On 2WikiMultiHopQA, which tests multi-hop document reasoning, MEMO reaches 73.91 F1 under a 128-token budget with Qwen3-VL-32B, compared to 56.26 for Text-only and 35.89 for Visual-only memory. These results show that MEMO’s evidence-level multimodal organization yields higher answer quality while staying within strict context limits.
BENCHMARK
F1 on 2WikiMultiHopQA under a 128-token memory budget with Qwen3-VL-32B as the reader.
KEY INSIGHT
MEMO uses only 83.93 tokens on average without source-memory truncation, yet still ranks first or second on every overall EM and F1 metric.
This is surprising because many systems assume more context is always better, but MEMO shows carefully organized multimodal memory can match or beat larger contexts with far fewer tokens.
WHY IT MATTERS
MEMO unlocks evidence-level multimodal memory planning, letting agents keep precise textual anchors while leveraging visual layouts for structure under tight budgets.
Builders can now design long-horizon agents that maintain effective working memory across thousands of steps without saturating context windows or sacrificing fine-grained semantic fidelity.
Jingbiao Mei, Jinghong Chen et al.
arXiv 2026 · 2026
ATM-Bench structurally evaluates long-term multimodal personal memory using Memory Ingestion, Retrieval, and Answer Generation with Schema-Guided Memory and Descriptive Memory variants. On ATM-Bench-Hard, Oracle with SGM reaches 47.3% QS while the best full system stays under 20% accuracy, revealing a large gap.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Answers use this explainer on Memory Papers.
Checking…