AuthorsDingli Liang, Yiqiao Xie, Yukai Huang et al.
TL;DR
CapMem uses caption-based episodic memory plus a retrieve-and-verify harness to beat direct VideoQA by up to 15.9 points on long egocentric videos.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
CapMem targets VLMs that face bounded frame budgets, growing visual-token costs, and long-context retrieval failures on continuous egocentric video.
These limits cause wearable assistants to miss object locations and specific actions over hours of video, breaking episodic recall for navigation, procedures, and object finding.
HOW IT WORKS
CapMem combines CleanQA, Direct VideoQA, CaptionQA, a frame-aligned control, and a retrieve-and-verify harness to test caption-based episodic memory.
Think of CapMem as turning raw egocentric video into a searchable text log plus index, then selectively reopening the original “tape” only when captions are uncertain.
This design lets CapMem probe how caption memory plus targeted visual verification supports long-horizon reasoning beyond what a plain context window of frames can hold.
DIAGRAM
This diagram shows how CapMem runs CaptionQA and the retrieve-and-verify harness at query time on egocentric videos.
DIAGRAM
This diagram shows how CapMem gates models with CleanQA, then compares Direct VideoQA and CaptionQA under multiple controls.
PROCESS
CleanQA
CapMem first runs CleanQA so models answer questions using only text, ensuring parametric priors are near chance before further evaluation.
Direct VideoQA
CapMem then runs Direct VideoQA by feeding uniformly sampled 1 FPS frames, capped at 128 or 768 frames depending on the VLM family.
Full-coverage CaptionQA
CapMem performs full-coverage CaptionQA by captioning every 30 or 60 second window at 1 FPS and answering questions using only these timestamped captions.
Frame-aligned Control and Cross-captioning Control
CapMem finally applies frame-aligned control and cross-captioning control, matching frame counts and swapping captioners to isolate caption memory and text QA effects.
KEY CONTRIBUTIONS
CapMem benchmark for Episodic Memory Video Caption QA
CapMem introduces a human-annotated benchmark of 75 videos totaling 33.7 hours and 1,000 QA pairs across 16 scenarios, with evidence timestamps and fine-grained visual tags.
Systematic comparison of CaptionQA and Direct VideoQA
CapMem shows that full-coverage CaptionQA surpasses Direct VideoQA for 8 of 12 models overall and for 10 of 12 models on videos longer than 20 minutes.
Retrieve-and-verify harness for caption memory
CapMem adds a caption-guided retrieve-and-verify harness that improves accuracy by up to 5.3 points and reduces repeated visual-frame processing by 7.1×.
RESULTS
Overall accuracy
49.3%
+4.3 over Qwen3.5-27B VideoQA
Overall accuracy
52.0%
+14.7 over GPT-5.2 VideoQA
Mean gap long videos
+3.22
frame-aligned CaptionQA30s minus VideoQA across six Qwen models
Harness gain
+5.3
retrieve-and-verify over CaptionQA60s for Qwen3.5-2B
CapMem evaluates CaptionQA and Direct VideoQA on 1,000 questions over 75 egocentric videos, stressing long-horizon episodic memory. The gains show CapMem’s caption memory plus harness can recover information that Direct VideoQA misses under current frame budgets.
BENCHMARK
CapMem evaluates CaptionQA and Direct VideoQA on 1,000 questions over 75 egocentric videos, stressing long-horizon episodic memory. The gains show CapMem’s caption memory plus harness can recover information that Direct VideoQA misses under current frame budgets.
BENCHMARK
Overall accuracy (%) on CapMem for CleanQA, CaptionQA30s, CaptionQA60s, and Direct VideoQA.
KEY INSIGHT
CapMem shows that 30-second CaptionQA on long videos can exceed Direct VideoQA by up to +15.9 points for GPT-5.2.
This is surprising because many expect raw visual input to be strictly superior, yet CapMem reveals that well-structured caption memory can be more usable than sparse frames under tight budgets.
WHY IT MATTERS
CapMem enables rigorous testing of caption-based episodic memory, showing when textual memory plus targeted visual checks beats direct frame reasoning.
With CapMem, builders can design wearable assistants that caption once, reuse memory across many queries, and still recover fine-grained details using retrieve-and-verify instead of brute-force video processing.
Jingbiao Mei, Jinghong Chen et al.
arXiv 2026 · 2026
ATM-Bench structurally evaluates long-term multimodal personal memory using Memory Ingestion, Retrieval, and Answer Generation with Schema-Guided Memory and Descriptive Memory variants. On ATM-Bench-Hard, Oracle with SGM reaches 47.3% QS while the best full system stays under 20% accuracy, revealing a large gap.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Answers use this explainer on Memory Papers.
Checking…