AuthorsZhe Yang, Shulin Tian, Kairui Hu et al.
TL;DR
HippoCamp benchmarks multimodal personal-computer agents with a hierarchical trajectory annotation schema, revealing only 48.3% profiling accuracy even for ChatGPT Agent Mode.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
HippoCamp shows that even the strongest commercial system, ChatGPT Agent Mode, achieves only 48.3% profiling accuracy in realistic personal file systems.
These failures mean contextual agents cannot reliably perform personalized memory, cross-modal reasoning, or long-horizon user profiling on real device-scale environments.
HOW IT WORKS
HippoCamp organizes supervision into Atomic Unit Annotations, Structured Trajectory Annotations, and Profiling QA Annotations plus Factual Retention QA Annotations with capability tags for search, perception, and reasoning.
Think of HippoCamp like a layered file cabinet: atomic snippets are folders, structured trajectories are labeled binders, and profiling questions are executive summaries over a lifetime of documents.
This hierarchy lets HippoCamp diagnose where agents fail—search, multimodal perception, or reasoning—far beyond what a flat context window or simple QA benchmark can reveal.
DIAGRAM
This diagram shows how a HippoCamp agent should search, perceive, and reason over a profile-local file system to answer a single benchmark query.
DIAGRAM
This diagram shows HippoCamp’s pipeline from real user data and interviews to archetypal profiles, QA trajectories, and unified evaluation.
PROCESS
Benchmark Scale
HippoCamp first instantiates device-scale profiles with 42.4 GB and over 2K files across five modalities, defining the search space for all trajectories.
Structured Trajectory Annotations
HippoCamp converts each question into a structured trajectory with step-wise rationale, localized evidence, and 46.1K annotation units for fine-grained diagnosis.
Factual Retention QA Annotations
HippoCamp builds 521 factual retention questions that require agents to retrieve and reason over an average of 4 files and 2 modalities per question.
Profiling QA Annotations
HippoCamp adds 60 profiling questions that aggregate around 15 files and 3 modalities per query to test long-horizon personalized memory.
KEY CONTRIBUTIONS
Realistic personal computing environments
HippoCamp constructs three archetypal profiles—Bei, Adam, and Victoria—covering 42.4 GB and over 2K files with deep folder hierarchies and heterogeneous file types.
Device-scale corpus with dense supervision
HippoCamp provides 581 QA pairs with 46.1K structured annotation units, including 4.6K step-wise rationales and 3.2K localized evidence annotations linked to atomic units.
Comprehensive agent capability evaluation
HippoCamp evaluates search, perception, and reasoning via capability tags, revealing multimodal perception and evidence grounding as primary bottlenecks for current agents.
RESULTS
Profiling Accuracy
48.3%
+22.5 over Standard RAG
Factual Retention Accuracy
62.8%
+32.8 over Standard RAG
Profiling F1
21.0
+2.6 over Standard RAG
Factual Retention F1
35.3
+5.3 over Standard RAG
On the HippoCamp benchmark, which tests multimodal search, perception, and reasoning over personal file systems, ChatGPT Agent Mode reaches 48.3% profiling accuracy and 62.8% factual retention accuracy. These results show that even the strongest evaluated agent leaves a large gap to robust personalized multimodal memory.
BENCHMARK
On the HippoCamp benchmark, which tests multimodal search, perception, and reasoning over personal file systems, ChatGPT Agent Mode reaches 48.3% profiling accuracy and 62.8% factual retention accuracy. These results show that even the strongest evaluated agent leaves a large gap to robust personalized multimodal memory.
BENCHMARK
Overall factual retention accuracy (%) on HippoCamp for different methods.
KEY INSIGHT
HippoCamp shows that profiling accuracy collapses to just 5.0% for Search-R1, despite strong search F1 scores above 34% on profiling.
This is surprising because one might expect better retrieval to guarantee better answers, but HippoCamp reveals that post-retrieval perception and reasoning dominate failures.
WHY IT MATTERS
HippoCamp unlocks a way to stress-test agents on realistic, multimodal, device-scale personalized memory with interpretable, capability-wise diagnostics.
Builders can now design and benchmark file-system agents that explicitly tackle multimodal grounding, entity disambiguation, and verification, rather than optimizing only for retrieval or context length.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Shannan Yan, Jingchen Ni et al.
· 2026
AdaMem organizes dialogue history into Working Memory, Episodic Memory, Persona Memory, and Graph Memory coordinated by a Memory Agent, Research Agent, and Working Agent. On LoCoMo with GPT-4.1-mini, AdaMem achieves 44.65 F1 overall, beating the best baseline LangMem at 41.76 F1 by +2.89.
Answers use this explainer on Memory Papers.
Checking…