AuthorsYuhao Chen, Yi Xu, Xinyun Ding et al.
TL;DR
VehicleMemBench couples an executable in-vehicle simulation with event-driven multi-user preference histories to reveal up to 25.8 ESM loss from autonomous memory vs gold memory.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
VehicleMemBench shows that Gemini-3-Pro-Preview drops from 90.60 to 64.80 Exact State Match when moving from Gold Memory to Recursive Summarization, a 25.8-point loss.
In realistic multi-user driving scenarios with over 80 historical memory events and 23 tool modules, in-vehicle agents mis-handle evolving preferences, leading to incorrect vehicle states and unsafe decisions.
HOW IT WORKS
VehicleMemBench centers on Persona Group Generation, Event Chain Construction, Temporal Interleaving, Conversation Generation, and Question and Answer Generation to synthesize rich multi-user histories.
You can think of VehicleMemBench like a driving simulator wired to a long-term diary: personas define habits, event chains write the diary, and the simulation executes every remembered preference.
This design lets VehicleMemBench test whether agents can transform long, interleaved histories into precise tool calls and final states, something a plain context window with static QA cannot approximate.
DIAGRAM
This diagram shows how VehicleMemBench runs an agent online: from dialogue history ingestion into memory through tool calls to final state comparison.
DIAGRAM
This diagram shows the full VehicleMemBench pipeline from persona initialization through event chains to executable evaluation.
PROCESS
Persona Group Generation
VehicleMemBench samples from Persona-Hub and refines user profiles, creating structured personas that drive Event Chain Construction and realistic multi-user behavior.
Event Chain Construction
VehicleMemBench builds multiple event chains per group, encoding preference conflict, coreference resolution, conditional constraint, state shift, and error correction dynamics.
Temporal Interleaving
VehicleMemBench assigns timestamps and interleaves all event chains into a single timeline, ensuring concurrent preference threads and realistic long-context histories.
Conversation Generation
VehicleMemBench converts interleaved events into dialogues and then runs Question and Answer Generation plus in-vehicle execution to evaluate state-based task completion.
KEY CONTRIBUTIONS
Executable benchmark for multi-user long-term memory in-vehicles
VehicleMemBench integrates Event Chain Construction and an In-Vehicle Simulation Environment with 23 tool modules and 111 APIs, enabling objective state-based evaluation of memory and tool use.
Event-driven data construction with structured preference evolution
VehicleMemBench uses Persona Group Generation, Event Chain Construction, and Temporal Interleaving to create over 80 historical memory events per instance across 50 multi-user scenarios.
Large-scale systematic evaluation
VehicleMemBench evaluates seven model families and five memory systems, revealing up to 25.8 Exact State Match loss when moving from Gold Memory to autonomous memory.
RESULTS
Exact State Match
90.60
+35.20 over GLM-4.7-Flash under Gold Memory
F-F1
96.13
field-level F1 for Gemini-3-Pro-Preview with Gold Memory
V-F1
94.81
value-level F1 for Gemini-3-Pro-Preview with Gold Memory
Calls
2.74
average tool calls per task for Gemini-3-Pro-Preview with Gold Memory
VehicleMemBench evaluates multi-user long-term memory and tool use in an executable in-vehicle environment. The 90.60 Exact State Match for Gemini-3-Pro-Preview with Gold Memory, versus 64.80 with Recursive Summarization, proves that memory construction and retrieval dominate failure modes over tool execution.
BENCHMARK
VehicleMemBench evaluates multi-user long-term memory and tool use in an executable in-vehicle environment. The 90.60 Exact State Match for Gemini-3-Pro-Preview with Gold Memory, versus 64.80 with Recursive Summarization, proves that memory construction and retrieval dominate failure modes over tool execution.
BENCHMARK
Exact State Match on VehicleMemBench with Gold Memory, comparing Gemini-3-Pro-Preview to several backbone models.
BENCHMARK
Exact State Match and Mem Token for Gemini-3-Pro-Preview across memory systems on VehicleMemBench.
KEY INSIGHT
VehicleMemBench shows that Recursive Summarization can match or beat sophisticated systems, with Gemini-3-Pro-Preview scoring 64.80 ESM versus 59.73 for MemOs.
This is surprising because many expect complex general-purpose memory systems to dominate simple summarization, yet VehicleMemBench reveals domain-specific, state-coupled memory is more critical.
WHY IT MATTERS
VehicleMemBench gives researchers an executable, state-based way to stress-test multi-user long-term memory and tool use in realistic in-vehicle scenarios.
Builders can now design and debug memory systems that survive 80 plus interleaved events, dynamic preference shifts, and 23 tool modules, instead of optimizing only static QA benchmarks.
Peilin Feng, Suorong Yang, Soujanya Poria
arXiv 2026 · 2026
Σ-Mem maintains historical competence evidence matrices M_p, a peer relationship matrix G, a memory readout r_{p,t}, and residual steering δ_{p,t} to guide multi-agent trust decisions. On the mixed counterfactual benchmark, Σ-Mem raises Qwen3-0.6B accuracy at CF@90 from 46.22% with the base model to 71.10% with Σ w/ G.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xiaohui Zhang, Zequn Sun et al.
· 2026
ActMem transforms dialogue history into atomic facts via Memory Fact Extraction, groups them with Fact Clustering, links them through a Memory KG Construction module, and uses Counterfactual-based Retrieval and Reasoning for action-aware answers. On ActMemEval, ActMem reaches 76.52% QA accuracy with DeepSeek-V3, beating LightMem’s 63.97% by 12.55 points and NaiveRAG’s 61.54%.
Answers use this explainer on Memory Papers.
Checking…