AuthorsElizaveta Shevtsova, Inna Glebkina, Mark Baushenko et al.
TL;DR
RUMBA uses a multi-axis memory taxonomy and unified evaluation pipeline to show up to 22.96-point accuracy drops on multi-session questions for long-context LLMs.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Existing memory benchmarks are English only and rely on flat taxonomies, missing interactions between long-range context, temporal information, and reasoning.
This leaves Russian conversational systems without diagnostic tools for multi-session memory, temporal reasoning, and forgetting, hindering reliable long-horizon assistants.
HOW IT WORKS
RUMBA combines a Semantic axis, Quantitative axis, and Axis of temporality over 1,543 questions tied to 85 long, timestamped dialogues.
Think of RUMBA like a card catalog for user memories: each question is filed by meaning, session span, and temporal behavior instead of a single flat label.
This multi-axis design lets RUMBA expose where long-context models and memory agents fail on multi-session, temporal, or forgetting tasks that plain context windows cannot disentangle.
DIAGRAM
This diagram shows how RUMBA ingests dialogues into memory systems and then evaluates answers with LLM-as-Judge scoring.
DIAGRAM
This diagram shows how RUMBA runs full-context and Agent/RAG baselines over Russian and English splits.
PROCESS
Taxonomy design
RUMBA defines the Semantic axis, Quantitative axis, and Axis of temporality, creating 17 base types across extraction, reasoning, and abstention.
Dataset creation
RUMBA collects 85 Russian dialogues with 1,543 questions, timestamped sessions, and human-written user turns plus GigaChat assistant responses.
English version translation
RUMBA generates an aligned English split via LLM translation, then validates 10% of dialogues to reach a 0.88 POLLUX-based translation quality score.
Add and eval pipeline
RUMBA runs the add stage to populate memory systems, then the eval stage with top k retrieval and LLM-as-Judge scoring for Russian and English.
KEY CONTRIBUTIONS
RUMBA benchmark release
RUMBA introduces 85 dialogues and 1,543 question–answer pairs in Russian with an aligned English subset, enabling cross-lingual memory diagnostics.
Fine-grained memory taxonomy
RUMBA’s Semantic axis, Quantitative axis, and Axis of temporality yield 17 base types and temporal expression tags for explicit versus implicit time.
Unified evaluation pipeline
RUMBA standardizes add and eval stages for both Agent/RAG systems and full-context models, using POLLUX and DeepSeek-R1 as judges.
RESULTS
LLM-as-Judge accuracy RU
83.60%
+23.52 over llama-4-maverick
LLM-as-Judge accuracy EN
83.99%
+33.18 over llama-4-maverick
Agent RAG mean RU
55.80%
-4.28 vs gpt-4.1-mini full context
Single minus multi RU FC
22.96 points
single-session advantage over multi-session questions
RUMBA evaluates memory systems on 1,543 questions over Russian and English long dialogues, measuring LLM-as-Judge accuracy. The main result shows gpt-5.4 reaching 83.60% in Russian and 83.99% in English, while multi-session questions are up to 22.96 points harder than single-session ones.
BENCHMARK
RUMBA evaluates memory systems on 1,543 questions over Russian and English long dialogues, measuring LLM-as-Judge accuracy. The main result shows gpt-5.4 reaching 83.60% in Russian and 83.99% in English, while multi-session questions are up to 22.96 points harder than single-session ones.
BENCHMARK
LLM-as-Judge accuracy on the Russian split of RUMBA for long-context models.
KEY INSIGHT
RUMBA shows that even full-context models like gpt-5.4 lose 22.96 accuracy points between single-session and multi-session Russian questions.
This is surprising because long-context windows are expected to handle extended histories, yet RUMBA reveals multi-session integration as a major unresolved challenge.
WHY IT MATTERS
RUMBA unlocks precise diagnosis of where conversational agents fail on long-term memory, temporal reasoning, and forgetting in Russian and English.
Builders can now benchmark memory agents and long-context LLMs slice-by-slice, targeting weaknesses in multi-session and implicit temporal questions that were previously invisible.
Guilin Zhang, Wei Jiang et al.
· 2026
A-MAC scores candidate memories using Utility, Confidence, Novelty, Recency, and Type Prior combined by a learned linear admission policy with Algorithm 1 A-MAC Memory Admission. On the LoCoMo benchmark, A-MAC achieves F1 0.583 and 2644 ms latency, improving F1 by 0.042 and reducing latency by 1187 ms compared to A-mem.
Robbyant Team, Zelin Gao et al.
arXiv 2026 · 2026
LingBot-World combines a Data Engine, Fundamental World Model, Action-Conditioned World Model, and Post-Training causal adaptation to turn a 28B-parameter video generator into a real-time interactive world simulator. On the VBench benchmark, LingBot-World achieves a dynamic degree of 0.8857 versus 0.7612 for Yume-1.5, while also improving imaging quality to 0.6683.
Manoj Madushanka Perera, Adnan Mahmood et al.
· 2026
AgenticAI-DialogGen chains ChatPreprocessor, KnowledgeExtractor, TopicAnalyzer, KnowledgeGraphBuilder, PersonaGenerator, DuelingChat Agent, ConversationValidator, ConversationRefiner, QAGeneration, and PostProcessing to turn raw multi-session chats into topic-guided, persona-grounded conversations with explicit short- and long-term memories. On the TGC / KG memory QA benchmark, Mistral-7B fine-tuned within AgenticAI-DialogGen achieves 87.36 F1, compared to GPT-4’s 83.77 F1 in a zero-shot setting on the same task.
Answers use this explainer on Memory Papers.
Checking…