AuthorsGuangyu Robert Yang, Igor Ganichev, Xiao-Jing Wang et al.
TL;DR
COG + recurrent controller with feature and spatial attention enables near-human CLEVR reasoning at 96.8% accuracy while supporting zero-shot generalization across 44 compositional tasks.
Read our summary here, or open the publisher PDF on the next tab.
THE PROBLEM
Existing VQA systems often exploit dataset biases instead of true reasoning, and typically avoid time and memory in their design.
Without robust working memory, agents struggle on tasks requiring tracking objects across frames, limiting reliable video understanding and sequential decision making.
HOW IT WORKS
SYS_NAME centers on a Visual processing CNN, Semantic processing BiLSTM, Visual short-term memory module, and a recurrent Controller coordinating attention and gating.
You can think of SYS_NAME like a brain-inspired system where the CNN is the eye, the BiLSTM is language cortex, vSTM is parietal spatial memory, and the Controller is prefrontal cortex.
This coordination plus iterative pondering lets SYS_NAME chain multi step operations over time, something a plain context window or single pass CNN LSTM stack cannot achieve.
DIAGRAM
This diagram shows how SYS_NAME iteratively processes each image and instruction token using pondering steps, attention, and visual short term memory.
DIAGRAM
This diagram shows how SYS_NAME is trained and evaluated on COG using operator graphs, backward image generation, and sequence level supervision.
PROCESS
Task execution
During Task execution, SYS_NAME receives each image and instruction, with Visual processing and Semantic processing feeding features into the Controller at every pondering step.
Image sequence generation
In Image sequence generation, SYS_NAME is trained on COG sequences produced by operator graphs that specify object attributes, relations, and required Visual short-term memory usage.
Forward pass through the graph
A Forward pass through the graph lets SYS_NAME simulate operator compositions by iteratively updating Controller state, attention, and Visual short-term memory across frames.
Backward pass through the graph
The Backward pass through the graph is used by COG to create minimally biased inputs, and SYS_NAME learns to invert this logic via its recurrent Controller and attention mechanisms.
KEY CONTRIBUTIONS
The COG dataset
SYS_NAME introduces COG, a compositional dataset with 8 operators, 44 tasks, and more than 2 trillion possible task instances exercising working memory and visual reasoning.
Multi modal recurrent architecture
SYS_NAME defines a multi modal architecture combining Visual processing, Semantic processing, Visual short-term memory, and a Controller with feature and spatial attention for temporal reasoning.
Zero shot task generalization
SYS_NAME demonstrates zero shot generalization where networks trained on 43 tasks reach 85.4% average accuracy on the excluded task, far above the 26.7% chance level.
RESULTS
Overall
96.8%
+1.3 over CNN+LSTM+RN
Count
91.7%
+1.6 over CNN+LSTM+RN
Exist
99.0%
+1.2 over CNN+LSTM+RN
Compare Numbers
95.5%
+1.9 over CNN+LSTM+RN
These metrics come from the CLEVR test set, which probes compositional visual reasoning. MAIN_RESULT shows SYS_NAME matches specialized CLEVR models while also supporting working memory for COG.
BENCHMARK
These metrics come from the CLEVR test set, which probes compositional visual reasoning. MAIN_RESULT shows SYS_NAME matches specialized CLEVR models while also supporting working memory for COG.
BENCHMARK
Overall accuracy on CLEVR.
KEY INSIGHT
SYS_NAME relies heavily on feature attention for CLEVR but primarily on spatial attention for COG, even though both are visual reasoning benchmarks.
This is surprising because one might expect the same attention mechanism to dominate across datasets, but SYS_NAME instead adapts mechanisms to object combinatorics and temporal demands.
WHY IT MATTERS
SYS_NAME shows that a single architecture can handle static CLEVR style reasoning and temporally extended COG tasks with explicit working memory.
Builders can now prototype agents that follow language instructions over image sequences, track objects through time, and even generalize to unseen tasks without retraining.
Nikhil Verma
· 2026
Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.
Xingyu Lyu, Jianfeng He et al.
· 2026
ADAM combines Anchor extraction, Distribution estimation, Anchor selection, and Query generation to adaptively probe agent memory via an auxiliary generator and entropy based selection. On the EHRAgent benchmark with Llama2-7b-chat, ADAM reaches EQ=77 and ASR=1.00, compared to MEXTRA’s EQ=44 and ASR=0.89.
Shannan Yan, Jingchen Ni et al.
· 2026
AdaMem organizes dialogue history into Working Memory, Episodic Memory, Persona Memory, and Graph Memory coordinated by a Memory Agent, Research Agent, and Working Agent. On LoCoMo with GPT-4.1-mini, AdaMem achieves 44.65 F1 overall, beating the best baseline LangMem at 41.76 F1 by +2.89.
Answers use this explainer on Memory Papers.
Checking…