Memory Papers
Agent Memory

$Σ$-Mem: An Online Reliability Memory for LLM-based Multi-Agent Systems

AuthorsPeilin Feng, Suorong Yang, Soujanya Poria

arXiv 20262026
arXivPDF

TL;DR

Σ-Mem uses symmetric reliability matrices with Weyl-bounded updates to steer multi-agent trust, boosting Qwen3-0.6B CF@90 accuracy from 46.22% to 71.10%.

SharePost on XLinkedIn

Read our summary here, or open the publisher PDF on the next tab.

THE PROBLEM

Multi-agent LLMs mis-trust peers under counterfactual shifts (Qwen3-0.6B drops to 46.22% at CF@90)

Existing multi-agent systems rely on content memory and ignore which peers are trustworthy under changing conditions, leading to brittle coordination. Qwen3-0.6B accuracy falls from 61.01% at CF@0 to 46.22% at CF@90.

When the reliable peer changes, the central coordinator keeps favoring the wrong peer, causing trust errors where correct answers are rejected and incorrect ones are selected.

HOW IT WORKS

Σ-Mem — Online reliability memory with symmetric competence and relationship states

Σ-Mem introduces historical competence evidence matrices M_p, a peer relationship matrix G, memory readout vectors r_{p,t}, and residual steering δ_{p,t} to encode peer reliability. Σ-Mem updates these states online from external correctness feedback using bounded, decayed rank one updates.

You can think of Σ-Mem as a reliability RAM sitting beside the LLM, where M_p stores per peer trust tracks and G stores a graph of who tends to succeed or fail together. Residual steering then injects this reliability RAM into the upper transformer layers like a bias card that tilts attention toward historically competent peers.

This design lets Σ-Mem accumulate long horizon reliability patterns and steer peer selection, routing, and voting in ways that a plain context window or content memory cannot express.

DIAGRAM

Event-time flow of Σ-Mem reliability steering and memory updates

This diagram shows how Σ-Mem processes a single event from competence direction encoding through residual steering, utility evaluation, and post decision updates to M_p and G.

DIAGRAM

Training and evaluation pipeline for Σ-Mem on counterfactual and OOD benchmarks

This diagram shows how Σ-Mem is trained on mixed domains and evaluated under counterfactual reliability shifts, enlarged peer pools, and OOD benchmarks.

PROCESS

How Σ-Mem Handles a Task Event in a Multi-Agent System

  1. 01

    Event Level Σ-Mem Record

    Σ-Mem encodes the task input into a competence direction φ(x_t) and updates historical competence evidence matrices M_p and the peer relationship matrix G with decayed rank one updates.

  2. 02

    Memory Readout and Residual Steering

    Σ-Mem computes memory readout r_{p,t} = M_p φ(x_t) and projects it into residual steering δ_{p,t}, injected into upper transformer layers of the central model.

  3. 03

    Utility Eval and Peer Selection

    Σ-Mem uses utility eval U_{p,t} from Yes versus No likelihoods, combines them with G, and selects the peer with the highest posterior correctness expectation.

  4. 04

    Direct Readouts for Other Selection Mechanisms

    Σ-Mem computes scalar scores s_{p,t} = φ(x_t)^T M_p φ(x_t) to support M routing and M weighted voting without using the central model at decision time.

KEY CONTRIBUTIONS

Key Contributions

  • 01

    Online reliability memory for MAS

    Σ-Mem records historical competence evidence in symmetric matrices M_p and peer relationship evidence in G, updating both online from correctness feedback over 2,963 mixed domain events.

  • 02

    Stable symmetric memory with Weyl bounds

    Σ-Mem constrains M_p and G as real symmetric matrices with decayed, bounded updates so that each event perturbs eigenvalues by at most ∥E∥_2, ensuring stable long horizon reliability accumulation.

  • 03

    Reusable reliability memory across mechanisms

    Σ-Mem shows that a single frozen reliability state supports steered peer selection, M routing, and M weighted voting, with M Vote surpassing majority voting and the best fixed peer over six OOD benchmarks.

RESULTS

By the Numbers

CF@90 Acc Qwen3-0.6B

71.10%

+24.88 over Qwen3-0.6B base at 46.22%

CF@0 Acc Qwen3.5-9B

71.84%

+5.36 over Qwen3.5-9B base at 66.48%

BBH Acc Qwen3-4B

28.66%

+8.28 over Qwen3-4B base at 20.38%

OOD Overall Acc Σ-Mem M Vote

60.99%

+1.87 over majority voting at 59.12%

The mixed counterfactual benchmark tests adaptation under reliability shifts, while BBH and six OOD datasets probe complex reasoning and generalization. These results show that Σ-Mem substantially raises counterfactual robustness and transfers reliability memory to unseen domains and selection mechanisms.

BENCHMARK

By the Numbers

The mixed counterfactual benchmark tests adaptation under reliability shifts, while BBH and six OOD datasets probe complex reasoning and generalization. These results show that Σ-Mem substantially raises counterfactual robustness and transfers reliability memory to unseen domains and selection mechanisms.

BENCHMARK

Counter Factual Attack results for Qwen3-0.6B at CF@90

Acc on mixed counterfactual benchmark at CF@90 for Qwen3-0.6B with and without Σ-Mem.

KEY INSIGHT

The Counterintuitive Finding

At CF@50, Σ-Mem can reduce accuracy compared to the base model, for example Qwen3-4B drops from 60.74% to 57.17% with Σ-Mem under four peers. This happens even though Σ-Mem improves the same model at CF@0, CF@70, and CF@90.

This is surprising because a better reliability memory might be expected to always help, but Σ-Mem faithfully accumulates ambiguous evidence at CF@50, revealing that history alone can mislead when the stream is intentionally contradictory.

WHY IT MATTERS

What this unlocks for the field

Σ-Mem unlocks a reusable, task conditioned reliability memory that can steer trust in multi agent LLM systems without retraining base models. Builders can now plug Σ-Mem into swarms or mixture of agents to get adaptive peer selection, routing, and voting that improve as more correctness feedback accumulates.

~14 min read← Back to papers

Related papers

Agent Memory

A Benchmark and Diagnostic Study of Epistemic Admission in Shared Agent Memory

Xiaoyang Li, Yiqi Wang et al.

arXiv 2026 · 2026

Correlated Promotion Benchmark (CPB) combines CPB-Static, CPB-Live, a gold admission rule, lineage collapse, and a governance rule to stress-test epistemic admission in shared agent memory. On CPB-Live, the governance rule keeps damage shares between 0.112 and 0.152 and false adoption between 0.06 and 0.09, while majority vote and LLM judges often match share-all’s false adoption.

arXiv:2609.30813Read explainer
BenchmarkAgent Memory

Active Context Compression: Autonomous Memory Management in LLM Agents

Nikhil Verma

· 2026

Focus Agent adds start_focus, complete_focus, a persistent Knowledge block, and an optimized Persistent Bash plus String-Replace Editor scaffold to actively compress context during long software-engineering tasks. On five hard SWE-bench Lite instances against a Baseline ReAct agent, Focus Agent achieves 22.7% token reduction (14.9M → 11.5M) while matching 3/5 = 60% task success.

arXiv:2601.07190Read explainer
Agent Memory

ActMem: Bridging the Gap Between Memory Retrieval and Reasoning in LLM Agents

Xiaohui Zhang, Zequn Sun et al.

· 2026

ActMem transforms dialogue history into atomic facts via Memory Fact Extraction, groups them with Fact Clustering, links them through a Memory KG Construction module, and uses Counterfactual-based Retrieval and Reasoning for action-aware answers. On ActMemEval, ActMem reaches 76.52% QA accuracy with DeepSeek-V3, beating LightMem’s 63.97% by 12.55 points and NaiveRAG’s 61.54%.

arXiv:2603.00026Read explainer

Questions about this paper?

Paper: $Σ$-Mem: An Online Reliability Memory for LLM-based Multi-Agent Systems

Answers use this explainer on Memory Papers.

Checking…