Muon Outperforms Adam in Tail-End Associative Memory Learning

THE PROBLEM

Optimizers Struggle With Tail-End Learning in Heavy-Tailed Data

HOW IT WORKS

Associative Memory View of Muon in Transformers

DIAGRAM

Update Flow for Associative Memory Gradients Under Muon

This diagram shows how Muon Outperforms Adam in Tail-End Associative Memory Learning transforms associative memory gradients into isotropic matrix updates via SVD based normalization.

DIAGRAM

Ablation Design for Applying Muon to Transformer Blocks

This diagram shows how Muon Outperforms Adam in Tail-End Associative Memory Learning structures independent-block and combined-configuration ablations over QK, VO, and FFN parameters.

PROCESS

How Muon Outperforms Adam in Tail-End Associative Memory Learning Handles Heavy-Tailed Next-Token Prediction

1. 01

   Associative memory identification

   Muon Outperforms Adam in Tail-End Associative Memory Learning first identifies VO attention weights and FFN matrices as associative memory parameters storing subject relation object facts.

2. 02

   Independent blocks ablation

   Muon Outperforms Adam in Tail-End Associative Memory Learning applies Muon separately to QK, VO, and FFN blocks in a 160M NanoGPT to measure validation loss impacts.

3. 03

   Combined configurations on VO and FFN

   Muon Outperforms Adam in Tail-End Associative Memory Learning then jointly applies Muon to VO and FFN while keeping QK on Adam to approximate full-Muon performance.

4. 04

   Heavy tailed knowledge evaluation

   Muon Outperforms Adam in Tail-End Associative Memory Learning finally evaluates First Token Accuracy on a power law biographical QA dataset to compare head and tail class learning.

KEY CONTRIBUTIONS

Key Contributions

• 01

  Associative memory centric view of Muon

  Muon Outperforms Adam in Tail-End Associative Memory Learning shows that VO attention weights and FFN blocks are the main beneficiaries of Muon, with VO+FFN nearly recovering full-Muon validation loss.

• 02

  Spectral isotropy analysis of Muon

  Muon Outperforms Adam in Tail-End Associative Memory Learning demonstrates that Muon consistently yields higher normalized SVD entropy and effective rank than Adam for VO and Wout across seeds.

• 03

  Theoretical balanced learning guarantees

  Muon Outperforms Adam in Tail-End Associative Memory Learning proves that Muon maintains ϱϵ_Muon ≥ 1 − ϵ(1 + O(log K K−1)) under class imbalance, while GD and Adam can have ϱϵ scaling like O(ϵ−r(α,β)Kr(α,β)−1).

RESULTS

By the Numbers

On the FineWeb dataset with a 160M NanoGPT, Muon Outperforms Adam in Tail-End Associative Memory Learning achieves validation loss 3.5654 versus 3.9242 for All Adam at 10000 steps. This shows that applying Muon to associative memories yields faster convergence and that VO+FFN with Muon (3.5858) nearly matches full-Muon performance.

BENCHMARK

By the Numbers

On the FineWeb dataset with a 160M NanoGPT, Muon Outperforms Adam in Tail-End Associative Memory Learning achieves validation loss 3.5654 versus 3.9242 for All Adam at 10000 steps. This shows that applying Muon to associative memories yields faster convergence and that VO+FFN with Muon (3.5858) nearly matches full-Muon performance.

BENCHMARK

Validation loss at 10000 training steps on 160M NanoGPT (non-gated FFN)

Validation loss after 10000 steps on FineWeb for different optimizer configurations on attention and FFN blocks.

KEY INSIGHT

WHY IT MATTERS

What this unlocks for the field