Skip to content

MemoryVAM: Integrating Memory into Video Action Model for Robot Manipulation

Session: Speaker(s): Paper: arXiv 2606.20679 (HTML) by Yuxin Jiang, Chang Yu, Yunuo Chen, Xiang Feng, Yin Yang, Nishank Gite, and Chenfanfu Jiang (UCLA, UCSD, Utah, Nirvana Robotics), June 2026 Project page: MemoryVAM.github.io. No code link was found. Name check: This is not MemoryWAM: Efficient World Action Modeling with Persistent Memory (arXiv 2606.20562, Sizhe Yang et al.), a separate paper with a similar name.

Paper summary (from the arXiv HTML and abstract)

  • Problem: Video-world-model policies such as VPP and Cosmos policies only see a short window. Long tasks then become non-Markovian. For example, in "pick up the red block three times," the frames from each repetition look the same.
  • Key claim: Memory should be supervised by the video generator, not only by the action head. Then memory shapes what the model imagines as well as what it does.

Method

  • History: Each frame gets one CLIP ViT-B/32 embedding (512-d), cached. History is capped at 600 frames at evaluation and 400 in training.
  • Recap Compressor: A Perceiver-style resampler (4 layers, 8 heads) that compresses the history into K memory tokens: K=16, d=384 (about 4.7M parameters) for VPP, and K=32, d=1024 (about 32M) for Cosmos. An order-aware positional encoding is used; shuffling the history hurts timing.
  • Two injection pathways:
  • Into the video backbone:
    • Cosmos DiT: the memory tokens are appended to the text conditioning.
    • SVD UNet: a parallel memory cross-attention branch is added.
  • Into the action decoder:
    • VPP: the memory tokens are prepended to the denoised video tokens.
    • Cosmos: per-layer projections are added to the action cross-attention.
  • Cue Gate: A small classifier that fuses backbone features, pooled memory, and language into a completion score, which tells the robot when to stop. It is trained with binary cross-entropy where only the last 10 frames of each episode count as "done," so no per-frame progress labels are needed.
  • Actions: A diffusion action head with 10 DDIM steps and 8-step action chunks. Video is predicted in 16-frame chunks at 256 px.
  • Losses: The video loss, plus a delta-reconstruction loss (weight 1.0) that predicts the change between consecutive CLIP embeddings from memory, plus the Cue Gate loss (weight 0.1).
  • Training stages:
  • Train the backbone and the memory. To avoid memory-token collapse, the memoryless backbone is first trained for 270K steps before memory is added, for 500K steps in all.
  • Freeze the backbone and train the action policy.

Results

LIBERO-Mem (10 memory tasks, avg success) %
VPP, no memory 5.0
VPP, Pathway 1 only / Pathway 2 only 28.5 / 7.5
MemoryVAM (UNet/VPP) 42.5
Cosmos-RF, no memory 3.0
MemoryVAM (DiT/Cosmos) 40.5
π0 / MemoryVLA / Gated Memory Policy / Embodied-SlotSSM (uses an oracle) 5.0 / 9.5 / 19.5 / 15.0
  • Both pathways are needed: Memory in the video model does most of the work (28.5%), memory in the action decoder alone barely helps (7.5%), and together they reach 42.5%.
  • Real robot (PiPER arms with an overhead D415 camera, Cosmos version):
Task Cosmos-RF (no memory) MemoryVAM
Toy Pickup 1× 0 95
Toy Pickup 3× 0 100
Toy Pickup 5× 0 40
Toy Pickup avg ("counting") 0 78.3
Place and Return ("spatial recall") 0 80.0
Shell Game ("sequential tracking") 5 75.0
  • Video quality: Memory also improves the generated video. FVD drops from 43.4 to 18.3 for VPP, and LPIPS drops from 0.167 to 0.115. For Cosmos, one long-horizon FVD setting gets slightly worse.
  • Delta-reconstruction loss: At a 0.9 gate threshold, it cuts the stop-timing error from 62.0 to 28.8 frames. The motion ratio after the task is done drops from 1.11 with no memory to 0.46.
  • Not reported: Inference speed, the number of training demonstrations, and the number of real-robot trials.

Limitations (stated by the authors)

  • Memory cost grows with episode length, so very long deployments would need truncation or a recurrent state.
  • It has only been tested at LIBERO data scale and on two backbone families.