MemoryVAM: Integrating Memory into Video Action Model for Robot Manipulation¶
Session: Speaker(s): Paper: arXiv 2606.20679 (HTML) by Yuxin Jiang, Chang Yu, Yunuo Chen, Xiang Feng, Yin Yang, Nishank Gite, and Chenfanfu Jiang (UCLA, UCSD, Utah, Nirvana Robotics), June 2026 Project page: MemoryVAM.github.io. No code link was found. Name check: This is not MemoryWAM: Efficient World Action Modeling with Persistent Memory (arXiv 2606.20562, Sizhe Yang et al.), a separate paper with a similar name.
Paper summary (from the arXiv HTML and abstract)¶
- Problem: Video-world-model policies such as VPP and Cosmos policies only see a short window. Long tasks then become non-Markovian. For example, in "pick up the red block three times," the frames from each repetition look the same.
- Key claim: Memory should be supervised by the video generator, not only by the action head. Then memory shapes what the model imagines as well as what it does.
Method
- History: Each frame gets one CLIP ViT-B/32 embedding (512-d), cached. History is capped at 600 frames at evaluation and 400 in training.
- Recap Compressor: A Perceiver-style resampler (4 layers, 8 heads) that compresses the history into K memory tokens: K=16, d=384 (about 4.7M parameters) for VPP, and K=32, d=1024 (about 32M) for Cosmos. An order-aware positional encoding is used; shuffling the history hurts timing.
- Two injection pathways:
- Into the video backbone:
- Cosmos DiT: the memory tokens are appended to the text conditioning.
- SVD UNet: a parallel memory cross-attention branch is added.
- Into the action decoder:
- VPP: the memory tokens are prepended to the denoised video tokens.
- Cosmos: per-layer projections are added to the action cross-attention.
- Cue Gate: A small classifier that fuses backbone features, pooled memory, and language into a completion score, which tells the robot when to stop. It is trained with binary cross-entropy where only the last 10 frames of each episode count as "done," so no per-frame progress labels are needed.
- Actions: A diffusion action head with 10 DDIM steps and 8-step action chunks. Video is predicted in 16-frame chunks at 256 px.
- Losses: The video loss, plus a delta-reconstruction loss (weight 1.0) that predicts the change between consecutive CLIP embeddings from memory, plus the Cue Gate loss (weight 0.1).
- Training stages:
- Train the backbone and the memory. To avoid memory-token collapse, the memoryless backbone is first trained for 270K steps before memory is added, for 500K steps in all.
- Freeze the backbone and train the action policy.
Results
| LIBERO-Mem (10 memory tasks, avg success) | % |
|---|---|
| VPP, no memory | 5.0 |
| VPP, Pathway 1 only / Pathway 2 only | 28.5 / 7.5 |
| MemoryVAM (UNet/VPP) | 42.5 |
| Cosmos-RF, no memory | 3.0 |
| MemoryVAM (DiT/Cosmos) | 40.5 |
| π0 / MemoryVLA / Gated Memory Policy / Embodied-SlotSSM (uses an oracle) | 5.0 / 9.5 / 19.5 / 15.0 |
- Both pathways are needed: Memory in the video model does most of the work (28.5%), memory in the action decoder alone barely helps (7.5%), and together they reach 42.5%.
- Real robot (PiPER arms with an overhead D415 camera, Cosmos version):
| Task | Cosmos-RF (no memory) | MemoryVAM |
|---|---|---|
| Toy Pickup 1× | 0 | 95 |
| Toy Pickup 3× | 0 | 100 |
| Toy Pickup 5× | 0 | 40 |
| Toy Pickup avg ("counting") | 0 | 78.3 |
| Place and Return ("spatial recall") | 0 | 80.0 |
| Shell Game ("sequential tracking") | 5 | 75.0 |
- Video quality: Memory also improves the generated video. FVD drops from 43.4 to 18.3 for VPP, and LPIPS drops from 0.167 to 0.115. For Cosmos, one long-horizon FVD setting gets slightly worse.
- Delta-reconstruction loss: At a 0.9 gate threshold, it cuts the stop-timing error from 62.0 to 28.8 frames. The motion ratio after the task is done drops from 1.11 with no memory to 0.46.
- Not reported: Inference speed, the number of training demonstrations, and the number of real-robot trials.
Limitations (stated by the authors)
- Memory cost grows with episode length, so very long deployments would need truncation or a recurrent state.
- It has only been tested at LIBERO data scale and on two backbone families.