Memory and long context¶
Cross-day theme. Facts come from the linked day and paper notes. Sentences marked "my reading" are interpretation.
Most robot foundation models act on the current frame. That's fine for short, repetitive skills, but tracking what happened, or finishing a 15-minute job, needs memory. The week's papers stored memory in very different places, with very different costs.
The approaches, from implicit to explicit¶
| Approach | Day | Where memory lives | Horizon | Cost per step | Result |
|---|---|---|---|---|---|
| TEMPO | 1 | motion and command history tokens in the VLM prefix | short | constant (+2M params) | 47.5% → 76.0% average success |
| π0.7 MEM | 1 | compressed video history | about 6 s (6 frames per camera, 1 s stride) | small | "insufficient" for long tracking (Day 4 talk) |
| RoboTTT | 1 | fast weights updated by test-time training | 8K steps, about 5 min at 30 Hz | constant | 79% vs 42% task completion |
| MemoryVAM | 1 | episodic memory in a video action model | episode | grows with the episode | LIBERO-Mem 5% → 42.5% |
| Mem-World | 1 | 4D surfel map, retrieved by future actions | episode | retrieval: top-K frames | none from the talk |
| π0.6-MEM (long-context VLA) | 4 | long-term text summary plus short-term video | 15+ min tasks | text stays small | about 78% vs 42% average task progress (read off a chart) |
| FARM | 4 | explicit 3D graph of objects and relations | whole building, 1,600–15,000 m² | 5–10 Hz online mapping | +164% Recall@5 over prior methods (unverified) |
Memory inside the policy¶
The Day 4 long-context talk named two problems:
- Spurious correlations between past observations and future actions. Sometimes adding memory makes the policy worse.
- Compute grows with context, for both training and inference.
Its fixes: - For the training signal: predict subtasks and memory narrations. - For speed: cache and reuse past computation, decode the narration in parallel, and group actions.
Memory as retrieval or a map¶
The Sim2Real2Field talk (Day 4) put it simply: how a robot builds its memory, and how it queries it, both matter.
Takeaways¶
- Two tiers keep coming up: dense visual memory for the last few seconds or minutes, and sparse semantic memory (text or an object graph) for anything longer.
- Constant cost per step is the dividing line for real-time control. TEMPO, RoboTTT, and text summaries have it; a growing video context doesn't.
- My reading: The text-memory result (π0.6-MEM) and FARM's object graph are the same move at different scales. Both compress history into symbols the model can query.
Papers tagged memory or temporal-context¶
- FARM: Real-time object-level relational 3D memory answers relational language queries; +164% Recall@5 over prior methods (unverified)
- MemoryVAM: LIBERO-Mem from 5% to 42.5%; 75–80% on real memory tasks
- Mem-World: Future actions pick which history a multi-view world model retrieves, via a wrist-view 4D surfel memory
- MemoryVAM: LIBERO-Mem from 5% to 42.5%; 75–80% on real memory tasks
- π0.7: Metadata turns low-quality data from harmful to helpful; zero-shot UR5e shirt folding at teleoperator level
- RoboTTT: 8K-step context at constant per-step cost; 79% vs 42% task completion
- TEMPO: Motion and command history for +2M parameters; 47.5% to 76.0% average success