Skip to content

Memory and long context

Cross-day theme. Facts come from the linked day and paper notes. Sentences marked "my reading" are interpretation.

Most robot foundation models act on the current frame. That's fine for short, repetitive skills, but tracking what happened, or finishing a 15-minute job, needs memory. The week's papers stored memory in very different places, with very different costs.

The approaches, from implicit to explicit

Approach Day Where memory lives Horizon Cost per step Result
TEMPO 1 motion and command history tokens in the VLM prefix short constant (+2M params) 47.5% → 76.0% average success
π0.7 MEM 1 compressed video history about 6 s (6 frames per camera, 1 s stride) small "insufficient" for long tracking (Day 4 talk)
RoboTTT 1 fast weights updated by test-time training 8K steps, about 5 min at 30 Hz constant 79% vs 42% task completion
MemoryVAM 1 episodic memory in a video action model episode grows with the episode LIBERO-Mem 5% → 42.5%
Mem-World 1 4D surfel map, retrieved by future actions episode retrieval: top-K frames none from the talk
π0.6-MEM (long-context VLA) 4 long-term text summary plus short-term video 15+ min tasks text stays small about 78% vs 42% average task progress (read off a chart)
FARM 4 explicit 3D graph of objects and relations whole building, 1,600–15,000 m² 5–10 Hz online mapping +164% Recall@5 over prior methods (unverified)

Memory inside the policy

RoboTTT fast-weight recurrence and constant per-step cost

A VLA with multiscale memory: long-term text memory and short-term video memory

The Day 4 long-context talk named two problems:

  1. Spurious correlations between past observations and future actions. Sometimes adding memory makes the policy worse.
  2. Compute grows with context, for both training and inference.

Its fixes: - For the training signal: predict subtasks and memory narrations. - For speed: cache and reuse past computation, decode the narration in parallel, and group actions.

Memory as retrieval or a map

Mem-World: future actions choose which history to retrieve

FARM: build a relational spatial memory, then query it

The Sim2Real2Field talk (Day 4) put it simply: how a robot builds its memory, and how it queries it, both matter.

Takeaways

  • Two tiers keep coming up: dense visual memory for the last few seconds or minutes, and sparse semantic memory (text or an object graph) for anything longer.
  • Constant cost per step is the dividing line for real-time control. TEMPO, RoboTTT, and text summaries have it; a growing video context doesn't.
  • My reading: The text-memory result (π0.6-MEM) and FARM's object graph are the same move at different scales. Both compress history into symbols the model can query.

Papers tagged memory or temporal-context

  • FARM: Real-time object-level relational 3D memory answers relational language queries; +164% Recall@5 over prior methods (unverified)
  • MemoryVAM: LIBERO-Mem from 5% to 42.5%; 75–80% on real memory tasks
  • Mem-World: Future actions pick which history a multi-view world model retrieves, via a wrist-view 4D surfel memory
  • MemoryVAM: LIBERO-Mem from 5% to 42.5%; 75–80% on real memory tasks
  • π0.7: Metadata turns low-quality data from harmful to helpful; zero-shot UR5e shirt folding at teleoperator level
  • RoboTTT: 8K-step context at constant per-step cost; 79% vs 42% task completion
  • TEMPO: Motion and command history for +2M parameters; 47.5% to 76.0% average success