World models and world action models¶
Cross-day theme. Facts come from the linked day and paper notes. Sentences marked "my reading" are interpretation.
The question running through the week was whether robots need a model of how the world changes. Day 4 had a whole workshop on it: Do Robotics Need World Models? The talks used world models in three different roles:
- As the policy itself: predict video and actions together.
- As a simulator: generate experience for training and evaluation.
- As a planner: imagine the outcome first, then extract actions.
Who used world models, and how¶
| Talk or paper | Day | Role | What it does | Key number |
|---|---|---|---|---|
| DreamZero (NVIDIA) | 1 | policy | 14B video model that predicts future frames and an action chunk together | over 2× generalization vs VLAs, at 7 Hz |
| MemoryVAM | 1 | policy | adds episodic memory to a video action model | LIBERO-Mem 5% → 42.5% |
| Mem-World | 1 | policy / simulator | multi-view action-conditioned world model; future actions choose which history to retrieve | none from the talk |
| π0.7 (Physical Intelligence) | 1 | planner | a world model generates subgoal images for the policy | zero-shot UR5e shirt folding at teleoperator level |
| LVP (MIT, Berkeley, Harvard) | 1 | planner | a 14B video model plans in pixels; actions are retargeted from the generated hands | 59.3% task completion on 100 third-party tasks |
| AGIBOT GO-3 | 2 | policy | "world reasoning action model": reason, predict the future state, then act | preview only |
| Tesla Optimus | 2 | simulator | learned world simulator for counterfactuals, closed-loop evaluation, and RL | none given |
| Hydra-0 (NVIDIA) | 3 | simulator | hybrid simulator: physics moves the robot, a video model predicts the world | none from the talk |
| SuperMimic (UC Berkeley), Best Paper | 3 | simulator | one differentiable world model, fit to a physics simulator, trains a humanoid whole-body controller by backpropagation | beats SONIC's return in 3 h on 1 GPU vs 7 days on 128 GPUs |
| Fei-Fei Li, Atlas (World Labs) | 3 | simulator | one "omni" model for generation and 3D reconstruction; renders, simulates, helps robots plan | #1 on Rapidata's camera-movement leaderboard |
| Dyna-2 (Dyna Robotics) | 4 | policy | world-action model pre-trained on 1M+ hours of human video only | zero-shot action MSE 0.180 → 0.117 from 1k to 1M hours |
| Long-context talk | 4 | none | argues that memory, subtasks, and prompting are enough without one | π0.6-MEM about 78% vs 42% average task progress |
| Remote talk, VLA vs WAM | 4 | comparison | VLA passes "what to do"; WAM passes "how the world should change" | "less context, better dynamics" |
What a world simulator has to get right¶
From Tesla's Day 2 talk: a world model used for training and evaluation must be causal (frame by frame), interactive (conditioned on fine-grained actions), consistent across cameras and time, low-latency, and physically accurate rather than visually appealing.
World model as the policy¶
The case against¶
The Day 4 long-context talk closed with "No need for explicit mechanisms that explicitly build in a model of the world" and "Simple recipes often scale more readily!" Its evidence was π0.6-MEM, a VLA with text and video memory and no world model. See Memory and long context.
Takeaways¶
- My reading: "World model" covered three different jobs this week. Comparisons are only fair within one role.
- The simulator role had the most industry backing: Tesla, NVIDIA (Hydra-0, Cosmos), and World Labs. Their shared requirement is physical accuracy under action conditioning, not pretty video.
- In the policy role, human video is the selling point. DreamZero transfers from minutes of human video, and Dyna-2 scales to a million hours of it. See Human data and scaling.
Papers tagged world-action-model¶
- DreamZero: 14B video model as policy; over 2× generalization vs VLAs at 7 Hz
- Dyna-2: World-action model pre-trained on 1M+ hours of human video only; zero-shot robot action MSE 0.180 to 0.117 from 1k to 1M hours
- Hydra-0: World model conditioned on action flow (robot actions as pixel motion); runs as a hybrid simulator, with physics for the robot and video for the world
- MemoryVAM: LIBERO-Mem from 5% to 42.5%; 75–80% on real memory tasks