Skip to content

World models and world action models

Cross-day theme. Facts come from the linked day and paper notes. Sentences marked "my reading" are interpretation.

The question running through the week was whether robots need a model of how the world changes. Day 4 had a whole workshop on it: Do Robotics Need World Models? The talks used world models in three different roles:

  1. As the policy itself: predict video and actions together.
  2. As a simulator: generate experience for training and evaluation.
  3. As a planner: imagine the outcome first, then extract actions.

Three roles for world models in robotics: the policy, a simulator, and a planner, with the talks that used each

Who used world models, and how

Talk or paper Day Role What it does Key number
DreamZero (NVIDIA) 1 policy 14B video model that predicts future frames and an action chunk together over 2× generalization vs VLAs, at 7 Hz
MemoryVAM 1 policy adds episodic memory to a video action model LIBERO-Mem 5% → 42.5%
Mem-World 1 policy / simulator multi-view action-conditioned world model; future actions choose which history to retrieve none from the talk
π0.7 (Physical Intelligence) 1 planner a world model generates subgoal images for the policy zero-shot UR5e shirt folding at teleoperator level
LVP (MIT, Berkeley, Harvard) 1 planner a 14B video model plans in pixels; actions are retargeted from the generated hands 59.3% task completion on 100 third-party tasks
AGIBOT GO-3 2 policy "world reasoning action model": reason, predict the future state, then act preview only
Tesla Optimus 2 simulator learned world simulator for counterfactuals, closed-loop evaluation, and RL none given
Hydra-0 (NVIDIA) 3 simulator hybrid simulator: physics moves the robot, a video model predicts the world none from the talk
SuperMimic (UC Berkeley), Best Paper 3 simulator one differentiable world model, fit to a physics simulator, trains a humanoid whole-body controller by backpropagation beats SONIC's return in 3 h on 1 GPU vs 7 days on 128 GPUs
Fei-Fei Li, Atlas (World Labs) 3 simulator one "omni" model for generation and 3D reconstruction; renders, simulates, helps robots plan #1 on Rapidata's camera-movement leaderboard
Dyna-2 (Dyna Robotics) 4 policy world-action model pre-trained on 1M+ hours of human video only zero-shot action MSE 0.180 → 0.117 from 1k to 1M hours
Long-context talk 4 none argues that memory, subtasks, and prompting are enough without one π0.6-MEM about 78% vs 42% average task progress
Remote talk, VLA vs WAM 4 comparison VLA passes "what to do"; WAM passes "how the world should change" "less context, better dynamics"

What a world simulator has to get right

From Tesla's Day 2 talk: a world model used for training and evaluation must be causal (frame by frame), interactive (conditioned on fine-grained actions), consistent across cameras and time, low-latency, and physically accurate rather than visually appealing.

Hydra: object flow as the shared action interface, and the hybrid simulator

World model as the policy

DreamZero closed-loop inference with the KV cache

VLA vs world action model, heavily simplified

The case against

The Day 4 long-context talk closed with "No need for explicit mechanisms that explicitly build in a model of the world" and "Simple recipes often scale more readily!" Its evidence was π0.6-MEM, a VLA with text and video memory and no world model. See Memory and long context.

Takeaways

  • My reading: "World model" covered three different jobs this week. Comparisons are only fair within one role.
  • The simulator role had the most industry backing: Tesla, NVIDIA (Hydra-0, Cosmos), and World Labs. Their shared requirement is physical accuracy under action conditioning, not pretty video.
  • In the policy role, human video is the selling point. DreamZero transfers from minutes of human video, and Dyna-2 scales to a million hours of it. See Human data and scaling.

Papers tagged world-action-model

  • DreamZero: 14B video model as policy; over 2× generalization vs VLAs at 7 Hz
  • Dyna-2: World-action model pre-trained on 1M+ hours of human video only; zero-shot robot action MSE 0.180 to 0.117 from 1k to 1M hours
  • Hydra-0: World model conditioned on action flow (robot actions as pixel motion); runs as a hybrid simulator, with physics for the robot and video for the world
  • MemoryVAM: LIBERO-Mem from 5% to 42.5%; 75–80% on real memory tasks