Skip to content

World Action Models are Zero-Shot Policies (DreamZero)

Session: Oral Session 2, Talk 1 Speaker(s): Paper: arXiv 2602.15922 (PDF) by Seonghyeon Ye, Yunhao Ge, et al. (36 authors, NVIDIA), February 2026 Model: DreamZero. Weights and inference code are released.

Paper summary (from the arXiv HTML and abstract; figure-only numbers not yet read)

Project leads: Seonghyeon Ye, Yuke Zhu, Linxi "Jim" Fan, and Joel Jang (NVIDIA). Submitted February 17, 2026. Links: Project page, code. The checkpoint and inference code are released. The AgiBot dataset is not released yet.

Method

  • Backbone: Wan2.1-I2V-14B-480P, a 14B image-to-video diffusion model. A 5B version is used only for ablations.
  • Joint denoising: One end-to-end model denoises video latents and actions together with a single flow-matching loss and a shared timestep. Video and actions have separate encoders and decoders.
  • Autoregressive video only: Earlier chunks are clean context (teacher forcing). After each chunk executes, the predicted frames in the KV cache are replaced with real observations, which stops video errors from compounding.
  • Actions: Relative joint positions. Each chunk is 48 steps at 30 Hz (1.6 s) on the bimanual robot.
  • Inputs: Multi-view images concatenated into one frame, language, and proprioception.
  • Getting to 7 Hz: A naive single-GPU run takes 5.7 s per chunk; the optimized version takes 150 ms on 2 GB200s, a 38× speedup. The steps:
  • Asynchronous execution, so the robot runs the current chunk while the next one is computed.
  • Classifier-free guidance split across 2 GPUs, which cuts per-step latency by 47%.
  • DiT caching, which reuses velocities when they barely change and cuts the effective steps from 16 to 4.
  • torch.compile with CUDA Graphs, plus NVFP4 quantization on Blackwell.
  • DreamZero-Flash: Video is trained mostly at low noise levels while actions keep a uniform noise schedule, so one denoising step is enough. This brings latency from about 350 ms to about 150 ms.

Training data

  • AgiBot G1: About 500 h of teleoperation in 22 environments, as 7.2K episodes averaging about 4.4 min.
  • DROID: Used for the Franka experiments.
  • Training run: 100K steps at batch size 128, with the full DiT trained. LoRA gave worse results.

Results (baselines are GR00T N1.6 and π0.5, both trained from scratch and from their pretrained checkpoints)

Setting DreamZero Best VLA
AgiBot, seen tasks (task progress) 62.2% 27.4%
AgiBot, unseen tasks (task progress) 39.5% 16.3% (from-scratch VLAs <1%)
DROID, unseen tasks (progress / success) 49% / 22.5% 33% / 12.5%
  • "2× generalization": This works out to 2.3× on seen tasks and 2.4× on unseen ones.
  • Video-only transfer (unseen tasks): The baseline is 38.3%. Adding 12 min of human video raises it to 54.3% (+42%), and adding 20 min of YAM robot video raises it to 55.4% (+45%). Neither source includes action labels.
  • New embodiment (YAM): About 30 min of play data (55 trajectories, 11 tasks) gives strong language following and generalization to new objects. Only qualitative results are given.
  • DreamZero-Flash on table bussing: 4 steps reach 83% at 350 ms. 1 step without Flash training drops to 52% at 150 ms. 1 step with Flash training reaches 74% at 150 ms.
  • Ablations:
  • Diverse data beats repetitive data: 50% vs 33% at equal hours.
  • 14B beats 5B: 50% vs 21%.
  • An autoregressive model matches a bidirectional one in success but is 3–4× faster and moves more smoothly.

Limitations (stated by the authors)

  • Most failures come from video-generation errors, not action errors.
  • It is much more expensive than VLAs, which run above 20 Hz on consumer GPUs.
  • Memory covers only about 6 s, so it acts as a System 1 policy and long-horizon tasks need a System 2 planner.
  • It is weak on sub-centimeter precision tasks such as key insertion.
  • Each embodiment is pretrained separately.
  • The human-video experiments are small, at 12 min.

Talk notes

Motivation

  • Humans can do these tasks zero-shot.
  • Video generation models already know how to do these tasks zero-shot. The slide (photo taken at 11:11 am) shows a person walking through a door and a hand washing a pan, then asks: "Can we leverage the general capability of video models for robot manipulation?"

Inference: closed-loop real-world execution (slide photo, 11:14 am)

  • Model: A joint video-action DiT (causal DiT blocks). Its inputs are past frames through a VAE encoder, plus proprioception and language through a state and text encoder. Its outputs are future frames through a VAE decoder, and a future action chunk through an action decoder.
  • Inference steps:
  • Use the KV cache for autoregressive flow sampling.
  • Execute the actions asynchronously on the real robot.
  • Update the KV cache with the real observations.
  • Poster numbers (photo taken at 5:42 pm): 38× faster inference, 7 Hz closed loop at about 150 ms per chunk, and 38.3% → 54.3% from 10–20 min of video. These match the paper summary above.

DreamZero closed-loop inference with the KV cache