Large Video Planner Enables Generalizable Robot Control (LVP)¶
Session: Poster, Mon Nov 9. Photo taken at 5:30 pm. Authors: Boyuan Chen*, Tianyuan Zhang*, Haoran Geng*, Caiyi Zhang*, Peihao Li*, Kiwhan Song, William T. Freeman, Jitendra Malik, Pieter Abbeel, Russ Tedrake, Vincent Sitzmann, Yilun Du (MIT, UC Berkeley, Harvard). *Equal contribution. Demo: huggingface.co/spaces/KempnerInstituteAI/LVP
Paper summary (from a poster photo; paper not yet found or read)¶
- Problem: General-purpose robots need planners that transfer to unseen tasks.
- Method:
- Video as the primary modality: Given a scene image and an instruction, LVP predicts a visual plan as video, then extracts executable robot actions from the generated motion.
- Model: Initialized from Wan 2.1 14B and trained with flow matching. Diffusion forcing supports both image and video conditioning, and a temporally causal VAE encodes video into latents. Training uses independent noise levels for the history and future segments.
- History guidance strengthens adherence to observed scene frames. An autoregressive extension enables multi-stage video plans.
- Video to actions: HaMeR and MegaSAM recover 3D hand motion. Dex-Retargeting maps fingers, and calibrated wrist poses guide cuRobo arm IK. Parallel grippers use GraspNet and grasping heuristics.
- The poster lists three contributions: (i) LVP, a 14B video model and framework for zero-shot robot control; (ii) LVP-1M, an open dataset of robot action clips (partly hidden in the photo); and (iii) task-level evaluation on real robots (partly hidden).
- Results:
- Video generation on 100 in-the-wild tasks proposed by third parties, with four videos per task scored independently. LVP scores 87.3% on contact, 63.2% on end state, 59.3% on task complete, and 44.0% on perfect. It beats Wan 2.1, Hunyuan, and Cosmos on every level. Task completion is 82.0% Best@4, meaning at least one of the four videos succeeds.
- Real robots: A Franka with a parallel gripper, and a G1 with Inspire hands. Tasks include unseen objects and scenes, such as opening a first-aid box, opening a fridge door, wiping a table, scooping coffee beans, and tearing off clear tape. π0 and OpenVLA are baselines only for the parallel-gripper tasks. The per-task success counts are in the poster's Figure 6 and are too small to transcribe reliably.
- Limitations (from the poster): Slow video generation, open-loop control, and retargeting errors.
Questions / discussion¶
- How does it compare with DreamZero, which predicts video and actions jointly instead of retargeting from generated video?