Skip to content

Hydra-0: Action Flow for Generalist World Modeling and Control

Session: NVIDIA's Hybrid Simulation: Cosmos Neural with Isaac Lab talk on Day 3. Slide photos taken 11:02–11:06 am. See the Day 3 notes. Paper: Technical report, preprint, August 2026. The slide showed an arXiv button but no ID, and code is "coming soon." Authors: Hongyu Li, Bowen Wen, Xinghao Zhu, Yixuan Wang, Yilun Du, Yunzhu Li, George Konidaris, Stan Birchfield, Soha Pouya, Chenran Li, Yan Chang (NVIDIA, Brown, Columbia, Harvard). Project page: nvidia-isaac.github.io/video_to_data/hydra-0. The URL suggests it belongs to NVIDIA's Video to Data effort.

Paper summary (from a slide photo of the project page's abstract; the paper itself is not yet read)

  • The idea: A generalist world model conditioned on action flow, which represents robot actions as pixel motion.
  • Hybrid simulator: At deployment, a physics engine moves the robot and a learned video model predicts how the world responds.
  • What one model does: It trains across embodiments, simulates them, evaluates policies, and drives a real robot.

Talk notes

  • Object flow as the shared action interface: A System 2 agent plans the desired object motion as flow, and a System 1 policy turns that flow into robot actions.
  • Demo: Given one image and "Drag the purple cloth towards the left," the agent proposed a 29-point flow on the cloth.

Hydra object-flow interface and hybrid simulator