Simulation and sim-to-real¶
Cross-day theme. Facts come from the linked day and paper notes. Sentences marked "my reading" are interpretation.
Simulation showed up at every layer this week: physics engines, building scenes from real video, training recipes, evaluation, and safe transfer. The table follows that stack from bottom to top.
The stack, layer by layer¶
| Layer | Talk or paper | Day | Key point |
|---|---|---|---|
| Physics | Isaac Lab 3.0 and Newton (NVIDIA) | 3 | One task API across PhysX and Newton. Newton solvers: MuJoCo-Warp, Kamino, VBD, MPM, Steel, and learned solvers |
| Steel (Skild AI) | 3 | GPU contact solver for many different robots in one batch, with no padding to the largest robot | |
| SuperDex (Meta) | 2 | open-source contact-rich dexterity sim, including soft objects | |
| Scenes from the real world | Video to Data and CHORD (NVIDIA) | 3 | agents turn human demo video into sim environments and a physics-grounded dataset |
| Hybrid Simulation, Hydra-0 (NVIDIA) | 3 | one image, one click: an agent builds and calibrates the digital twin | |
| Agentic Robotics, GaP (Goldberg) | 3 | inverse physics fits a Newton sim to one demo video | |
| Training recipes | Lutter (Boston Dynamics) | 4 | demo resets plus a frontier reset curriculum: 96.5% success on one task, vs below 60% for the baselines |
| SuperMimic (UC Berkeley), Best Paper | 3 | learn a differentiable world model from MuJoCo Warp, then train through it: >1000× fewer GPU-hours than SONIC, zero-shot to a real G1 | |
| FOVI-Dex (NVIDIA) | 3 | camera rendering limits RL to 208 environments per GPU; foveation is the fix | |
| MolmoBot (Ai2) | 1 | sim-only training: 79.2% real pick-and-place vs 39.2% for π0.5 | |
| GaP | 3 | coding agents self-learn a graph of skills in sim; FlipAnything 24.7% → 95.0% | |
| Evaluation | Isaac Lab-Arena (NVIDIA) | 3 | author, execute, analyze: finds where and why a policy fails |
| DexBench (RLWRLD) | 3 | 15 industrial dexterity tasks with frozen evaluation conditions | |
| Safe transfer | TAM (Ai2, KAIST) | 2 | sim-only residual torque layer under any policy |
| Safety filters and ResSafe | 4 | a residual RL correction keeps a reference policy inside the safe set |
From real video to simulation¶
Training in simulation with demos¶
Keeping transfer safe¶
Takeaways¶
- Heterogeneity is the new scaling axis. Locomotion scaled with thousands of identical environments, one task, and one PPO. Generalist robots need many robots, tasks, and solvers in one batch, which is the point of Steel, Newton, and Arena.
- Rendering is now the bottleneck for vision-based RL, not physics (FOVI-Dex: 208 environments per GPU).
- Agents are moving into the sim pipeline: building digital twins (Hydra), fitting physics to video (GaP), authoring benchmarks (Arena), and turning video into data (Video to Data).
- My reading: Lutter's fleet talk is the reminder that sim-to-real also depends on the real robots still matching the sim model. See Deployment and reliability.
Papers tagged sim-to-real or simulation¶
- GaP: Coding agents self-learn a graph of modular skills in a Newton Real2Sim, then export plain ROS 2 code; no frontier model runs on the robot
- MolmoBot: Sim-only training; 79.2% real pick-and-place vs 39.2% for π0.5
- SuperMimic: One learned world model + analytic gradients; beats SONIC's return in 3 h on 1 GPU vs 7 days on 128 GPUs (>1000× fewer GPU-hours)
- TAM: Sim-only residual torque layer below any policy; real Panda box pushing 47.6% to 76.2%, BC flip 50% to 72%
- GaP: Coding agents self-learn a graph of modular skills in a Newton Real2Sim, then export plain ROS 2 code; no frontier model runs on the robot
- Hydra-0: World model conditioned on action flow (robot actions as pixel motion); runs as a hybrid simulator, with physics for the robot and video for the world
- MolmoBot: Sim-only training; 79.2% real pick-and-place vs 39.2% for π0.5
- SuperMimic: One learned world model + analytic gradients; beats SONIC's return in 3 h on 1 GPU vs 7 days on 128 GPUs (>1000× fewer GPU-hours)
- TAM: Sim-only residual torque layer below any policy; real Panda box pushing 47.6% to 76.2%, BC flip 50% to 72%