Skip to content

Simulation and sim-to-real

Cross-day theme. Facts come from the linked day and paper notes. Sentences marked "my reading" are interpretation.

Simulation showed up at every layer this week: physics engines, building scenes from real video, training recipes, evaluation, and safe transfer. The table follows that stack from bottom to top.

The stack, layer by layer

Layer Talk or paper Day Key point
Physics Isaac Lab 3.0 and Newton (NVIDIA) 3 One task API across PhysX and Newton. Newton solvers: MuJoCo-Warp, Kamino, VBD, MPM, Steel, and learned solvers
Steel (Skild AI) 3 GPU contact solver for many different robots in one batch, with no padding to the largest robot
SuperDex (Meta) 2 open-source contact-rich dexterity sim, including soft objects
Scenes from the real world Video to Data and CHORD (NVIDIA) 3 agents turn human demo video into sim environments and a physics-grounded dataset
Hybrid Simulation, Hydra-0 (NVIDIA) 3 one image, one click: an agent builds and calibrates the digital twin
Agentic Robotics, GaP (Goldberg) 3 inverse physics fits a Newton sim to one demo video
Training recipes Lutter (Boston Dynamics) 4 demo resets plus a frontier reset curriculum: 96.5% success on one task, vs below 60% for the baselines
SuperMimic (UC Berkeley), Best Paper 3 learn a differentiable world model from MuJoCo Warp, then train through it: >1000× fewer GPU-hours than SONIC, zero-shot to a real G1
FOVI-Dex (NVIDIA) 3 camera rendering limits RL to 208 environments per GPU; foveation is the fix
MolmoBot (Ai2) 1 sim-only training: 79.2% real pick-and-place vs 39.2% for π0.5
GaP 3 coding agents self-learn a graph of skills in sim; FlipAnything 24.7% → 95.0%
Evaluation Isaac Lab-Arena (NVIDIA) 3 author, execute, analyze: finds where and why a policy fails
DexBench (RLWRLD) 3 15 industrial dexterity tasks with frozen evaluation conditions
Safe transfer TAM (Ai2, KAIST) 2 sim-only residual torque layer under any policy
Safety filters and ResSafe 4 a residual RL correction keeps a reference policy inside the safe set

Isaac Lab-Arena stack

From real video to simulation

Video to Data: real to sim to real

Agentic Robotics pipeline

Training in simulation with demos

Demo resets: teacher RL starts from sampled demo states

Frontier reset curriculum

Keeping transfer safe

Safety filter and ResSafe residual RL

Takeaways

  • Heterogeneity is the new scaling axis. Locomotion scaled with thousands of identical environments, one task, and one PPO. Generalist robots need many robots, tasks, and solvers in one batch, which is the point of Steel, Newton, and Arena.
  • Rendering is now the bottleneck for vision-based RL, not physics (FOVI-Dex: 208 environments per GPU).
  • Agents are moving into the sim pipeline: building digital twins (Hydra), fitting physics to video (GaP), authoring benchmarks (Arena), and turning video into data (Video to Data).
  • My reading: Lutter's fleet talk is the reminder that sim-to-real also depends on the real robots still matching the sim model. See Deployment and reliability.

Papers tagged sim-to-real or simulation

  • GaP: Coding agents self-learn a graph of modular skills in a Newton Real2Sim, then export plain ROS 2 code; no frontier model runs on the robot
  • MolmoBot: Sim-only training; 79.2% real pick-and-place vs 39.2% for π0.5
  • SuperMimic: One learned world model + analytic gradients; beats SONIC's return in 3 h on 1 GPU vs 7 days on 128 GPUs (>1000× fewer GPU-hours)
  • TAM: Sim-only residual torque layer below any policy; real Panda box pushing 47.6% to 76.2%, BC flip 50% to 72%
  • GaP: Coding agents self-learn a graph of modular skills in a Newton Real2Sim, then export plain ROS 2 code; no frontier model runs on the robot
  • Hydra-0: World model conditioned on action flow (robot actions as pixel motion); runs as a hybrid simulator, with physics for the robot and video for the world
  • MolmoBot: Sim-only training; 79.2% real pick-and-place vs 39.2% for π0.5
  • SuperMimic: One learned world model + analytic gradients; beats SONIC's return in 3 h on 1 GPU vs 7 days on 128 GPUs (>1000× fewer GPU-hours)
  • TAM: Sim-only residual torque layer below any policy; real Panda box pushing 47.6% to 76.2%, BC flip 50% to 72%