Day 3: Wed, Nov 11, 2026¶
Awards¶
Reported in press coverage on Nov 11. corl.org's awards page didn't list winners when checked.
- Best Paper: SuperMimic (UC Berkeley). One differentiable world model plus analytic gradients beats SONIC's return in 3 h on 1 GPU, against 7 days on 128 GPUs.
- Best Student Paper: RoboTTT (NVIDIA GEAR, Stanford, UT Austin). Test-time training gives an 8K-step context at constant cost per step.
Keynotes¶
- Fei-Fei Li (Stanford; founder and CEO, World Labs), about 3:10 pm: Atlas: A Unified World Model for Generation and Reconstruction. Notes under Live notes.
Sessions attended¶
- Tech Session: Isaac Lab 3.0 (NVIDIA). Speaker and time not yet recorded. Notes under Live notes.
- Ken Goldberg: Agentic Robotics. Session not yet recorded. About 12:50 pm, from a photo. Notes under Live notes.
- FOVI-Dex (Nicholas M. Blauch et al.), about 11:20 am. Notes under Live notes.
- NVIDIA Isaac Lab-Arena (evaluation framework), 12:22–12:36 pm, from slide photos. Speaker not recorded. Notes under Live notes.
- RLWRLD: DexBench. Speaker and session not yet recorded. 12:40–12:47 pm, from slide photos. Notes under Live notes.
- Skild AI: Steel: Simulation for Generalist Robot Learning. Speaker and session not yet recorded. Slide photo taken at 10:43 am. Notes under Live notes.
Papers covered¶
| Paper | Group | Headline | Status | Links | Weights |
|---|---|---|---|---|---|
| GaP | UC Berkeley, NVIDIA, Bosch | Coding agents self-learn a graph of modular skills in a Newton Real2Sim, then export plain ROS 2 code; no frontier model runs on the robot | abstract | arXiv · page · code | ? |
| Hydra-0 | NVIDIA, Brown, Columbia, Harvard | World model conditioned on action flow (robot actions as pixel motion); runs as a hybrid simulator, with physics for the robot and video for the world | talk-only | page | ? |
| SuperMimic | UC Berkeley | One learned world model + analytic gradients; beats SONIC's return in 3 h on 1 GPU vs 7 days on 128 GPUs (>1000× fewer GPU-hours) | summarized | arXiv · page | ? |
Day 3 themes¶
Cross-day versions: simulation, world models, deployment.
- Simulation for heterogeneous robots: Isaac Lab 3.0 and Newton (one task API across solvers), Steel (many robots in one batch), and Arena (one evaluation framework).
- Agents build the sim: Video to Data, Hydra's one-click digital twin, and Goldberg's inverse physics all use agents to turn real video into simulation.
- Evaluation as a bottleneck: Arena's why-and-where failure analysis and DexBench's frozen, industrial dexterity benchmark.
- World models as simulators: Hydra-0's hybrid physics plus video simulator, and Fei-Fei Li's Atlas, which unifies generation and 3D reconstruction.
- Rendering limits vision RL: FOVI-Dex fits only 208 environments per GPU with cameras, and uses foveation to fix it.
Live notes¶
Tech Session: Isaac Lab 3.0¶
Tech session. Speaker not recorded. The backend slides were photographed at 10:15–10:16 am.
Background, from the release notes and not from the talk: Isaac Lab 3.0 went GA around Oct 19–20, 2026 (v3.0.0 on GitHub, isaaclab 3.0.0 on PyPI). Its headline change is one task API across physics backends. The new backend is Newton, a Warp-native engine with MuJoCo-Warp for articulated rigid bodies, VBD for deformables, and MPM for particles, and these solvers can be coupled within one task. Newton workflows run without launching Isaac Sim, so Isaac Sim is now optional. Some features are still PhysX-only, including deformable objects, surface grippers, and material randomization. Sources disagree on the pinned Isaac Sim, Newton, and Warp versions, so check the GitHub release. Other sources: release notes, NVIDIA forum announcement.
- Isaac Lab 3.0 at a glance:
- An open-source, GPU-accelerated framework for robot learning at scale.
- Multi-physics: (Slide cut off here. My reading, from the release notes: one task API across PhysX and Newton's MuJoCo-Warp, VBD, and MPM solvers.)
- Throughput: Environments are CUDA-graphed. Physics and rendering run batched and entirely on the GPU. (My reading: a CUDA graph captures the whole step once and replays it, which removes per-step CPU launch overhead.)
- Sim-to-real transfer: Policies trained in Isaac Lab transfer to real hardware.
- Workloads running on Isaac Lab:
- Perceptive RL: Thousands of environments per GPU, using tiled cameras. (A tiled camera renders every environment's camera into one batched image.)
- WAM and VLA post-training: Billion-parameter policies running on separate GPUs from the simulation.
- Video to data: CHORD, a benchmark of 4,739 tasks. (Task count as shown on the slide. I haven't checked it.)
- How video becomes sim tasks or data: See Video to Data: Agentic Native Pipeline below. CHORD stands for Contact Wrench Guidance from Human Demonstration in Robotic Dexterous Manipulation.
- Whole-body control: SONIC, the humanoid motion tracker (SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Control, Science Robotics 2026). SuperMimic uses it as its baseline.
- World models: SuperMimic and Cosmos. SuperMimic (UC Berkeley) won the CoRL 2026 Best Paper Award, as reported. Cosmos is NVIDIA's family of world foundation models.
- Framing question: What are the models, libraries, and frameworks we need to accelerate solving robot learning problems?
- Multi-physics backends: "PhysX, Newton behind the same task code" (slide photo):
- The demo ran the same task three ways: Newton (MJWarp) in sim, PhysX in sim, and a real Unitree G1 walking over stepping stones.
- Newton: Open source, GPU-native, with multiple solvers. NVIDIA co-initiated it with Disney Research and Google DeepMind.
- OvPhysX: "PhysX without Kit."
-
Newton physics engine: "Open-source, GPU-accelerated physics library built on NVIDIA Warp" (slide photo).
- Built on NVIDIA Warp.
- The primary backend is MuJoCo-Warp.
- Imports: URDF, MJCF, and USD.
- Simulation loop:
ModelBuilder(build or import) →Model(finalized) →State,Control, andCollision→solver.step()→State, which goes to the sensors and viewer. - Solvers listed on the slide for
solver.step(): MuJoCo, Featherstone, Kamino, XPBD, VBD, ImplicitMPM, "...". - "Every solver implements the same step, so Isaac Lab switches solvers by configuration."
- Newton's solvers, and what each one handles:
Solver Used for MuJoCo-Warp Rigid robots. The primary path. Kamino Closed kinematic loops and hard contacts. (My reading of "loops.") VBD Cloth, soft bodies, and cables. MPM Granular materials. Steel A high-throughput rigid-body solver, from Skild AI. See Skild AI: Steel below. Learned solvers Learned physics models, through newton.neural.- What models
newton.neuralsupports, and how they're trained:- FlashNeRD runs through
newton.neural. (My guess: a faster version of NeRD, NVIDIA's Neural Robot Dynamics from CoRL 2025. Confirm.) - Demo: PPO on ANYmal. (I assume PPO trained with the learned dynamics model as the simulator. Confirm.)
- FlashNeRD runs through
- Multi-rendering backends:
- New visualizers and a web UI: Lightweight, with high fidelity.
- Real-world scenes and photoreal rendering: New scene and sensor sources plug into the same task.
- Which scene sources (Gaussian splats? real-to-sim scans?):
- The docs page names NuRec, which turns real-world captures into simulation-ready scenes. See From the ecosystem docs page below.
- RL post-training with Isaac RLPT: Post-training GR00T N1.7 with PPO inside Isaac Lab.
- Pretrained policies and deployment:
- Example pretrained policies are pulled automatically.
- Simplified deployment to real robots.
- Deployment path (ROS 2? which robots?):
- The docs have Sim2Real examples: gear assembly, COMPASS/NuRec, reach, LEAPP export, SO-101 typing, and G1 visual locomotion. See From the ecosystem docs page below.
- Where to start: Example tasks and a command builder.
- Learning libraries:
- The docs page lists the RL libraries that ship in
isaaclab_rl: RSL-RL, skrl, Stable Baselines 3, and RL Games. For VLA post-training it names RLinf. See From the ecosystem docs page below. Is RLinf the same thing as the talk's Isaac RLPT?
From the ecosystem docs page¶
Source: the Isaac Lab ecosystem page, "main" docs branch, last updated Nov 11, 2026, Isaac Sim 6.2.0 badge. This is from the docs, not the talk. The page doesn't mention Kamino, Steel, newton.neural, or Cosmos.
What Isaac Lab is. The page says Isaac Lab is "not a robotics simulator; it is a framework for building robot learning applications on top of a simulator." It compares this to RoboSuite on MuJoCo, MuJoCo Playground on MJX, and Isaac Gym on PhysX. Shared scene, asset, sensor, and task APIs work across every physics backend, renderer, and learning library. Environment code doesn't import modules specific to one backend.
The stack, top to bottom. This is my reading. The page doesn't draw it as a diagram.
- Applications: Isaac Lab-Arena and the GR00T workflows.
- Learning libraries: RSL-RL, skrl, SB3, RL Games, RLinf, and
isaaclab_mimic. - Isaac Lab: tasks, written either manager-based or direct, with a gymnasium interface and Hydra configs.
- Physics and rendering backends: Newton, OvPhysX, Isaac Sim PhysX, and the renderers listed below.
Physics backends. The page's own text says "two physics backends," but its table lists three.
| Backend | Best fit | Trade-off |
|---|---|---|
| Newton | Lightweight training. Warp-native on the GPU, runs without Isaac Sim, with MJWarp, VBD, MPM, and other solvers. Included in the default install. | Changing solvers can require retuning. |
| OvPhysX | Standalone PhysX without Kit. | Can't be mixed with Isaac Sim workflows. |
| Isaac Sim PhysX | Workflows that need full Kit integration. | Needs the larger Isaac Sim and Kit runtime. |
Rendering backends:
| Renderer | Best fit | Trade-off |
|---|---|---|
| Newton Tiled Camera | Kit-less training where small camera images are enough. Warp-native ray tracing, with RGB, depth, normals, and semantic and instance segmentation. | Lower fidelity. |
| OvRTX | Kit-less rendering with high-fidelity RTX image quality. | An optional runtime, and it may need more VRAM. |
| Isaac RTX | Isaac Sim workflows that need full RTX fidelity and the widest range of sensor outputs. | Needs Isaac Sim. |
- Visualizers (
isaaclab_visualizers): Kit, Newton, Rerun, and Viser. (These likely back the talk's "new visualizers and web UI." That's my guess.) - Sensors: Cameras, ray-casters, contact sensors, IMUs, frame transformers, joint-wrench sensors, and visuo-tactile sensors.
Two ways to write a task:
- Manager-based: Typed managers handle observations, rewards, terminations, events, curriculum, commands, actions, and recording. Each is driven by term functions in isaaclab.envs.mdp.
- Direct: A subclass implements _get_observations, _get_rewards, _get_dones, and _reset_idx.
Both kinds of task expose a vectorized gymnasium Env. Configuration uses Hydra with presets.
Learning, data, and teleop:
- RL libraries (in isaaclab_rl): RSL-RL, skrl, Stable Baselines 3, and RL Games.
- RLinf: "RL post-training capabilities for fine-tuning large foundation models." The docs have an experimental page on it: RL post-training for VLA models.
- isaaclab_mimic: Planners based on cuRobo, plus a full pipeline for generating datasets from human demonstrations.
- isaaclab_teleop: Supports OpenXR, CloudXR, gamepads, SpaceMice, and Haply devices, with retargeters for manipulators and humanoids.
Applications built on Isaac Lab: - Isaac Lab-Arena: Composes scenes, robots, and tasks into benchmarks, then evaluates generalist policies under controlled variations. - GR00T workflows: Use Isaac Lab for collecting demonstrations, generating synthetic trajectories, training policies, and evaluating generalist models in closed loop. - NuRec: Turns real-world captures into simulation-ready scenes for training and evaluation.
Running without Kit: URDF and MJCF conversion can also run without Kit, once the isaacsim-asset-isolated wheel is installed.
Docs pages worth reading later: - Physics: physics backends, backends and presets, solver differences, coupled solvers, using MPM, extending Newton solvers - Rendering: renderers, visualization - Scale: multi-GPU, HPC clusters, cloud, population-based training, Ray - Sim-to-real: policy deployment, with the gear assembly, COMPASS/NuRec, reach, LEAPP export, SO-101 typing, and G1 visual locomotion examples - Imitation learning: imitation learning - Migration: migrating to 3.0, release notes
Skild AI: Steel: Simulation for Generalist Robot Learning¶
Speaker and session not recorded. Slide photo taken at 10:43 am. Steel is also listed as one of Newton's solvers in the Isaac Lab 3.0 tech session above.
- What Steel is: A GPU contact solver built to simulate many different robots and tasks in one batch. (My reading: most GPU simulators batch many copies of one robot and one task. A generalist policy needs many embodiments and tasks in the same batch.)
- Why Skild built it:
- The recipe that worked for locomotion: Thousands of identical environments, one task, and one PPO policy.
- Where GPU simulators strain:
- Dense factorizations: Cost grows with the number of degrees of freedom. Example: a Kuka arm with an Allegro hand... (The slide was cut off here.) (My reading: Kuka's 7 joints plus Allegro's 16 give 23 DoF. Factorizing a dense matrix costs roughly the cube of the DoF, so high-DoF robots get expensive fast.) The layout slide below confirms 23 DoF for arm + hand.
- Built for heterogeneity: laid out per environment (from slide photo):
| Environment | Factor size | Active rows, out of a 192-row budget |
|---|---|---|
| Arm + mug | 7 DoF | 41 |
| Arm + gripper + box | 9 DoF | 64 |
| Arm + hand + cube | 23 DoF | 163 |
| Arm + part in a bin | 7 DoF | 85 |
| Arm alone | 7 DoF | 7 (no contacts) |
- Factors are grouped by exact DoF count. No robot is padded to the size of the largest one in the batch.
- Each environment has a fixed row budget. The solver's sweep visits only the active rows.
- The slide footer says the counts come "from one batch" and the row counts "from Figure 2 of the paper," so a Steel paper exists.
- (My reading: rows are probably constraint rows, one per joint or contact constraint. That explains why the arm alone needs only 7.)
Video to Data: Agentic Native Pipeline¶
NVIDIA. Speaker and session not recorded. Slide photos taken at 10:50 and 10:51 am. This is the same work as the "video to data" workload (CHORD, 4,739 tasks) in the Isaac Lab 3.0 tech session above.
- Summary to come: You'll summarize this pipeline later.
- Project overview: real to sim to real. These are the pipeline stages, from the slide photo:
- Human demo video: the input.
- Ingestion.
- Reconstruction.
- Human-object trajectory and simulation environments: a dataset.
- Robotic grounding, in Isaac Lab.
- Data augmentation, in Isaac Lab.
- Physics-grounded dataset.
- Foundation models.
- Real robot.
- Stages 2–6 sit in a dashed box labelled "Cosmos / GPT-6 Enhanced" (as the label reads in the photo). Each of those stages has a small agent badge. (My reading: that's what "agentic native" means here.)
- A second arrow runs straight from robotic grounding to the real robot, skipping the foundation-model step.
-
The example images are a person handling an object, the object in sim with a robot hand, and a humanoid doing the task.
-
CHORD (from slide photo): Robot Learning from Human Videos in Isaac Lab. CHORD stands for Contact Wrench Guidance from Human Demonstration in Robotic Dexterous Manipulation.
- The idea, in three steps:
- Learning from a human demonstration means imitating the object's 6D trajectory.
- Imitating the trajectory means imitating the object's 6D acceleration.
- Imitating the acceleration means imitating the contact wrench on the object.
- The rewards: an imitation reward and a contact-wrench reward, guided by reference contacts from the human demo.
- (My reading: the contact wrench is the force and torque that produce that acceleration, so matching it gives a physically grounded target rather than pose alone.)
Hybrid Simulation: Cosmos Neural with Isaac Lab¶
NVIDIA. Speaker and session not recorded. Slide photos taken 11:02–11:06 am. It connects to the "World models: SuperMimic and Cosmos" workload in the Isaac Lab 3.0 tech session above. The paper is Hydra-0.
- The idea: Combine known robot physics with learned environment dynamics. ("Dynamics" is the slide's wording.) (My reading: Isaac Lab simulates the robot's own dynamics, and a Cosmos world model predicts the parts of the scene that are hard to model by hand.)
- From an image to a pipeline ready for real to sim to real (slide photos):
- Automating the setup: "Agentic hybrid simulation with one image and one click." Asset discovery, scene assembly, and replay calibration happen in one agent-guided workflow.
- The demo video, "Fully agentic digital-twin setup": Isaac Lab ↔ Cosmos 3. A Codex agent takes a reference image (real bimanual grippers with a blue rope) and builds an Isaac Lab digital twin in three steps:
- Discover the URDF and assets.
- Assemble the digital twin.
- Calibrate and validate the physics.
- The physics is checked against recorded motion, through recorded-action replay. "Asset discovery, model assembly and calibration are performed by the agent."
- Hydra series: a shared action interface across embodiments (slide photos):
- The interface is object flow: System 2 (the agent) → object flow → System 1 (the policy). "The System-2 agent plans desired motion; the System-1 policy turns that flow into robot actions."
- Demo: Given one image and the instruction "Drag the purple cloth towards the left," the agent proposes an object flow: 29 points on the cloth, with motion trails. The flow is projected onto the unchanged starting image.
- A caption mentions "MolmoAct2" (unclear in the photo; check).
- Hydra-0: Action Flow for Generalist World Modeling and Control, an NVIDIA technical report from August 2026. One model trains across embodiments, simulates them, evaluates policies, and drives a real robot. See the Hydra-0 notes.
FOVI-Dex: foveated perception for dexterous manipulation in simulation¶
Nicholas M. Blauch et al. (with Karl Van Wyk, Nathan Ratliff, Ankur Handa, Gavriel State, and others). Slide photos taken 11:16–11:31 am. The slides were shown over a Teams screen share. The title slide lists the Isaac Lab Scaling Workshop and the Dex Manipulation Workshop at CoRL 2026. Everything here is from slide photos.
- Full title: FOVI-Dex: Foveation enables efficient perceptive learning of dexterous manipulation in simulation.
- The usual approach to sim-to-real for vision:
- Teacher–student distillation: A teacher policy trained with RL on privileged simulator state is frozen. A vision-only student, given RGB images, learns by copying the teacher's actions. (This slide was photographed at 11:16, just before the title slide, so it may belong to the previous talk.)
- "Vision-based policies usually require state estimation or distillation from state-based policies." The slide gives two examples:
- SILO (Tao et al., 2026): stereo RGB from a ZED 2 → FoundationStereo depth plus SAM 2 masks → point-cloud processing and a polyline fit → a sim digital twin.
- Play2Perfect (Lum, Kehdia, et al., 2026): pose estimation of the part and fixture, plus goal poses derived from CAD, feeds a fine-tuned policy that outputs joint targets for the arm and hand.
- "Perception limits the scalability of parallel experience":
- The setup is one 300 × 480 fisheye camera, rendered with RTX in Isaac Lab 3.0 (PhysX).
- Policy: a DINOv3-S+ vision backbone and a recurrent LSTM policy with a 24-step horizon. It runs at 15 Hz with 4-step action chunks, which gives 60 Hz commands to a controller based on Geometric Fabrics.
- Scale: Calibrated to 45 GB of memory on an L40S, that's only 208 environments per GPU, 832 on 4 GPUs and 1,664 on 8.
- (My reading: rendering cameras is the bottleneck. Compare that with the thousands of environments per GPU that state-based RL gets.)
- Jointly learning vision and motor policies in simulation:
- Where to look: A vision policy predicts a fixation point. It's trained by visual target distillation, to match a teacher fixation taken from the simulator's target.
- How to move: A motor policy outputs arm and hand actions. It's trained with PPO on a grasp, lift, and hold reward.
- Humanoids combine software gaze and physical motion: "Software eyes move first; the physical neck restores coverage."
- The example is hybrid eye-head gaze on a Galbot humanoid, with left and right fisheye "retinal views."
- The readout showed eye-head 0.5°, vergence 4.4°, and 87% valid retinal samples per eye.
- The slide also mentions EyeRobot2 (partly hidden in the photo).
- (My reading: foveation means rendering or processing only a small high-resolution patch where the policy looks. That cuts the per-environment camera cost that limits scale above.)
NVIDIA Isaac Lab-Arena: one evaluation framework¶
Speaker not recorded. Slide photos taken 12:22–12:36 pm. Everything here is from slide photos.
- "One Common Evaluation Framework for Every Evaluation." Isaac Lab-Arena sits on Isaac Lab 3.0, with PhysX or Newton underneath. It has three jobs:
- Author: "Agents author, extend and scale principled benchmarks for robustness, with ease."
- Execute: "Repeatable, reliable, large-scale parallel evaluation. More coverage in less time."
-
Analyze: "Beyond a gate: where and why a policy fails, to inform the next round of learning."
-
Users and benchmarks shown on the slides:
- Model developers: RLWRLD (RLDX-1) and NVIDIA (GR00T).
- ISVs: Lightwheel and BIFROST.
- OEMs: JD.com, Samsung, and X Square.
- Benchmarks: AUTOLAB's VABAR, Lightwheel RoboFinals, RLWRLD DexBench, RoboDojo, NVIDIA G1 Factory, RoboChallenge, Sharpa, and X-Square.
- Author once, then evaluate or learn: A benchmark written for Arena can be evaluated right away. It can also be plugged into Isaac Lab through an environment-registration callback, for teleop, Mimic, or RL training, and then evaluated.
- The future plan, a unified flywheel: Real2Sim, CAD2USD, and V2D feed physics calibration and SimReady tooling. That fills a USD library and unified Isaac environments, which drive the loop of data generation, training, evaluation, SIL, and deployment, and then sim to real.
- Analyze why and where a policy fails (available in v0.3):
-
The why, by sensitivity analysis: Arena perturbs one factor at a time, such as HDR background, light colour, colour temperature, light direction, cube mass, and wrist-camera position.
Policy Stable to Sensitive to π0.5 cube mass (0.1–0.9 kg), lighting (50–150%) wrist-camera displacement: fails beyond +2 cm GR00T N1.7 wrist-camera Y (±5 cm), cube mass low light: fails below 70% -
The where, by subtask predicates: progress is tracked inside every rollout. One example policy had 20% success, and the stages showed where it dropped:
Grasp Lift Transport Place 30.3% 22.9% (−7.4 pp) 21.3% (−1.6 pp) 20.0% (−1.3 pp) "Failure stage: Lift." 75 rollouts grasp the object but never lift it. - Benchmarks already on Arena: - VABAR (UC Berkeley AUTOLAB): variational automation benchmarks for agentic robotics. See Goldberg's talk below. - DexBench (RLWRLD): multi-finger dexterity. See below. - RoboDojo: bimanual manipulation. "Now in Arena, DexDojo next." - RoboFinals (Lightwheel): 100 industrial-grade tasks. - Where they're headed, with questions for the audience: - Author: benchmarks that resist overfitting. "What makes a benchmark principled?" - Execute: high-throughput evaluation for any policy, with heterogeneity from objects to robots and batched evaluation of agents as policies. - Analyze: dense failure labels and agentic auto-grading. - Sim-to-real correlation as a science: "How would you bound sim-to-real correlation?"
RLWRLD: DexBench¶
Speaker and session not recorded. Slide photos taken 12:40–12:47 pm, right after the Isaac Lab-Arena talk. DexBench runs on Isaac Lab-Arena.
- The evaluation bottleneck: There are many methods to try, and too few experiments we can run.
- New models, training recipes, and inference strategies all need real-world evaluation, with its resets, failures, and repeated trials, before they become reliable skills in the field.
- "Costly evaluation means slower iteration toward deployment."
- What DexBench cares about (from slide photo):
- Real industrial work: Every task comes from real workflows in retail, logistics, manufacturing, and hotel operations.
- Beyond the tabletop: Shelves, conveyors, carts, and workstations, with clutter, tight gaps, and occlusion.
- Built for dexterity: Five-finger hands, precise insertion, and both hands working together.
- Real assets, real conditions: Objects modelled to spec from real parts and products. Policies are scored on objects and layouts they've never seen.
- Physically diverse tasks: Rigid, contact-rich, and deformable objects. "Bags fold, paper flexes, liquid sloshes."
- "Each of these exists somewhere. All five together, nowhere else."
- What's in the box:
- Code: Isaac Lab-Arena environments for all 15 tasks, an evaluation runner, and frozen evaluation conditions.
- Assets: USD scenes and fixtures, plus about 80 real-object SimReady assets, validated in Newton.
- Data: multi-finger demo datasets per task and for multitask, in LeRobot format.
- Baselines: GR00T N1.7, Cosmos 3, FastWAM, and RLDX, with open checkpoints and serving code.
- Leaderboard: public, with the same frozen conditions for every submission.
- How to run it, in four steps:
- Clone the repo and pull Isaac Lab-Arena and the assets. "One container, one command."
- Serve your model. GR00T serving code ships with it, and any model fits behind the same
PolicyBaseadapter. - Run a cell. Frozen start states replay identically for every model:
run_benchmark_cell.py --task r2 --cell train --model gr00t. - Score and submit: per-cell success with 95% intervals, using
score_benchmark_cells.py. - Links: GitHub · Hugging Face · project page · Manifold docs
Ken Goldberg: Agentic Robotics¶
Session and time not recorded. UC Berkeley AUTOLab and NVIDIA. Project page. The main paper is GaP, a CoRL 2026 Spotlight.
- The pipeline: It takes one real demonstration as input and produces ROS 2 code that runs reliably on a real robot. It has three stages:
- Real2Sim: Inverse physics builds a simulation from the demonstration.
- Code generation: Self-learning in simulation produces a "graph-as-policy" program that works across the variation ranges.
-
Sim2Real: Real rollouts test the exported code, with no agent running on the robot. Failure videos go back either to the simulation or to the code.
-
"A third path between two cultures" (from a photo of the write-up on screen, 12:51 pm): Agentic Robotics links model-based and model-free robotics.
| Model-based engineering | Model-free and VLA | Agentic Robotics | |
|---|---|---|---|
| Strength | Fast, interpretable, and highly reliable | Potential to adapt more broadly | Frontier-model generality with the reliability of executable programs |
| Cost | Substantial human engineering for each new task | Not yet reliable enough for industrial uses | not given |
- What each stage takes in and uses:
| 1. Real2Sim | 2. Code generation | 3. Sim2Real | |
|---|---|---|---|
| Input | Video of a person doing the task; the desired cameras, sensors, and robot specs; task geometry and materials | Task variation ranges, and a detailed parameterized Newton simulation | ROS 2 graph or code for the desired sensors and robot |
| Tools | RSS, Newton, SuperDex | GaP skill library, GaP harness, frontier LLM | Real sensors and robot |
From the project page¶
Source: the Agentic Robotics page, not the talk.
- The core idea: "Given a task description, a frontier LLM and VLM system composes a library of clearly defined, modular robot skills, which can include VLA policies, into an interpretable robot control system."
- "Block 2" is a numbered step in the system diagram. The numbers follow Goldberg's revision of the research agenda, dated 21 September 2026:
- Block 1: the human demo.
- Block 1B: the variation ranges. These are changes in pose, objects, environment, and lighting. The page gives no numeric ranges.
- Block 2: agentic inverse physics, meaning "an agent fits a physics model that reproduces a real video."
- Block 2A: RoboSimStudio.
- Blocks 5 and 7: real rollouts, and comparing sim with real.
- Stage 1, Real2Sim:
- Inputs: a video of a person doing the task, the robot URDF, the cameras and sensors, and the task geometry and materials.
- Output: a parameterized Newton simulation. The agent compares the simulation with the real video and revises it.
- Tools: RSS, Newton, and SuperDex. The figure lists OpenAI, Claude, and Gemini as the agents, without saying which does what.
- Stage 2, code generation:
- Graph-as-policy: "a robot policy represented as a graph of modular skills." Example modules are perception, grasp selection, motion planning, force control, and failure recovery.
- How it learns: Recursive self-improvement, which aims to maximize success rate and minimize cycle time. "Each round, the agent runs hundreds of episodes, groups the failures and rewrites the code."
- Tools: a frontier LLM, the GaP skill library, the GaP harness, and Newton.
- Stage 3, Sim2Real:
- "The robot runs exported, lightweight, ROS2-compatible code and never a frontier model in its inner control loop."
- Failures come back as videos, either to re-model the environment or to revise the code. The page doesn't say how that choice is made.
- Robots: Franka, bimanual YAM, bimanual UR5e, and ABB YuMi. "Every policy on this page is self-learned with the Agentic Robotics framework: zero lines of code from humans."
- Related projects:
| Project | What it is | Links |
|---|---|---|
| GaP | The graph-as-policy harness. CoRL 2026 Spotlight. | paper notes |
| RoboSimStudio | A toolkit for fast Real2Sim, for humans and agents | arXiv 2610.18835 · page · code |
| VABAR | A variational-automation benchmark: 6 task classes (tool sorting, tool hanging, gear meshing, syringe packing, cable insertion, cable routing), two difficulty levels, and a 33-skill library on Newton | page · code |
| FlipAnything | A bimanual YAM policy, written by a coding agent, that turns any tool face down. 24.7% → 95.0% success on 24 tools, over 36 policy versions, with 360 simulated episodes per round. CoRL 2026 late-breaking result. | page |
| Push-Alphabet | Two UR5e arms push seven letter blocks into a row that spells AUTOLAB, revisiting the Push-T task | arXiv 2608.18227 |
| Cable untangling | ABB YuMi | page |
- Limitations, as the page states them:
- "Physics models remain imperfect."
- Self-learning needs to become more efficient.
- Safety-critical uses need stronger verification and validation.
- "Coding agents are slow," which suits offline policy development but not real-time adaptation.
- It is "not yet known whether the architecture scales to open-ended generalist robots or humanoids."
Keynote: Fei-Fei Li, Atlas: A Unified World Model for Generation and Reconstruction¶
Fei-Fei Li: Stanford, and founder and CEO of World Labs. About 3:10 pm, followed by a fireside chat with Lisa Su (AMD). This section uses only public sources: the CoRL 2026 keynotes page, World Labs' Atlas blog post (Sept 1, 2026), and Rapidata's public camera-movement leaderboard.
- The abstract: Atlas is "an autoregressive frame generation model that unifies pixel generation and 3D scene reconstruction."
- What Atlas is, in World Labs' words: "an omni model designed to handle many tasks and many kinds of input and output data in a single unified architecture." It's a multimodal autoregressive diffusion transformer, pretrained from scratch on text, images, video, and 3D.
- Inputs today: text, images, camera poses, and 3D depth maps.
- Outputs: images and video "with pixel-perfect camera control," new views from any camera position, and worlds as point clouds or 3D Gaussian splats. Video runs up to 1 minute at 1440p.
- Results World Labs reports:
- It outperforms recent video models at camera-controlled generation. Atlas is #1 on Rapidata's public camera-movement leaderboard.
- It outperforms the best specialized open-source reconstruction models on seven benchmarks: DTU, ETH3D, KITTI, NRGBD, 7-Scenes, Tanks and Temples, and ScanNet.
- Why it matters for robots: World Labs says a world model can "render imagined worlds for creative users, simulate the real world in high fidelity, and help robots plan actions." It also provides diverse training data and test environments for robotics at scale.
- (Link to the Day 4 workshop Do Robotics Need World Models?: this keynote argues for world models, and Day 4's long-context talk argued against needing them.)