Human data and scaling¶
Cross-day theme. Facts come from the linked day and paper notes. Sentences marked "my reading" are interpretation.
Robot data is expensive and split across many embodiments. Human data is abundant and comes from one embodiment. Most of the week's scaling stories were about how much of the gap human data can close, and how to map human motion onto robots.
How much data, and of what kind¶
| Work | Day | Human data | Robot data | How human data is used | Key result |
|---|---|---|---|---|---|
| Being-H0.5 (BeingBeyond) | 1 | 16K h of human demos | 14K h over 30+ embodiments (+ 5K h-equivalent vision-text) | human hand and robots write into one shared action space | with pre-training, the generalist is within 2 points of specialists (66.4 vs 68.6) |
| DreamZero (NVIDIA) | 1 | 12 min of human video | AgiBot, DROID | video-only transfer, no action labels | unseen tasks 38.3% → 54.3% (+42%) |
| LVP | 1 | video model trained on video | Franka, G1 | generate a video plan, then retarget human hands | 59.3% task completion on 100 third-party tasks |
| Meta Robotics Studio | 2 | egocentric video (Aria) at the base of the pyramid | UMI grippers, then teleop | "learn from people," then practice in sim | not shared |
| Tedrake / Walden | 2 | egocentric human video in the mix | robot demos, UMI; then deployment data | 1k / 10k / 100k h of external pretraining, 10 h per task | LBM 2.0 at 340.3 s between interventions |
| AGIBOT World | 2 | none mentioned | open robot datasets (2024, 2026) | shared data for the field | Genie Evolver across 96 robots |
| Video to Data and CHORD (NVIDIA) | 3 | human demo videos | generated in sim | reconstruct the object's motion, then imitate its contact wrench | CHORD: 4,739 tasks (as shown) |
| Dyna-2 (Dyna Robotics) | 4 | 1M+ h of human video, some with hand poses | none in pre-training | world-action model; zero-shot transfer to robots | action MSE 0.180 → 0.117, accuracy 0.067 → 0.159 (1k → 1M h) |
Mapping human motion onto robots¶
The data pyramid, and where deployment fits¶
Takeaways¶
- Scaling laws on human video now exist. Dyna-2's zero-shot robot error falls with human hours, but the curve is flat from 1k to 10k hours and only improves sharply by 100k. My reading: There may be a data threshold below which human video doesn't help much.
- The hard part is the mapping, not the data: shared action slots (Being-H0.5), retargeting generated hands (LVP), action-free video in a world model (DreamZero, Dyna-2), and contact wrenches (CHORD).
- Tedrake's counterpoint: In the limit, deployed robot data will dwarf everything else. Stronger base models make deployment data more valuable, not less.
- Action labels are the scarce part. The Day 4 remote talk's first data premise: if action labels are scarce, learn from video without them.
Papers tagged human-data or cross-embodiment¶
- Being-H0.5: 35k h across 30 embodiments with a unified action space; 98.9% LIBERO
- DreamZero: 14B video model as policy; over 2× generalization vs VLAs at 7 Hz
- Dyna-2: World-action model pre-trained on 1M+ hours of human video only; zero-shot robot action MSE 0.180 to 0.117 from 1k to 1M hours
- LVP: 14B video planner for zero-shot robot control; 59.3% task completion on 100 third-party tasks vs Wan 2.1, Hunyuan, Cosmos
- Being-H0.5: 35k h across 30 embodiments with a unified action space; 98.9% LIBERO
- DreamZero: 14B video model as policy; over 2× generalization vs VLAs at 7 Hz
- Hydra-0: World model conditioned on action flow (robot actions as pixel motion); runs as a hybrid simulator, with physics for the robot and video for the world
- LVP: 14B video planner for zero-shot robot control; 59.3% task completion on 100 third-party tasks vs Wan 2.1, Hunyuan, Cosmos
- π0.7: Metadata turns low-quality data from harmful to helpful; zero-shot UR5e shirt folding at teleoperator level
- TAM: Sim-only residual torque layer below any policy; real Panda box pushing 47.6% to 76.2%, BC flip 50% to 72%