Day 4: Thu, Nov 12, 2026¶
Workshop day.
Sessions attended¶
- Workshop: Do Robotics Need World Models? Organizers and room not yet recorded. Talks photographed at 9:15–9:32 am and 10:03–10:06 am. Notes under Live notes.
- Michael Lutter (Boston Dynamics): Sim2Real2Field at Boston Dynamics: The Unglamorous Part of Scaling Humanoids, in the Sim2Real2Field workshop, 3:41–3:59 pm from slide photos. Notes under Live notes.
- Dyna Robotics: Ingredients of Scaling Foundation Models in Deployments. Speaker not yet recorded. About 2:45–3:05 pm. Notes under Live notes.
- Workshop: Sim2Real2Field: A Model-Based Perspective. Organizers and room not yet recorded. The talk ran 2:17–2:29 pm. Notes under Live notes.
On the schedule, from Boston Dynamics' Day 2 slide (Day 2's Boston Dynamics notes):
- 9:50 am, Everything Beneath the Policy workshop: Yeuhi Abe and Robin Deits (Boston Dynamics), Our Journey from MPC to RL.
- 2:55 pm, Sim 2 Real 2 Field workshop: Michael Lutter (Boston Dynamics), Sim-2-Real RL for Manipulation. It was given at about 3:40 pm as Sim2Real2Field at Boston Dynamics: The Unglamorous Part of Scaling Humanoids. Notes under Live notes.
Papers covered¶
| Paper | Group | Headline | Status | Links | Weights |
|---|---|---|---|---|---|
| Dyna-2 | Dyna Robotics | World-action model pre-trained on 1M+ hours of human video only; zero-shot robot action MSE 0.180 to 0.117 from 1k to 1M hours | abstract | page | ? |
| FARM | UC Berkeley, Stanford | Real-time object-level relational 3D memory answers relational language queries; +164% Recall@5 over prior methods (unverified) | abstract | arXiv · page · code | ? |
| π0.7 | Physical Intelligence | Metadata turns low-quality data from harmful to helpful; zero-shot UR5e shirt folding at teleoperator level | summarized | arXiv · page | — |
Day 4 themes¶
Cross-day versions: world models, memory, deployment, simulation.
- Do robots need world models? The long-context talk said no: memory, subtasks, and prompting are enough. The remote talk framed VLAs and world action models as two ends of one design, and Dyna-2 scaled a world action model on human video.
- Memory in two tiers: dense video for the recent past, and compressed text (π0.6-MEM) or a 3D object graph (FARM) for everything else.
- The unglamorous parts of scaling: Dyna's recovery arithmetic and worn-gripper lesson, and Boston Dynamics' daily Atlas Check and fleet logging.
- Making RL work for real robots: demo resets and a frontier curriculum (Boston Dynamics), and model-based safety filters with residual RL (ResSafe).
Live notes¶
Workshop: Do Robotics Need World Models?¶
Organizers, room, and time not recorded. The title is as you typed it. Check whether the program says "Robotics" or "Robots."
Talk: Long-Context, Long-Horizon, Compositional Generalization without World Models¶
Speaker not recorded. Given in the Do Robotics Need World Models? workshop, with slides photographed 9:15–9:32 am. The memory slides cite Kim, Wang, Sridhar, Muppidi, Torne, Finn, Long-Context VLA: Rea[...] Robot Control, 2026. The middle of the title was hidden in every photo. With Chelsea Finn as last author and π0.6/π0.7 as the models, the speaker is probably from Physical Intelligence or Stanford, but that's unconfirmed. The title argues the "no" side of the workshop question.
- The question: How can we apply robots to complex situations? The talk names three capabilities, each with an example from the slide:
- Handling long context, such as tracking what happened.
- Completing full long-horizon jobs, such as cleaning the kitchen.
- Reasoning in new situations, such as applying a task to a different robot.
1. Handling long context
- Many robot foundation models don't actually have memory or context. That's fine for repetitive, short motor skills.
- Memory is critical for tracking.
- Key challenges:
- Spurious correlations between past observations and future actions. Sometimes adding memory makes the policy worse.
- Substantial compute, for both training and inference. This gets harder as the context grows.
- Why existing approaches fall short:
- π0.7's context length is insufficient for these tasks. Its video history covers about 6 s: up to 6 frames per camera at a 1 s stride.
- Keyframes don't provide enough information for tracking. ("Keyframes" is my reading. Your note said "Key.")
- "Can we extend memory of VLAs to hundreds of frames?" (slide photo):
| Problem | Answer |
|---|---|
| Training signal over so many images | Predict subtasks, and narrations of the memory. |
| Inference and training speed | Cache and reuse past computation while streaming: the slide shows "cached history · KV + recurrent state reused." Decode the memory narration in parallel, in one forward pass. Group actions so fewer forward passes are needed. |
- The model, from the slide:
- Backbone: a Qwen3.5-4B VLM with a 262K-token context.
- It outputs three things:
- Reasoning: a narration of the memory, decoded in parallel in one forward pass. Example: "At 2 s, the ball is placed under the middle cup. At 5 s, the middle and right cups swap. At 9 s, …"
- Subtask: for example, "lift the left cup."
- Action tokens: grouped as OAT with group sizes 1, 1, 2, 4, 8, and 16, then decoded into the action chunk at:t+H.
- Conclusion of part 1: "End-to-end training with subtask, narration prediction yields strong long-context reasoning."
Even longer memory: a VLA with multiscale memory
- Two tiers of memory:
| Medium-horizon memory | Longer-horizon memory | |
|---|---|---|
| Form | Visual: dense and based on observations | Text: sparse and semantic |
| Examples on the slide | Espresso machine, crepes, window wiping | Grocery aisle, kitchen stove |
- Multi-scale embodied memory: A VLM keeps a long-term text memory, such as "I wiped the counter with the sponge and dried it with the...", and updates it with a memory summary on each step. It also takes a short-term video memory of recent frames, and outputs a subtask. A VLA turns the subtask into robot actions.
-
Example prompt: "Clean up the kitchen. Wipe the countertop with the sponge and dry it with the paper towel and throw the towels in the trash under the ..."
-
Results: task progress (%), with values read by eye off the bar chart, not printed on the slide:
| Task | π0.6 (no memory) | Video memory only | Naive text + video | Text memory only | π0.6-MEM |
|---|---|---|---|---|---|
| Recipe set-up | ~40 | ~29 | ~36 | ~31 | ~65 |
| Clean kitchen | ~23 | ~38 | ~46 | ~33 | ~93 |
| Average | ~42 | ~38 | ~45 | ~38 | ~78 |
- "Text summaries offer strong improvements on 15+ min tasks over no text memory or naive text memory."
- Compressed text memory expands the task horizon.
- (The results slide compares π0.6 variants. Check whether the speaker said π0.6 or π0.7 when calling the context length insufficient.)
3. Compositional generalization to new, out-of-distribution situations
- The tried-and-tested recipe for generalization:
- Sufficiently large and diverse data.
- Models with enough capacity.
- "Let's use all the data we have!" Diverse robot demonstration data. (The rest of the list was cut off in your notes.)
- Key idea: detailed, multimodal prompting. (Your note had "Prompt the model with..." cut off. This matches π0.7's metadata, subtask, and subgoal-image prompts.)
- Out-of-distribution test: Can a bimanual UR5e fold clothes without any folding data on that robot? The slide showed an arm folding a blue shirt.
- This is π0.7's zero-shot cross-embodiment result. The paper reports 85.6% task progress and 80% success, against 90.9% and 80.6% for expert teleoperators. Full numbers are in the π0.7 notes.
- Takeaway: Strong generalization.
Overarching takeaways (slide photo, 9:31 am)
- "With the right data, supervision, & context, models can reason through complex situations."
- "No need for explicit mechanisms that explicitly build in a model of the world."
- "Simple recipes often scale more readily!"
Remote talk: VLAs, world action models, and two data premises¶
Speaker not recorded. A remote talk over Zoom in the Do Robotics Need World Models? workshop. Slides photographed 10:03–10:06 am. Everything here is from slide photos.
- What do robot foundation models have in common? A simplified comparison of a VLA and a world action model (WAM):
| Backbone | Passes to the action expert | |
|---|---|---|
| VLA | VLM | Language in latent form: what to do |
| WAM | Video model | Video in latent form: how the world should change |
-
The backbone side means "more semantics, more System 2," pretrained with world knowledge. The action-expert side means "more dynamics, more System 1," fine-tuned with embodied actions.
-
"Performance, generalization, scale: all still data problems." The question is which part of the model learns from which data, and how available that data is.
- Two data premises:
- Action labels are scarce, so use video data without action labels.
- Successful demonstrations are scarce, so use counterfactual and failed rollouts.
- Data premise 2 in detail: Train with local context only, meaning the current frame plus future video or actions:
- Inverse dynamics (IDM): generate actions conditioned on video.
- Forward dynamics (FDM): predict video conditioned on actions.
- Finding: "less context, better dynamics." Given language and history, the model can lean on the task instead of learning how actions change the world.
- (Link: compare the long-context talk just before this one, where more context helped. Here, less context helps the dynamics model.)
Workshop: Sim2Real2Field: A Model-Based Perspective¶
Organizers and room not recorded. The talk below ran 2:17–2:29 pm, from slide photos, and no title slide or speaker was photographed. "A model-based perspective" may be this talk's title rather than part of the workshop's name, so check the program. It isn't Boston Dynamics' Michael Lutter, who was scheduled here at 2:55 pm with Sim-2-Real RL for Manipulation. Lutter's talk has its own section below.
- Spatial memory matters.
- How the robot builds its memory, and how it queries that memory, both matter.
- Robots can benefit from relational spatial memory. (My reading: memory that stores how objects and places relate to each other, rather than raw frames.)
- Paper to keep in mind: FARM: Find Anything using Relational Spatial Memory.
- FARM's memory, from the slides: Each object node stores attributes (3D Gaussians, features, views, captions, and more). Objects are joined by covisibility and adjacency edges only, with no higher-order relations. Details in the FARM notes.
- Examples shown: a multi-floor house (~1,600 m²), "Find the tall lamp below the dartboard and to the left of the poster." A construction site (~15,000 m²), "Find the blue portable toilet with 'TOI TOI' branding closest to the white DBE-labeled delivery van." A 4,000 m² warehouse mapped live on a Spot at 10.6 Hz.
- Model-based safety filters can help. The filter checks each action the task controller proposes, and blocks or adjusts unsafe ones before they run.
-
The architecture on the slide:
Task controller ──▶ Safety filter ◀──▶ System- The task controller (the policy) only sends commands to the safety filter. It doesn't drive the system directly.
- The safety filter and the system talk both ways. The filter sends safe commands to the system and gets the system's state back.
- (My reading: the filter sits between the policy and the robot, so the policy can be swapped or retrained without touching the safety layer.)
- The filter, formally (slide photo): The controller sends an unfiltered command uref(t). The filter returns the closest safe command, solving minu ‖u − uref‖ subject to u ∈ K(x). K(x) is the safety constraint, a set-valued map from the state x to the allowed controls. The system then follows ẋ = f(x, u).
- The slide's question: "Can we directly design K(x)?"
- Safety analysis, conditioned on the control input: The Hamilton-Jacobi-Isaacs PDE, min{∂V/∂t + H(z, ∇V), l(z) − V(z, t)} = 0 with H(z, ∇V) = minu ∇V·f(z, u), defines the safe set. "But we only need to guarantee that ∇V·f(z, u) ≤ 0." (My reading: any control that doesn't push the state out of the safe set will do, which is what makes K(x) a set and not a single action.)
- Residual reinforcement learning: ResSafe (Gechen Qu et al., CoRL 2026).
- Framework: A reference policy, a motion-tracking policy for a humanoid, proposes πref(ot), which may be unsafe. A learned residual nudges it across the implicit safety constraint: πsafe(ot) = πref(ot) + Δπθ(ot, πref(ot)). It runs in sim and on the real robot.
- Training: offline, over a distribution that ranges from easy to difficult motions, from safe to unsafe reference policies, and across domain randomization of the environment.
- Reward: survival time inside the safe set (reachability), plus a minimum-norm penalty on the residual (filtering), so it changes the reference only as much as needed.
- (Related: Day 2's TAM is also a residual layer under any policy, but for torque adaptation rather than safety.)
-
Multi-agent interaction models are important. (These points come from your notes only. No photos were taken of this part.)
- Models can take closed-loop behavior into account: how each agent reacts to the others.
- That enables predictable, safe, high-confidence interaction between agents, even when the other agents' specification is unknown.
- Multi-agent RL with layered safety. (My reading: RL learns the interaction behavior, and model-based safety layers like the filter in point 2 keep it safe.)
Michael Lutter (Boston Dynamics): Sim2Real2Field at Boston Dynamics: The Unglamorous Part of Scaling Humanoids¶
Sim2Real2Field workshop. Slides photographed 3:41–3:59 pm, so it ran later than its scheduled 2:55 slot, where it was listed as Sim-2-Real RL for Manipulation. Your notes, checked against slide photos. Where a photo added detail, that's marked "slide."
Part 1. Real2Field: scaling from 4 to hundreds of humanoids
- The goal: no "favorite robots." The slide's example of the problem: "I want to run my experiments on HD-003." Every robot should behave the same, so results don't depend on which unit you used.
- All robots have to match the model.
- Robot health has to be easy to measure.
- "Is my robot working as expected?" The questions to answer for every robot:
- Did any parts bend when it fell last night?
- Does it have the latest operating system?
- Are all motors working and calibrated?
- Is the camera calibration correct?
- Is the simulation model correct?
- Run an "Atlas Check" on every robot every morning.
- The robot performs a 5-minute routine that checks its calibration. The slide shows Atlas in front of a calibration board.
- If it passes, it's considered healthy.
- Double-check everything: Log, analyze, and visualize all the data you can record, including the firmware version and the policy ID.
- Slide example: side-by-side videos of the same DeepMimic policy, one with a firmware bug and one with the firmware fixed.
- "Is my new policy better than the previous one?" First: what does "better" even mean?
- Fewer falls, and falls that cause less damage.
- More comfortable for the robot pilots.
- Smaller tracking error.
- How much data do we need to collect to tell?
- Lower failure rates require collecting more data. (My reading: when failures are rare, you need many more runs to measure a difference with confidence.)
- There's more data from the old policy than from the new one.
- Robot pilots might have adapted to the old policy. (My reading: both of these bias a head-to-head comparison toward the old policy.)
- The slide shows a time-series chart next to a teleoperated Atlas video. The caption was partly hidden: "Teleoperated Jen[...] Speed."
Part 2. Sim2Real: reinforcement learning for manipulation
- Teleop for multi-fingered hands is hard. The slide shows a teleoperated hinge insertion on Atlas.
- Use demos to narrow the search space.
- Exploration is hard: Policies often don't learn with sparse rewards. Tuning dense rewards is laborious and tedious. Policies learn to exploit the reward.
- Use demo resets to guide the exploration: Collect human or agentic demos for the task, then start exploration from states along those demos. Demos are just a hint to the policy about where to search, and policies can diverge from the demo distribution.
- Teacher training with reset-state initialization: human demos → demo states → reset the simulator to a demo state → teacher RL. When an episode ends, a new start state is sampled. Starting always from s₀, exploration stays near s₀. Starting from demo states, exploration covers the whole task.
- Demo resets are more common in the literature. Papers named on the slide:
- 2018: DeepMimic (Peng et al.), Overcoming Exploration (Nair et al.)
- 2024: DemoStart (Bauza et al.), RFCL (Tao et al.)
- 2026: OmniReset (Yin et al.), SGS (Zhang et al.)
- Frontier reset curriculum:
- Frontier states are states from which the policy can complete the task.
- Sample reset states near the frontier, where the policy isn't yet successful.
- Advance the frontier when the policy completes the task from there.
- Augment frontier samples with uniform resets to increase diversity (slide; your note was cut off here).
- Hyperparameters: success-rate threshold, window size, and the frontier-to-uniform ratio.
- How the frontier moves: The slide's grids of demo time show resets starting late in the demo and moving toward the beginning as training goes on. This confirms my earlier reading.
- Results: Atlas doing cup stacking, block stacking, and plug-in-socket tasks.
- Preliminary results for frontier sampling (one task):
- Frontier sampling reaches 96.5% mean evaluation success by about 600M environment steps. It passes the 90% target at about 300M steps.
- The baselines all stay below about 60% over the same steps: SGS as in the paper, SGS matched, SGS with κ = 10, SGS Pedro, and uniform resets. (Read by eye off the chart.)
- Heatmaps of start position within the demo, over training, show the frontier covering the whole demo, while SGS and uniform resets cover less of it.
- The slide's own caveat: "Comparison only includes one task and is missing baselines!"
Summary (slide)
- Real2Field, scaling from 4 to hundreds of humanoids:
- It takes a village to make sure all robots work as expected at all times.
- Log, analyze, and visualize all the data you can record, to make sure each robot is working as expected.
- The unglamorous work of calibration, testing, and root-causing problems becomes super important.
- Comparing two policies becomes hard, because there are so many confounding factors and data is scarce.
- Sim2Real, reinforcement learning for manipulation:
- Teleoperating multi-fingered hands is hard, and usually gives suboptimal demonstrations.
- RL can learn from suboptimal demonstrations and discover better behaviours.
- Guide exploration with demonstrations, to minimize reward tuning and reward hacking.
- Simulation tools and vision encoders have improved so much that RGB policies are feasible.
- (Link: Part 1 is the fleet-operations side of sim-to-real. A policy trained against one simulation model only transfers if every real robot still matches that model. It also fits Boston Dynamics' Day 2 theme, "data quality before scale," and its slide "for very dexterous manipulation, RL is hard to beat.")
Dyna Robotics: Ingredients of Scaling Foundation Models in Deployments¶
Speaker not recorded. Given at about 2:45–3:05 pm. This section uses only Dyna's own public reports, which the talk pointed to: - Dyna-2, August 2026 - Dyna-2.1, September 29, 2026 - Scaling customer deployments, August 2026
- The talk's themes: learning from human data, and a flywheel between research and deployment.
- Firsts Dyna claims publicly (each hedged "to the best of our knowledge"):
- Aug 2026: a 1-million-hour world-action model and scaling law (Dyna-2).
- Aug 2026: the first commercial manipulation pilot taken all the way to scaled deployment.
- Sep 2026: "the first physical agent to perform hour-long, non-linear loco-dexterous manipulation workflows" (Dyna-2.1).
Dyna-2: a 1-million-hour scaling law for world-action models
- A world-action model pre-trained on more than one million hours of egocentric human video. Some clips have 3D hand-pose tracks, and the rest have no action labels.
- It shows a scaling law on held-out human data and, Dyna says, the first human-to-robot transfer scaling law: zero-shot on 39 robot tasks never seen in pre-training.
- Human-to-robot transfer (scaling-law models pre-trained on human data only):
| Pre-training human hours | 1k | 10k | 100k | 1M | Fit |
|---|---|---|---|---|---|
| Zero-shot action MSE ↓ | 0.180 | 0.174 | 0.124 | 0.117 | MSE = 0.306 · D−0.0713, R² = 0.884 |
| Zero-shot accuracy@0.5 ↑ | 0.067 | 0.074 | 0.136 | 0.159 | acc = 0.0241 · D0.139, R² = 0.918 |
- The report notes that its production Dyna-2 models use different training recipes from these scaling-law variants. Dyna-2.1's policy mixes the human video with robot data from Dyna's fleet.
- Post-training: "With just a few hours of robot data alone in post-training." One example: "a mere 10 minutes of teleoperation data was enough to fine-tune Dyna-2" to open bottle caps.
- Details are in the Dyna-2 notes.
Dyna-2.1: a physical agent for end-to-end workflows
- Taku is a semi-humanoid robot: a 21-DoF upper body with two 7-DoF arms, on a mobile base with four steerable wheels.
- The stack, from top to bottom:
| Layer | What it is | Rate |
|---|---|---|
| Workflow orchestrator | vision-language model; tracks the workflow and decides the next step | not given |
| Dyna-2 policy | world-action model | ~5 Hz |
| Whole-body controller | RL, trained in simulation | 100 Hz |
| Taku | 21-DoF upper body plus mobile base | not applicable |
- Why recovery matters: In the report's example, a workflow cycle has about 79 steps. At 95% per-step success, all 79 succeed only 1.7% of the time. The report calls the per-step rate "illustrative, not measured," and says real numbers are worse because one failure tends to set up the next.
| Per-step success | Whole ~79-step cycle |
|---|---|
| 95% | 1.7% (from the report) |
| 99% | 45.2% |
| 99.9% | 92.4% |
| 99.99% | 99.2% |
(The last three rows are my arithmetic: the per-step rate to the 79th power.) - (Link: this is the same arithmetic behind Tedrake's Day 2 point that deployment needs 97–99% success.)
Scaling customer deployments (Dyna's deployment report)
- The requirement: each robot has to produce 1,500 table-ready napkins in an 18-hour shift.
- Dyna-1 at launch: about 35 napkins an hour, with only 75% clearing the quality bar.
- Dyna-2 (August 2026): 95 napkins an hour with 93% meeting the bar, for 1,590 table-ready napkins a day.
- Zero-shot acceptance: Dyna-2, pre-trained on human video alone, passed 87% of customer acceptance tests zero-shot, against 46% for Dyna-1. That was on sites with no data in training.
- Speed and scale: As little as three days from setup to the production ROI bar. Dyna expects "hundreds of robots by the first half of 2027."
- A field lesson: Missed grabs roughly doubled week over week. The root cause "turned out to be a worn gripper."
- (Link: compare Boston Dynamics' daily Atlas Check in Lutter's talk below. Hardware wear shows up as policy failures.)