Skip to content

Day 4: Thu, Nov 12, 2026

Workshop day.

Sessions attended

  • Workshop: Do Robotics Need World Models? Organizers and room not yet recorded. Talks photographed at 9:15–9:32 am and 10:03–10:06 am. Notes under Live notes.
  • Michael Lutter (Boston Dynamics): Sim2Real2Field at Boston Dynamics: The Unglamorous Part of Scaling Humanoids, in the Sim2Real2Field workshop, 3:41–3:59 pm from slide photos. Notes under Live notes.
  • Dyna Robotics: Ingredients of Scaling Foundation Models in Deployments. Speaker not yet recorded. About 2:45–3:05 pm. Notes under Live notes.
  • Workshop: Sim2Real2Field: A Model-Based Perspective. Organizers and room not yet recorded. The talk ran 2:17–2:29 pm. Notes under Live notes.

On the schedule, from Boston Dynamics' Day 2 slide (Day 2's Boston Dynamics notes):

  • 9:50 am, Everything Beneath the Policy workshop: Yeuhi Abe and Robin Deits (Boston Dynamics), Our Journey from MPC to RL.
  • 2:55 pm, Sim 2 Real 2 Field workshop: Michael Lutter (Boston Dynamics), Sim-2-Real RL for Manipulation. It was given at about 3:40 pm as Sim2Real2Field at Boston Dynamics: The Unglamorous Part of Scaling Humanoids. Notes under Live notes.

Papers covered

Paper Group Headline Status Links Weights
Dyna-2 Dyna Robotics World-action model pre-trained on 1M+ hours of human video only; zero-shot robot action MSE 0.180 to 0.117 from 1k to 1M hours abstract page ?
FARM UC Berkeley, Stanford Real-time object-level relational 3D memory answers relational language queries; +164% Recall@5 over prior methods (unverified) abstract arXiv · page · code ?
π0.7 Physical Intelligence Metadata turns low-quality data from harmful to helpful; zero-shot UR5e shirt folding at teleoperator level summarized arXiv · page —

Day 4 themes

Cross-day versions: world models, memory, deployment, simulation.

  1. Do robots need world models? The long-context talk said no: memory, subtasks, and prompting are enough. The remote talk framed VLAs and world action models as two ends of one design, and Dyna-2 scaled a world action model on human video.
  2. Memory in two tiers: dense video for the recent past, and compressed text (π0.6-MEM) or a 3D object graph (FARM) for everything else.
  3. The unglamorous parts of scaling: Dyna's recovery arithmetic and worn-gripper lesson, and Boston Dynamics' daily Atlas Check and fleet logging.
  4. Making RL work for real robots: demo resets and a frontier curriculum (Boston Dynamics), and model-based safety filters with residual RL (ResSafe).

Live notes

Workshop: Do Robotics Need World Models?

Organizers, room, and time not recorded. The title is as you typed it. Check whether the program says "Robotics" or "Robots."

Talk: Long-Context, Long-Horizon, Compositional Generalization without World Models

Speaker not recorded. Given in the Do Robotics Need World Models? workshop, with slides photographed 9:15–9:32 am. The memory slides cite Kim, Wang, Sridhar, Muppidi, Torne, Finn, Long-Context VLA: Rea[...] Robot Control, 2026. The middle of the title was hidden in every photo. With Chelsea Finn as last author and π0.6/π0.7 as the models, the speaker is probably from Physical Intelligence or Stanford, but that's unconfirmed. The title argues the "no" side of the workshop question.

  • The question: How can we apply robots to complex situations? The talk names three capabilities, each with an example from the slide:
  • Handling long context, such as tracking what happened.
  • Completing full long-horizon jobs, such as cleaning the kitchen.
  • Reasoning in new situations, such as applying a task to a different robot.

1. Handling long context

  • Many robot foundation models don't actually have memory or context. That's fine for repetitive, short motor skills.
  • Memory is critical for tracking.
  • Key challenges:
  • Spurious correlations between past observations and future actions. Sometimes adding memory makes the policy worse.
  • Substantial compute, for both training and inference. This gets harder as the context grows.
  • Why existing approaches fall short:
  • π0.7's context length is insufficient for these tasks. Its video history covers about 6 s: up to 6 frames per camera at a 1 s stride.
  • Keyframes don't provide enough information for tracking. ("Keyframes" is my reading. Your note said "Key.")
  • "Can we extend memory of VLAs to hundreds of frames?" (slide photo):
Problem Answer
Training signal over so many images Predict subtasks, and narrations of the memory.
Inference and training speed Cache and reuse past computation while streaming: the slide shows "cached history · KV + recurrent state reused." Decode the memory narration in parallel, in one forward pass. Group actions so fewer forward passes are needed.
  • The model, from the slide:
  • Backbone: a Qwen3.5-4B VLM with a 262K-token context.
  • It outputs three things:
    1. Reasoning: a narration of the memory, decoded in parallel in one forward pass. Example: "At 2 s, the ball is placed under the middle cup. At 5 s, the middle and right cups swap. At 9 s, …"
    2. Subtask: for example, "lift the left cup."
    3. Action tokens: grouped as OAT with group sizes 1, 1, 2, 4, 8, and 16, then decoded into the action chunk at:t+H.
  • Conclusion of part 1: "End-to-end training with subtask, narration prediction yields strong long-context reasoning."

Even longer memory: a VLA with multiscale memory

  • Two tiers of memory:
Medium-horizon memory Longer-horizon memory
Form Visual: dense and based on observations Text: sparse and semantic
Examples on the slide Espresso machine, crepes, window wiping Grocery aisle, kitchen stove
  • Multi-scale embodied memory: A VLM keeps a long-term text memory, such as "I wiped the counter with the sponge and dried it with the...", and updates it with a memory summary on each step. It also takes a short-term video memory of recent frames, and outputs a subtask. A VLA turns the subtask into robot actions.
  • Example prompt: "Clean up the kitchen. Wipe the countertop with the sponge and dry it with the paper towel and throw the towels in the trash under the ..." A VLA with multiscale memory: long-term text memory, prompt, and short-term video memory feed a VLM that writes a memory summary and a subtask for the VLA

  • Results: task progress (%), with values read by eye off the bar chart, not printed on the slide:

Task π0.6 (no memory) Video memory only Naive text + video Text memory only π0.6-MEM
Recipe set-up ~40 ~29 ~36 ~31 ~65
Clean kitchen ~23 ~38 ~46 ~33 ~93
Average ~42 ~38 ~45 ~38 ~78
  • "Text summaries offer strong improvements on 15+ min tasks over no text memory or naive text memory."
  • Compressed text memory expands the task horizon.
  • (The results slide compares π0.6 variants. Check whether the speaker said π0.6 or π0.7 when calling the context length insufficient.)

3. Compositional generalization to new, out-of-distribution situations

  • The tried-and-tested recipe for generalization:
  • Sufficiently large and diverse data.
  • Models with enough capacity.
  • "Let's use all the data we have!" Diverse robot demonstration data. (The rest of the list was cut off in your notes.)
  • Key idea: detailed, multimodal prompting. (Your note had "Prompt the model with..." cut off. This matches π0.7's metadata, subtask, and subgoal-image prompts.)
  • Out-of-distribution test: Can a bimanual UR5e fold clothes without any folding data on that robot? The slide showed an arm folding a blue shirt.
  • This is π0.7's zero-shot cross-embodiment result. The paper reports 85.6% task progress and 80% success, against 90.9% and 80.6% for expert teleoperators. Full numbers are in the π0.7 notes.
  • Takeaway: Strong generalization.

Overarching takeaways (slide photo, 9:31 am)

  • "With the right data, supervision, & context, models can reason through complex situations."
  • "No need for explicit mechanisms that explicitly build in a model of the world."
  • "Simple recipes often scale more readily!"

Remote talk: VLAs, world action models, and two data premises

Speaker not recorded. A remote talk over Zoom in the Do Robotics Need World Models? workshop. Slides photographed 10:03–10:06 am. Everything here is from slide photos.

  • What do robot foundation models have in common? A simplified comparison of a VLA and a world action model (WAM):
Backbone Passes to the action expert
VLA VLM Language in latent form: what to do
WAM Video model Video in latent form: how the world should change
  • The backbone side means "more semantics, more System 2," pretrained with world knowledge. The action-expert side means "more dynamics, more System 1," fine-tuned with embodied actions. VLA vs world action model: a VLM backbone passes language latents, a video backbone passes video latents, to an action expert

  • "Performance, generalization, scale: all still data problems." The question is which part of the model learns from which data, and how available that data is.

  • Two data premises:
  • Action labels are scarce, so use video data without action labels.
  • Successful demonstrations are scarce, so use counterfactual and failed rollouts.
  • Data premise 2 in detail: Train with local context only, meaning the current frame plus future video or actions:
  • Inverse dynamics (IDM): generate actions conditioned on video.
  • Forward dynamics (FDM): predict video conditioned on actions.
  • Finding: "less context, better dynamics." Given language and history, the model can lean on the task instead of learning how actions change the world.
  • (Link: compare the long-context talk just before this one, where more context helped. Here, less context helps the dynamics model.)

Workshop: Sim2Real2Field: A Model-Based Perspective

Organizers and room not recorded. The talk below ran 2:17–2:29 pm, from slide photos, and no title slide or speaker was photographed. "A model-based perspective" may be this talk's title rather than part of the workshop's name, so check the program. It isn't Boston Dynamics' Michael Lutter, who was scheduled here at 2:55 pm with Sim-2-Real RL for Manipulation. Lutter's talk has its own section below.

  1. Spatial memory matters.
  2. How the robot builds its memory, and how it queries that memory, both matter.
  3. Robots can benefit from relational spatial memory. (My reading: memory that stores how objects and places relate to each other, rather than raw frames.)
  4. Paper to keep in mind: FARM: Find Anything using Relational Spatial Memory.
    • FARM's memory, from the slides: Each object node stores attributes (3D Gaussians, features, views, captions, and more). Objects are joined by covisibility and adjacency edges only, with no higher-order relations. Details in the FARM notes.
    • Examples shown: a multi-floor house (~1,600 m²), "Find the tall lamp below the dartboard and to the left of the poster." A construction site (~15,000 m²), "Find the blue portable toilet with 'TOI TOI' branding closest to the white DBE-labeled delivery van." A 4,000 m² warehouse mapped live on a Spot at 10.6 Hz.
  5. Model-based safety filters can help. The filter checks each action the task controller proposes, and blocks or adjusts unsafe ones before they run.
  6. The architecture on the slide:

    Task controller ──▶ Safety filter ◀──▶ System

    • The task controller (the policy) only sends commands to the safety filter. It doesn't drive the system directly.
    • The safety filter and the system talk both ways. The filter sends safe commands to the system and gets the system's state back.
    • (My reading: the filter sits between the policy and the robot, so the policy can be swapped or retrained without touching the safety layer.)
    • The filter, formally (slide photo): The controller sends an unfiltered command uref(t). The filter returns the closest safe command, solving minu ‖u − uref‖ subject to u ∈ K(x). K(x) is the safety constraint, a set-valued map from the state x to the allowed controls. The system then follows ẋ = f(x, u).
    • The slide's question: "Can we directly design K(x)?"
    • Safety analysis, conditioned on the control input: The Hamilton-Jacobi-Isaacs PDE, min{∂V/∂t + H(z, ∇V), l(z) − V(z, t)} = 0 with H(z, ∇V) = minu ∇V·f(z, u), defines the safe set. "But we only need to guarantee that ∇V·f(z, u) ≤ 0." (My reading: any control that doesn't push the state out of the safe set will do, which is what makes K(x) a set and not a single action.)
    • Residual reinforcement learning: ResSafe (Gechen Qu et al., CoRL 2026).
    • Framework: A reference policy, a motion-tracking policy for a humanoid, proposes πref(ot), which may be unsafe. A learned residual nudges it across the implicit safety constraint: πsafe(ot) = πref(ot) + Δπθ(ot, πref(ot)). It runs in sim and on the real robot.
    • Training: offline, over a distribution that ranges from easy to difficult motions, from safe to unsafe reference policies, and across domain randomization of the environment.
    • Reward: survival time inside the safe set (reachability), plus a minimum-norm penalty on the residual (filtering), so it changes the reference only as much as needed.
    • (Related: Day 2's TAM is also a residual layer under any policy, but for torque adaptation rather than safety.) Safety filter between task controller and system, and ResSafe residual RL adding a learned correction to a reference policy
  7. Multi-agent interaction models are important. (These points come from your notes only. No photos were taken of this part.)

  8. Models can take closed-loop behavior into account: how each agent reacts to the others.
  9. That enables predictable, safe, high-confidence interaction between agents, even when the other agents' specification is unknown.
  10. Multi-agent RL with layered safety. (My reading: RL learns the interaction behavior, and model-based safety layers like the filter in point 2 keep it safe.)

Michael Lutter (Boston Dynamics): Sim2Real2Field at Boston Dynamics: The Unglamorous Part of Scaling Humanoids

Sim2Real2Field workshop. Slides photographed 3:41–3:59 pm, so it ran later than its scheduled 2:55 slot, where it was listed as Sim-2-Real RL for Manipulation. Your notes, checked against slide photos. Where a photo added detail, that's marked "slide."

Part 1. Real2Field: scaling from 4 to hundreds of humanoids

  • The goal: no "favorite robots." The slide's example of the problem: "I want to run my experiments on HD-003." Every robot should behave the same, so results don't depend on which unit you used.
  • All robots have to match the model.
  • Robot health has to be easy to measure.
  • "Is my robot working as expected?" The questions to answer for every robot:
  • Did any parts bend when it fell last night?
  • Does it have the latest operating system?
  • Are all motors working and calibrated?
  • Is the camera calibration correct?
  • Is the simulation model correct?
  • Run an "Atlas Check" on every robot every morning.
  • The robot performs a 5-minute routine that checks its calibration. The slide shows Atlas in front of a calibration board.
  • If it passes, it's considered healthy.
  • Double-check everything: Log, analyze, and visualize all the data you can record, including the firmware version and the policy ID.
  • Slide example: side-by-side videos of the same DeepMimic policy, one with a firmware bug and one with the firmware fixed.
  • "Is my new policy better than the previous one?" First: what does "better" even mean?
  • Fewer falls, and falls that cause less damage.
  • More comfortable for the robot pilots.
  • Smaller tracking error.
  • How much data do we need to collect to tell?
  • Lower failure rates require collecting more data. (My reading: when failures are rare, you need many more runs to measure a difference with confidence.)
  • There's more data from the old policy than from the new one.
  • Robot pilots might have adapted to the old policy. (My reading: both of these bias a head-to-head comparison toward the old policy.)
  • The slide shows a time-series chart next to a teleoperated Atlas video. The caption was partly hidden: "Teleoperated Jen[...] Speed."

Real2Field fleet loop: a daily Atlas Check on every robot, healthy robots run experiments, and everything is logged to compare policies

Part 2. Sim2Real: reinforcement learning for manipulation

  • Teleop for multi-fingered hands is hard. The slide shows a teleoperated hinge insertion on Atlas.
  • Use demos to narrow the search space.
  • Exploration is hard: Policies often don't learn with sparse rewards. Tuning dense rewards is laborious and tedious. Policies learn to exploit the reward.
  • Use demo resets to guide the exploration: Collect human or agentic demos for the task, then start exploration from states along those demos. Demos are just a hint to the policy about where to search, and policies can diverge from the demo distribution.
  • Teacher training with reset-state initialization: human demos → demo states → reset the simulator to a demo state → teacher RL. When an episode ends, a new start state is sampled. Starting always from s₀, exploration stays near s₀. Starting from demo states, exploration covers the whole task.
  • Demo resets are more common in the literature. Papers named on the slide:
    • 2018: DeepMimic (Peng et al.), Overcoming Exploration (Nair et al.)
    • 2024: DemoStart (Bauza et al.), RFCL (Tao et al.)
    • 2026: OmniReset (Yin et al.), SGS (Zhang et al.)

Demo resets: teacher RL starts from sampled demo states, so exploration covers the whole task instead of staying near s0

  • Frontier reset curriculum:
  • Frontier states are states from which the policy can complete the task.
  • Sample reset states near the frontier, where the policy isn't yet successful.
  • Advance the frontier when the policy completes the task from there.
  • Augment frontier samples with uniform resets to increase diversity (slide; your note was cut off here).
  • Hyperparameters: success-rate threshold, window size, and the frontier-to-uniform ratio.
  • How the frontier moves: The slide's grids of demo time show resets starting late in the demo and moving toward the beginning as training goes on. This confirms my earlier reading.

Frontier reset curriculum: resets start near the end of the demo and the frontier moves back toward the start as training succeeds

  • Results: Atlas doing cup stacking, block stacking, and plug-in-socket tasks.
  • Preliminary results for frontier sampling (one task):
  • Frontier sampling reaches 96.5% mean evaluation success by about 600M environment steps. It passes the 90% target at about 300M steps.
  • The baselines all stay below about 60% over the same steps: SGS as in the paper, SGS matched, SGS with κ = 10, SGS Pedro, and uniform resets. (Read by eye off the chart.)
  • Heatmaps of start position within the demo, over training, show the frontier covering the whole demo, while SGS and uniform resets cover less of it.
  • The slide's own caveat: "Comparison only includes one task and is missing baselines!"

Summary (slide)

  • Real2Field, scaling from 4 to hundreds of humanoids:
  • It takes a village to make sure all robots work as expected at all times.
  • Log, analyze, and visualize all the data you can record, to make sure each robot is working as expected.
  • The unglamorous work of calibration, testing, and root-causing problems becomes super important.
  • Comparing two policies becomes hard, because there are so many confounding factors and data is scarce.
  • Sim2Real, reinforcement learning for manipulation:
  • Teleoperating multi-fingered hands is hard, and usually gives suboptimal demonstrations.
  • RL can learn from suboptimal demonstrations and discover better behaviours.
  • Guide exploration with demonstrations, to minimize reward tuning and reward hacking.
  • Simulation tools and vision encoders have improved so much that RGB policies are feasible.
  • (Link: Part 1 is the fleet-operations side of sim-to-real. A policy trained against one simulation model only transfers if every real robot still matches that model. It also fits Boston Dynamics' Day 2 theme, "data quality before scale," and its slide "for very dexterous manipulation, RL is hard to beat.")

Dyna Robotics: Ingredients of Scaling Foundation Models in Deployments

Speaker not recorded. Given at about 2:45–3:05 pm. This section uses only Dyna's own public reports, which the talk pointed to: - Dyna-2, August 2026 - Dyna-2.1, September 29, 2026 - Scaling customer deployments, August 2026

  • The talk's themes: learning from human data, and a flywheel between research and deployment.
  • Firsts Dyna claims publicly (each hedged "to the best of our knowledge"):
  • Aug 2026: a 1-million-hour world-action model and scaling law (Dyna-2).
  • Aug 2026: the first commercial manipulation pilot taken all the way to scaled deployment.
  • Sep 2026: "the first physical agent to perform hour-long, non-linear loco-dexterous manipulation workflows" (Dyna-2.1).

Dyna-2: a 1-million-hour scaling law for world-action models

  • A world-action model pre-trained on more than one million hours of egocentric human video. Some clips have 3D hand-pose tracks, and the rest have no action labels.
  • It shows a scaling law on held-out human data and, Dyna says, the first human-to-robot transfer scaling law: zero-shot on 39 robot tasks never seen in pre-training.
  • Human-to-robot transfer (scaling-law models pre-trained on human data only):
Pre-training human hours 1k 10k 100k 1M Fit
Zero-shot action MSE ↓ 0.180 0.174 0.124 0.117 MSE = 0.306 · D−0.0713, R² = 0.884
Zero-shot accuracy@0.5 ↑ 0.067 0.074 0.136 0.159 acc = 0.0241 · D0.139, R² = 0.918
  • The report notes that its production Dyna-2 models use different training recipes from these scaling-law variants. Dyna-2.1's policy mixes the human video with robot data from Dyna's fleet.
  • Post-training: "With just a few hours of robot data alone in post-training." One example: "a mere 10 minutes of teleoperation data was enough to fine-tune Dyna-2" to open bottle caps.
  • Details are in the Dyna-2 notes.

Dyna-2.1: a physical agent for end-to-end workflows

  • Taku is a semi-humanoid robot: a 21-DoF upper body with two 7-DoF arms, on a mobile base with four steerable wheels.
  • The stack, from top to bottom:
Layer What it is Rate
Workflow orchestrator vision-language model; tracks the workflow and decides the next step not given
Dyna-2 policy world-action model ~5 Hz
Whole-body controller RL, trained in simulation 100 Hz
Taku 21-DoF upper body plus mobile base not applicable

Dyna full stack: VLM workflow orchestrator, Dyna-2 world-action policy at about 5 Hz, RL whole-body controller at 100 Hz, and the Taku body

  • Why recovery matters: In the report's example, a workflow cycle has about 79 steps. At 95% per-step success, all 79 succeed only 1.7% of the time. The report calls the per-step rate "illustrative, not measured," and says real numbers are worse because one failure tends to set up the next.
Per-step success Whole ~79-step cycle
95% 1.7% (from the report)
99% 45.2%
99.9% 92.4%
99.99% 99.2%

(The last three rows are my arithmetic: the per-step rate to the 79th power.) - (Link: this is the same arithmetic behind Tedrake's Day 2 point that deployment needs 97–99% success.)

Scaling customer deployments (Dyna's deployment report)

  • The requirement: each robot has to produce 1,500 table-ready napkins in an 18-hour shift.
  • Dyna-1 at launch: about 35 napkins an hour, with only 75% clearing the quality bar.
  • Dyna-2 (August 2026): 95 napkins an hour with 93% meeting the bar, for 1,590 table-ready napkins a day.
  • Zero-shot acceptance: Dyna-2, pre-trained on human video alone, passed 87% of customer acceptance tests zero-shot, against 46% for Dyna-1. That was on sites with no data in training.
  • Speed and scale: As little as three days from setup to the production ROI bar. Dyna expects "hundreds of robots by the first half of 2027."
  • A field lesson: Missed grabs roughly doubled week over week. The root cause "turned out to be a worn gripper."
  • (Link: compare Boston Dynamics' daily Atlas Check in Lutter's talk below. Hardware wear shows up as policy failures.)