Systems 0 / 1 / 2¶
Talk: Organizing Physical AI Across Timescales (MARS Lab, Day 1)¶
Speaker not recorded. Slide photos taken Mon Nov 9, 9:58–10:05 am. Everything here is from the slides.
What goes down, and what comes back up
| Layer | Clock | Intent passed down | Feedback passed up |
|---|---|---|---|
| System 2 | seconds | Language, goals, programs, action macros | Progress, confidence, "I'm not sure" |
| System 1 | tens of Hz | Mostly motion, today | Contact, force, tracking state |
| System 0 | hundreds of Hz | n/a | n/a |
- "Most work today is on the way down. The way up is mostly missing."
- Example: OMG, one motion generator with many inputs (Siqiao Huang). OMG: Omni-Modal Motion Generation for Generalist Humanoid Control, arXiv 2606.10340. It takes text, a person's motion, music, or text plus music. The demo was one take, played at 2×, with the text "Wave hands while dancing" and the audio "Paper Rings."
- The slide's point: the input is "rich from above," but only motion goes down.
The speaker's view: the future goes two ways
- Layers close their own loops. Each pair of layers passes down intent and "what matters," and passes up "what went wrong."
- Today: Mostly motion goes down, and almost nothing comes back up.
- The hard part: The lower layer must sense, correct, and send feedback on its own.
- One model, several clocks inside. Systems 2, 1, and 0 all live in one model, with System 0 drawn dashed.
- The good part: A unified architecture and shared context, with inference at different clocks.
- The hard part: Training Systems 0, 1, and 2 together, across BC and RL.
"Either way, the clock has to be part of the design."
What to take home
- Clock: What a system must react to sets its clock.
- Interface: Intent goes down and feedback comes up. Each system closes its own loop.
- Design a multi-clock system, and work on what passes between the clocks.
One request: In simulation, also report the success rate before a deadline, with inference time counted.
Across the week: the stacks people actually built¶
The MARS Lab talk gave the framing. Other talks across the week presented concrete stacks. This is how they line up:
| Talk | Day | System 2 | Interface down | System 1 | System 0 |
|---|---|---|---|---|---|
| Meta Robotics Studio | 2 | Muse plans the task | a plan | Muse-based policy | not described |
| Hydra (NVIDIA) | 3 | agent | object flow | policy | not described |
| Dyna | 4 | VLM workflow orchestrator | "the next step" | Dyna-2 world-action model, ~5 Hz | RL whole-body controller, 100 Hz |
| Long-context VLA | 4 | VLM with long-term text memory | a subtask | VLA | not described |
| Remote talk, VLA vs WAM | 4 | backbone: "more semantics, more System 2" | a latent | action expert: "more dynamics, more System 1" | none |
Takeaways
- The System 2 → 1 interface is where designs differ most: language or subtasks, object flow, plans. MARS Lab's point that "the way up is mostly missing" still holds. No stack this week described rich feedback from System 1 back to System 2.
- System 0 is usually left implicit. Dyna is the exception, with a 100 Hz RL whole-body controller. TAM and ResSafe show learned System 0 layers that sit under any policy.
- My reading: The rates line up with MARS Lab's framing: seconds for System 2, about 5 Hz for a world-action policy, and 100 Hz and up for control.
System 0¶
Fast reflexive or low-level control layer.
Talks / papers¶
- SuperMimic: One learned world model + analytic gradients; beats SONIC's return in 3 h on 1 GPU vs 7 days on 128 GPUs (>1000× fewer GPU-hours)
- TAM: Sim-only residual torque layer below any policy; real Panda box pushing 47.6% to 76.2%, BC flip 50% to 72%
System 1¶
Fast, intuitive policy layer, for example VLA action experts.
Talks / papers¶
- DreamZero: 14B video model as policy; over 2× generalization vs VLAs at 7 Hz
System 2¶
Slow, deliberative reasoning and planning layer, for example VLM planners and world models.
Talks / papers¶
-