Skip to content

Systems 0 / 1 / 2

Talk: Organizing Physical AI Across Timescales (MARS Lab, Day 1)

Speaker not recorded. Slide photos taken Mon Nov 9, 9:58–10:05 am. Everything here is from the slides.

What goes down, and what comes back up

Layer Clock Intent passed down Feedback passed up
System 2 seconds Language, goals, programs, action macros Progress, confidence, "I'm not sure"
System 1 tens of Hz Mostly motion, today Contact, force, tracking state
System 0 hundreds of Hz n/a n/a
  • "Most work today is on the way down. The way up is mostly missing."
  • Example: OMG, one motion generator with many inputs (Siqiao Huang). OMG: Omni-Modal Motion Generation for Generalist Humanoid Control, arXiv 2606.10340. It takes text, a person's motion, music, or text plus music. The demo was one take, played at 2×, with the text "Wave hands while dancing" and the audio "Paper Rings."
  • The slide's point: the input is "rich from above," but only motion goes down.

MARS Lab: intent down, feedback up, and two ways forward

The speaker's view: the future goes two ways

  1. Layers close their own loops. Each pair of layers passes down intent and "what matters," and passes up "what went wrong."
  2. Today: Mostly motion goes down, and almost nothing comes back up.
  3. The hard part: The lower layer must sense, correct, and send feedback on its own.
  4. One model, several clocks inside. Systems 2, 1, and 0 all live in one model, with System 0 drawn dashed.
  5. The good part: A unified architecture and shared context, with inference at different clocks.
  6. The hard part: Training Systems 0, 1, and 2 together, across BC and RL.

"Either way, the clock has to be part of the design."

What to take home

  1. Clock: What a system must react to sets its clock.
  2. Interface: Intent goes down and feedback comes up. Each system closes its own loop.
  3. Design a multi-clock system, and work on what passes between the clocks.

One request: In simulation, also report the success rate before a deadline, with inference time counted.

Across the week: the stacks people actually built

The MARS Lab talk gave the framing. Other talks across the week presented concrete stacks. This is how they line up:

Systems 2, 1, and 0 as presented by MARS Lab, Meta, Hydra, Dyna, and the long-context VLA

Talk Day System 2 Interface down System 1 System 0
Meta Robotics Studio 2 Muse plans the task a plan Muse-based policy not described
Hydra (NVIDIA) 3 agent object flow policy not described
Dyna 4 VLM workflow orchestrator "the next step" Dyna-2 world-action model, ~5 Hz RL whole-body controller, 100 Hz
Long-context VLA 4 VLM with long-term text memory a subtask VLA not described
Remote talk, VLA vs WAM 4 backbone: "more semantics, more System 2" a latent action expert: "more dynamics, more System 1" none

Takeaways

  • The System 2 → 1 interface is where designs differ most: language or subtasks, object flow, plans. MARS Lab's point that "the way up is mostly missing" still holds. No stack this week described rich feedback from System 1 back to System 2.
  • System 0 is usually left implicit. Dyna is the exception, with a 100 Hz RL whole-body controller. TAM and ResSafe show learned System 0 layers that sit under any policy.
  • My reading: The rates line up with MARS Lab's framing: seconds for System 2, about 5 Hz for a world-action policy, and 100 Hz and up for control.

System 0

System 0 / 1 / 2 stack

Fast reflexive or low-level control layer.

Talks / papers

  • SuperMimic: One learned world model + analytic gradients; beats SONIC's return in 3 h on 1 GPU vs 7 days on 128 GPUs (>1000× fewer GPU-hours)
  • TAM: Sim-only residual torque layer below any policy; real Panda box pushing 47.6% to 76.2%, BC flip 50% to 72%

System 1

Fast, intuitive policy layer, for example VLA action experts.

Talks / papers

  • DreamZero: 14B video model as policy; over 2× generalization vs VLAs at 7 Hz

System 2

Slow, deliberative reasoning and planning layer, for example VLM planners and world models.

Talks / papers

-