Skip to content

TEMPO: Learning Temporal Context for Dynamic Robot Manipulation

Session: Oral (top 1%), Mon Nov 9, 11:00 am, Hilton Austin. Poster: Mon 5:00 pm, Lone Star, Booth 48. Speaker(s): Paper: arXiv 2609.16864 (HTML) by Zhenyang Feng, Jimin Heo, Erik B. Sudderth, and Unnat Jain (UC Irvine), September 2026 Project page: tempo-robot.github.io · Code: tempo-robot/TEMPO (Apache-2.0). The README calls the paper "Closing the Representational Gap for VLAs in Dynamic Settings."

Paper summary (from the arXiv HTML, project page, and README; the limitations section was not reached)

  • Problem: Single-frame VLAs fail in dynamic scenes for two reasons:
  • Motion ambiguity: One frame cannot show where moving objects are going.
  • State aliasing: Frames that look the same can need different actions at different points in the task, for example a hand approaching versus pulling away.
  • The authors argue the bottleneck is missing temporal context, not model scale or latency.

Method (adds about 2M parameters, 0.08%, to π0.5 running on VLASH's asynchronous pipeline)

  • TEMPO-MOT (motion):
  • A frozen video foundation model, SAM 2.1-Tiny by default, used without any fine-tuning.
  • The current frame's patch features cross-attend to a streaming cache of the previous frames' features. Each frame is encoded once, and the encoder runs in a background thread at sensor rate (about 11 ms per frame).
  • TEMPO-ACT (action history):
  • The last 300 proprioceptive commands, averaged into 10 buckets of 30. That is about 10 s at 30 Hz (my arithmetic).
  • This part has no parameters.
  • Injection:
  • Both signals are projected and appended to the VLM prefix, next to the images and the prompt.
  • The action history also feeds an AdaRMS conditioning MLP in the action expert.
  • Zero-initialization keeps the pretrained behavior at the start of fine-tuning. The flow-matching loss is unchanged.
  • Training: Fine-tuned from π0.5-base on 8 GPUs at batch size 128. A secondary summary says about 2 h per policy on RTX PRO 6000 GPUs (unverified).

Results (bimanual YAM arms, 50 real rollouts per entry)

Method Bottle Handover Drop Catch Flick Catch Wine Pour* Avg
RTC 44 80 48 80.8 63.2
VLASH 38 38 20 94.0 47.5
TEMPO 74 66 66 98.1 76.0
TEMPO + UVT 88 88 76 96.7 87.2

*Wine Pour reports the percentage of wine kept in the pot, not success.

  • What "44% → 74%" means: The abstract's Bottle Handover number compares against RTC. The "+36%" in the text compares against VLASH. On Drop Catch, RTC still beats TEMPO.
  • State aliasing: On the full, untrimmed tasks, RTC and VLASH score 0/50 on Flick Catch and on Wine Pour, while TEMPO scores 68% and 97.6%.
  • Works with other backbones (Bottle Handover):
  • π0.5: 26% → 56%
  • RTC: 44% → 80%
  • VLASH: 38% → 74%
  • Latency (median): TEMPO takes 35.3 ms, against 33.8 ms for VLASH and 127.7 ms for RTC. The cost per frame does not grow with history length.
  • Ablations (Bottle Handover):
  • Action history alone: 52%. Motion alone: 56%. Both: 74%.
  • Encoder choice: SAM 2.1-Tiny 74%, Reducio 76%, VidTwin 60%, VideoLaVIT 52%. SAM was chosen for streaming speed.
  • Motion probe: A probe that predicts object velocity from the policy's features scores R² = 0.47 on average for TEMPO, against about 0.2 for the baselines.
  • TEMPO-Bench: More than 50K frames of human–robot dynamic manipulation annotated with object velocity. It comes in a regression format and a multiple-choice format in the MVBench style. Data: tempo-robot/TempoBench.