TEMPO: Learning Temporal Context for Dynamic Robot Manipulation¶
Session: Oral (top 1%), Mon Nov 9, 11:00 am, Hilton Austin. Poster: Mon 5:00 pm, Lone Star, Booth 48. Speaker(s): Paper: arXiv 2609.16864 (HTML) by Zhenyang Feng, Jimin Heo, Erik B. Sudderth, and Unnat Jain (UC Irvine), September 2026 Project page: tempo-robot.github.io · Code: tempo-robot/TEMPO (Apache-2.0). The README calls the paper "Closing the Representational Gap for VLAs in Dynamic Settings."
Paper summary (from the arXiv HTML, project page, and README; the limitations section was not reached)¶
- Problem: Single-frame VLAs fail in dynamic scenes for two reasons:
- Motion ambiguity: One frame cannot show where moving objects are going.
- State aliasing: Frames that look the same can need different actions at different points in the task, for example a hand approaching versus pulling away.
- The authors argue the bottleneck is missing temporal context, not model scale or latency.
Method (adds about 2M parameters, 0.08%, to π0.5 running on VLASH's asynchronous pipeline)
- TEMPO-MOT (motion):
- A frozen video foundation model, SAM 2.1-Tiny by default, used without any fine-tuning.
- The current frame's patch features cross-attend to a streaming cache of the previous frames' features. Each frame is encoded once, and the encoder runs in a background thread at sensor rate (about 11 ms per frame).
- TEMPO-ACT (action history):
- The last 300 proprioceptive commands, averaged into 10 buckets of 30. That is about 10 s at 30 Hz (my arithmetic).
- This part has no parameters.
- Injection:
- Both signals are projected and appended to the VLM prefix, next to the images and the prompt.
- The action history also feeds an AdaRMS conditioning MLP in the action expert.
- Zero-initialization keeps the pretrained behavior at the start of fine-tuning. The flow-matching loss is unchanged.
- Training: Fine-tuned from π0.5-base on 8 GPUs at batch size 128. A secondary summary says about 2 h per policy on RTX PRO 6000 GPUs (unverified).
Results (bimanual YAM arms, 50 real rollouts per entry)
| Method | Bottle Handover | Drop Catch | Flick Catch | Wine Pour* | Avg |
|---|---|---|---|---|---|
| RTC | 44 | 80 | 48 | 80.8 | 63.2 |
| VLASH | 38 | 38 | 20 | 94.0 | 47.5 |
| TEMPO | 74 | 66 | 66 | 98.1 | 76.0 |
| TEMPO + UVT | 88 | 88 | 76 | 96.7 | 87.2 |
*Wine Pour reports the percentage of wine kept in the pot, not success.
- What "44% → 74%" means: The abstract's Bottle Handover number compares against RTC. The "+36%" in the text compares against VLASH. On Drop Catch, RTC still beats TEMPO.
- State aliasing: On the full, untrimmed tasks, RTC and VLASH score 0/50 on Flick Catch and on Wine Pour, while TEMPO scores 68% and 97.6%.
- Works with other backbones (Bottle Handover):
- π0.5: 26% → 56%
- RTC: 44% → 80%
- VLASH: 38% → 74%
- Latency (median): TEMPO takes 35.3 ms, against 33.8 ms for VLASH and 127.7 ms for RTC. The cost per frame does not grow with history length.
- Ablations (Bottle Handover):
- Action history alone: 52%. Motion alone: 56%. Both: 74%.
- Encoder choice: SAM 2.1-Tiny 74%, Reducio 76%, VidTwin 60%, VideoLaVIT 52%. SAM was chosen for streaming speed.
- Motion probe: A probe that predicts object velocity from the policy's features scores R² = 0.47 on average for TEMPO, against about 0.2 for the baselines.
- TEMPO-Bench: More than 50K frames of human–robot dynamic manipulation annotated with object velocity. It comes in a regression format and a multiple-choice format in the MVBench style. Data: tempo-robot/TempoBench.