PAPER DEEP DIVE
TemporalFlow-VLA: Learning Physically Grounded Execution History for Long-Horizon Robot Manipulation
Vision-language-action (VLA) models leverage pretrained vision-language representations for robot control, yet simply adding historical frames does not reliably capture recent physical change. This is especially problematic in multi-stage manipulation, where visually similar states may require different actions depending on prior execution. To address this challenge, we present TemporalFlow-VLA, which learns compact execution history through physically grounded temporal supervision. Using recorded robot states, robot geometry, and calibrated cameras, we construct robot-surface temporal flow as a training-only target and supervise two execution-aligned temporal queries that provide structured history to the action expert. The geometric supervision path is not evaluated at deployment. TemporalFlow-VLA achieves 97.63 +/- 0.26% average success on LIBERO, including 96.60 +/- 0.87% on LIBERO Long, and 85.5%/84.2% Clean/Randomized success across 12 RoboTwin tasks. It shows its clearest advantage over prior methods on longer-horizon, multi-stage manipulation. Controlled history interventions show that action prediction depends on both historical content and temporal order. With asynchronous feature caching, temporal conditioning maintains single-frame-level server-side sampling latency without additional historical-encoding overhead. Overall, TemporalFlow-VLA provides a compact, physically grounded interface for exploiting ordered execution history without explicit motion estimation or geometric processing at deployment.
Paper: TemporalFlow-VLA: Learning Physically Grounded Execution History for Long-Horizon Robot Manipulation
Authors: Jiarui Yang, Yehao Lu, Yuning Su, Yu Zhong, Yufeng Xie, Yazhou Zhang, Haiyu Lan, Kaixiang Lu, Peiwen Lin, Chuang Wang, Junwei Liang, Enyu Li
Affiliations: HKUST (Guangzhou) · Zhejiang University · Simon Fraser University · AgiBot
Link: arXiv:2608.26821
One-Sentence Summary
TemporalFlow-VLA supervises two compact temporal queries with kinematics-derived robot-surface flow — a training-only signal constructed from recorded joint states, URDF geometry, and calibrated cameras — so the policy learns what physically changed between frames. On RoboTwin 2.0 it reaches 85.5%/84.2% Clean/Randomized success across 12 tasks (beating the strongest baseline by 8.0 points under randomization), and on LIBERO averages 97.63±0.26% with 96.60±0.87% on LIBERO Long; on an AgiBot A3 real robot, three-stage tasks improve by up to +20.0 points, while asynchronous feature caching actually makes temporal conditioning 7.8% faster at inference.
1. Background and Motivation
Most representative VLA policies generate each action chunk almost entirely from the current RGB frame, the language instruction, and robot state. This becomes ambiguous whenever visually similar observations arise from different execution histories: a failed grasp looks like a pre-grasp state; the same end-effector pose can mean approaching, carrying, or recovering. Without recent temporal evolution, a policy can misidentify the local task phase, overlook what the previous chunk accomplished, or repeat already-executed behavior — errors that compound across replanning steps in multi-stage manipulation.
The obvious fix — feeding historical frames into the backbone — turns out to be unreliable. The authors' diagnostic (Fig. 2) is striking: on a baseline trained with chronological history, shuffling the three historical observations while fixing the current frame leaves the offline action flow-matching loss nearly unchanged, whereas removing history entirely raises it by about 4.6%. History helps, but the unconstrained baseline is largely insensitive to the correct temporal order. External audits agree: ViSTR-Bench reports persistent VLM difficulty with motion perception and physical dynamics from continuous cues, and a mechanistic audit of frozen multi-frame VLAs found history-unique information weak, mainly influencing actions when the current observation is unreliable.
TraceVLA provides the counterpoint: explicitly exposing tracked-point trajectories as visual prompts beats a six-frame history baseline substantially. The lesson the authors draw: the challenge is not providing history but representing recent physical change in an action-usable, order-sensitive form. TemporalFlow-VLA's answer is to supervise temporal representations with geometry — projecting recorded robot motion into the RGB plane as an explicit training target.
2. Method Overview: A Parallel Temporal Pathway
The system augments a pretrained π0.5-style VLA with a parallel temporal pathway while keeping the original action-generation path untouched. The base policy predicts an action chunk from the current RGB observation, language, robot state, and diffusion timestep. The temporal pathway additionally receives head-camera observations at t−15, t−8, and t. For a 16-step action chunk, t−15 and t−8 roughly correspond to the beginning and midpoint of the previous chunk — one query covers the full preceding chunk, the other its more recent half.
Two supervised temporal queries, Q8 and Q15, are the interface: Q8 represents recent motion from t−8 to t; Q15 captures the full t−15 to t evolution and can build on Q8's short-range summary. Action tokens receive history only through these two queries — they cannot directly attend to historical image patches. During training, robot kinematics provides flow supervision for the queries; at deployment the kinematic renderer is absent and the flow-reconstruction heads are never evaluated.
3. Kinematics-Grounded Robot-Surface Temporal Flow
The supervision target is constructed offline from recorded joint configurations, the robot URDF, and hand-eye calibration — no manual flow annotation. For each interval ρ ∈ {8, 15} with source time s = t − ρ, let q_τ be the joint configuration, T_B^l(q_τ) the link-to-base forward kinematics, and T_{C←B} the calibrated base-to-camera transform. A robot-only renderer assigns each visible robot pixel p its 3D base-frame surface point X_s^B(p) and owning link l(p). The point is pulled back to its link frame, transported with the target-time kinematics, and re-projected:
$$\bar{\mathbf{x}}^{l(p)}(p) = [\mathbf{T}_B^{l(p)}(\mathbf{q}_s)]^{-1}\bar{\mathbf{X}}_s^B(p) \tag{1}$$
$$\bar{\mathbf{X}}_t^B(p) = \mathbf{T}_B^{l(p)}(\mathbf{q}_t)\,\bar{\mathbf{x}}^{l(p)}(p) \tag{2}$$
$$\tilde{\mathbf{u}}_t(p) = \Pi_K(\mathbf{T}_{C\leftarrow B}\bar{\mathbf{X}}_t^B(p)) \tag{3}$$
A nearest-pixel lookup in the target-time robot render keeps a correspondence only if the projection is in bounds, belongs to the same link, and has a 3D residual ≤ 5 mm. Displacements are normalized by the 224 px policy image plane:
$$\mathbf{f}_\rho(p) = \frac{\mathbf{u}_t(p) - \mathbf{u}_s(p)}{224} \tag{4}$$
Valid robot flows are pooled by masked mean over non-overlapping 14×14 regions into a 16×16×2 target; a per-patch coverage ratio c_ρ(k) = |V_k|/|S_k| gates the loss with mask M_ρ(k) = 1[c_ρ(k) ≥ 0.1] — coverage gates, it does not weight. Crucially, the robot-surface restriction applies only to the auxiliary target: the temporal pathway still consumes full RGB, so object and scene changes remain available to the action objective.
4. Hierarchical Temporal Queries and Joint Attention
Learnable tokens q8 and q15 are appended to the standard prefix (current image tokens V0, language L, robot state s); learned frame-identity embeddings mark the historical patches V15 and V8. Cross-attention over each interval ρ produces hidden states H_ρ, decoded by interval-specific FiLM-modulated pointwise MLPs into flow predictions on the 16×16 grid:
$$\hat{\mathbf{F}}_\rho = \mathbf{W}_\rho^o\,\mathrm{GELU}(\mathbf{W}_\rho^h\mathbf{H}_\rho + \mathbf{b}_\rho^h) + \mathbf{b}_\rho^o, \quad \rho \in \{8, 15\} \tag{5}$$
The two decoders share no parameters and use no convolution or upsampling. The masked Huber loss is evaluated only where M_ρ = 1, and the temporal objective averages the two scales:
$$\mathcal{L}_{\mathrm{flow}}^\rho = \frac{\sum_k \mathbf{M}_\rho(k)\,\ell_{\mathrm{Huber}}(\hat{\mathbf{F}}_\rho(k), \mathbf{F}_\rho(k))}{\sum_k \mathbf{M}_\rho(k) + \epsilon}, \qquad \mathcal{L}_{\mathrm{temp}} = \tfrac{1}{2}(\mathcal{L}_{\mathrm{flow}}^8 + \mathcal{L}_{\mathrm{flow}}^{15}) \tag{6}$$
Temporal information reaches the action expert through the model's existing joint masked self-attention. Prefix P^l and action suffix A^l form queries, keys, values that are concatenated and attended with a structured mask B(M):
$$\mathbf{O}^l = \mathrm{Softmax}\left(\frac{\mathbf{Q}^l(\mathbf{K}^l)^\top}{\sqrt{d}} + \mathbf{B}(\mathbf{M})\right)\mathbf{V}^l \tag{7}$$
For action tokens the mask preserves the original VLA context and exposes Q8 and Q15, while blocking direct access to V8 and V15: historical visual information must be compressed through the flow-supervised queries before it can influence actions. No separate cross-attention module or history-specific gate is needed, and the action expert's AdaRMS modulation is preserved. The final objective keeps flow supervision on throughout training with λ_temp = 1.0:
$$\mathcal{L} = \mathcal{L}_{\mathrm{action}} + \lambda_{\mathrm{temp}}\,\mathcal{L}_{\mathrm{temp}} \tag{8}$$
flowchart LR
subgraph TRAIN["Training only"]
JS["Recorded joint states q_t"] --> FK["Forward kinematics
T_B^l(q)"]
URDF["URDF geometry + calibration"] --> REND["Robot-only renderer
surface points + link IDs"]
FK --> REND
REND --> FLOW["Robot-surface flow labels
F_8, F_15 (16×16×2)"]
FLOW --> SUP["Supervise Q8/Q15
masked Huber loss"]
end
subgraph DEPLOY["Train + Deploy"]
H["Head-camera frames
t−15, t−8, t"] --> VENC["Visual encoder"]
VENC --> V15["V15, V8 patches
+ frame identity"]
V15 --> XATT["Interval cross-attention"]
XATT --> Q["Q8: recent motion
Q15: full chunk evolution"]
Q --> MASK["Joint masked attention
action tokens see Q8/Q15
never V8/V15"]
CUR["Current frame + language + state"] --> MASK
MASK --> ACT["Action chunk (16 steps)"]
end
SUP -.-> Q
5. Asynchronous Historical-Feature Caching
Synchronously encoding t−15, t−8, and t at every replan would add two visual forward passes to the critical path. Instead, incoming head-camera frames go into a timestamped ring buffer while the robot executes the current chunk; a background process encodes them: V_τ = VisualEncoder(I_τ), cached as C[τ]. Because chunk length and the Q8/Q15 offsets are fixed, both historical encodings complete before the next replan, which retrieves them from cache:
$$\mathbf{P}_{\mathrm{temp}} = [\mathbf{V}_{15}^{\mathrm{cache}};\ \mathbf{V}_8^{\mathrm{cache}};\ \mathbf{V}_0^{\mathrm{current}}] \tag{9}$$
The latency accounting is simple: the naive path costs T_naive = 3T_vision + T_joint + T_action, while caching brings it to T_cache ≈ T_vision + T_joint + T_action — single-frame-level sampling latency. At episode start, unavailable history is filled with the earliest observation and marked invalid until both required frames exist.
6. Experiments
Setup. All policies train on 8×H100, per-GPU batch size 32. LIBERO: four suites jointly, 30k steps, mean±SD over seeds 0/2/5, 500 rollouts per suite and seed. RoboTwin 2.0: 12 tasks jointly for 60k steps from 50 clean + 500 randomized demos per task, 100 rollouts per task per condition, grouped by execution horizon H∈{1,2,3} following LingBot-VA. Temporal-flow labels are precomputed offline; the temporal module adds ~20% wall-clock training time. Real robot: AgiBot A3, two three-stage tasks, 280 demos each, 3 rounds × 15 trials per method.
RoboTwin 2.0 (Table I). TemporalFlow-VLA reaches 85.5% Clean / 84.2% Randomized overall — the randomized average beats the best published baseline by 8.0 points. The advantage scales with horizon: at H=2 the method scores 80.8% (+5.5 over runner-up), at H=3 it scores 87.5% (+14.5), while the H=1 average is not the best — consistent with the claim that explicit history matters most when success depends on preserving progress across stages. Standout per-task gaps include Stack Blocks Three (97/95 vs π0.5's 72/52) and Handover Mic (100/99 vs X-VLA's 0/0).
| Horizon | Ours | π0 | π0.5 | X-VLA† | GigaWorld |
|---|---|---|---|---|---|
| H=1 | 84.0 / 84.3 | 61.5 / 56.3 | 81.3 / 75.3 | 90.3 / 88.8 | 92.5 / 87.0 |
| H=2 | 83.5 / 80.8 | 71.3 / 65.8 | 76.8 / 75.3 | 52.3 / 52.0 | 71.5 / 68.5 |
| H=3 | 89.0 / 87.5 | 57.8 / 44.0 | 79.0 / 66.5 | 57.5 / 61.0 | 69.5 / 73.0 |
| Overall | 85.5 / 84.2 | 63.5 / 55.3 | 79.0 / 72.3 | 66.7 / 67.3 | 77.8 / 76.2 |
LIBERO (Table II). The suites are near-saturated, yet TemporalFlow-VLA averages 97.63±0.26% and reaches 96.60±0.87% on LIBERO Long — 2.1 points above the strongest listed prior mean (HAMLET's 92.2 on Long). The gain concentrates exactly where the method aims: multi-stage manipulation where knowing how the current state was reached matters.
Ablations (Table III). Two queries without flow already beat raw multi-frame (83.2%/83.6% vs 79.8%/81.5% Clean/Rand.); adding flow supervision reaches 85.5%/84.2% and contributes +3.7 points at H=2. Q8 alone (84.8%) edges out Q15 alone (83.9%), but both together are best — the scales are complementary. Because "2Q w/o Flow" keeps identical query slots and history window, the remaining gap isolates flow supervision from temporal capacity. Offline perturbations on six tasks confirm the learned representation is order-sensitive: both removing and shuffling history raise the action flow-matching loss, with shuffling worst on five of six tasks.
Efficiency and real robot. Asynchronous caching reduces server-side policy sampling time by 7.8% over a matched multi-replan control segment — temporal conditioning that is faster than not conditioning. On the AgiBot A3, Three-Cup Stacking improves 57.8% → 77.8% (+20.0) and Two-Bottle Packing 86.7% → 97.8% (+11.1); the larger gain lands on the task that demands preserving progress across sequential placements. No geometric inputs are required at deployment.
| Task | Baseline SR (%) | TemporalFlow-VLA SR (%) | Gain (pp) |
|---|---|---|---|
| Three-Cup Stacking | 57.8 | 77.8 | +20.0 |
| Two-Bottle Packing | 86.7 | 97.8 | +11.1 |
7. Limitations
The authors themselves flag the main open question: the temporal scale is fixed — three observations at t−15/t−8/t — rather than searched. How history length and sampling granularity should adapt to task horizon is unexplored, and the optimal temporal horizon may differ across embodiments and tasks. The training-only label pipeline also presumes accurate joint-state recording, URDF geometry, and camera calibration; the 5 mm correspondence gate and 0.1 coverage threshold are design choices whose sensitivity is not swept.
From a third-party perspective: (i) the supervision signal covers robot-surface motion only — object motion is available to the pathway through raw RGB but receives no explicit geometric supervision, so tasks whose ambiguity lives in object state rather than robot pose may benefit less; (ii) the strongest LIBERO baselines are near saturation, so the headline LIBERO average is a modest +0.63 over HAMLET and the real differentiator is LIBERO Long and RoboTwin; (iii) real-robot evaluation uses 45 trials per task per method — meaningful, but with wide confidence intervals; (iv) the comparison relies partly on baseline numbers reported by other papers (π0/π0.5 via LingBot-VA, X-VLA† via GigaWorld-Policy's re-evaluation), not same-lab reruns of every baseline.
8. Summary and Outlook
TemporalFlow-VLA's thesis is that history should be supervised, not merely stacked. By projecting recorded kinematics into the policy's own RGB plane, it manufactures an explicit, interpretable signal — robot-surface temporal flow — that teaches two compact queries what physically changed between frames, while a masked-attention bottleneck forces all historical influence through them. The result is order-sensitive history use at zero deployment-time geometry cost, with gains that grow exactly where multi-stage execution demands memory: +8.0 points on randomized RoboTwin, +14.5 at horizon 3, +20 on the harder real-robot task — plus an inference scheme that ends up 7.8% faster than the baseline path. It is a pragmatic recipe: auxiliary geometric supervision at train time, a compact supervised interface at run time.
Golden Quote
"Frames tell you what the world looks like; temporal flow tells you what just happened. TemporalFlow-VLA makes the second one a supervised subject instead of hoping attention figures it out."
SOURCE LINKS
