MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction

Paper: MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction
Link: arXiv:2607.16192v1, 2026; Project page: motionforesight.github.io
Code: Project page mentioned; no direct GitHub link in paper (⏳ pending)

1. Abstract

This paper addresses forecasting future 3D scene flow of manipulated objects from casual monocular human videos. The key insight: large-scale video models already encode temporal priors about contact, affordance, object persistence, and plausible short-horizon dynamics, but their native output is future pixels or video latents rather than explicit geometric motion. MotionForesight does not generate future RGB frames and then recover motion — it directly predicts the future 3D trajectory field of object-attached reference points. Built on TrackCraft3R (a feed-forward dense 3D tracker on a video DiT, Wan2.1), the model converts the retrospective tracker into a forecasting model by applying learnable mask latents to future frames. Training only a lightweight LoRA adapter on 40K SSv2 human-interaction videos achieves compelling motion forecasting without language or action labels.

2. Background and Motivation

"One of the most fundamental properties of thought is its power of predicting events" — Kenneth Craik. When people watch a hand approach a mug, a knife press into fruit, or a cabinet door begin to move, they infer the likely future configuration before the motion completes. This ability goes beyond visual prediction, reflecting understanding of object affordances, contact, constraints, and goals.

Limitations of existing methods:

• Language/action-conditioned methods: Rely on language descriptions or action labels to predict sparse 3D trajectories or rigid 6-DoF trajectories, but cannot naturally describe articulated parts, deformable surfaces, or local non-rigid motion.
• 2D point trajectory methods: Predict image-plane 2D tracks, lacking metric 3D information for downstream control.
• Video generation methods: Generate future pixels then recover motion afterward; computationally expensive and visually plausible pixels don't necessarily preserve point correspondence or metric 3D motion.

MotionForesight chooses dense, reference-anchored 3D scene flow as the output representation — metric rather than image-plane, object-centered rather than embodiment-specific, and not restricted to a single rigid pose parameterization.

3. Core Method

3.1 Problem Formalization

Given $T_1$ observed RGB frames $I_{0:T_1-1}$, predict the future 3D trajectories of reference points on the manipulated object for $T_2$ steps. Estimate a pointmap $P_t \in \mathbb{R}^{H\times W\times 3}$ for each frame, sample query points $\mathcal{Q}$ from the object mask in reference frame $a=0$, and predict future 3D locations:

$$\hat{X}_{t}(q)\in\mathbb{R}^{3},\qquad q\in\mathcal{Q},\quad t=T_{1},\ldots,T_{1}+T_{2}-1.$$

Default: $T_1=7$ (observed), $T_2=15$ (predicted), $T=22$. No language or action inputs. All geometry expressed in last-observed camera frame $t=T_1-1$ to reduce apparent camera motion.

3.2 Data Curation: From RGB to 3D Trajectories

Pseudo-ground-truth supervision from ~40K SSv2 human-object interaction videos. Pipeline:

1. Object segmentation: Use dataset object name as cue, initialize SAM on anchor frame, propagate mask forward/backward;
2. Monocular depth: Estimate per-frame depth with DepthAnything3, recover camera motion, construct temporally aligned pointmaps;
3. 3D tracking: Run TrackCraft3R on complete clips for reference-anchored 3D trajectories;
4. Coordinate transform: Transform tracks to last-observed camera frame, store validity masks.

Complete clips used only for offline pseudo-label generation; future frames/pointmaps never provided to the forecasting model during training.

3.3 Architecture: Dual-Latent Construction

Preserves TrackCraft3R's dual-latent construction. RGB and pointmap streams encoded separately and concatenated into visual-geometry latents:

$$c_{t}=\left[E^{\mathrm{rgb}}(I_{t});E^{\mathrm{pm}}(P_{t})\right],\qquad t<T_{1}.$$

where $E^{\mathrm{rgb}}$ is the RGB VAE encoder and $E^{\mathrm{pm}}$ is the pointmap VAE encoder. RGB stream captures appearance, contact, and interaction cues; pointmap stream provides metric scene structure and observed 3D motion.

Query stream repeats reference-frame state at every timestamp:

$$r_{t}=\left[E^{\mathrm{rgb}}(I_{a});E^{\mathrm{pm}}(P_{a})\right],\qquad a=0,\qquad t=0,\ldots,T-1.$$

Repeating the query stream turns every temporal output slot into the same geometric question: where is each reference-frame point at time $t$? The transformer receives context and query latents with temporal RoPE, predicts residual-track latent $\hat{z}^{\Delta}_{t}$, and a frozen track decoder maps it to a 3D residual added to the reference point:

$$\hat{X}_{t}(q)=X_{0}(q)+D^{\mathrm{track}}(\hat{z}^{\Delta}_{t})(q).$$

Figure 3: Architecture. Observed frames produce RGB and pointmap geometry latents; future geometry slots filled by learnable mask latents; frame-0 query latent repeated across timestamps; frozen video DiT + fresh LoRA predicts future residual-track latents.

3.4 Turning Tracking into Forecasting: Mask Latent Mechanism

Key innovation — converting retrospective tracker to forecaster: hide all future observations.

Observed timestamps contain actual visual-geometry latents:

$$\tilde{c}_{t}=\left[E^{\mathrm{rgb}}(I_{t});E^{\mathrm{pm}}(P_{t})\right],\qquad t<T_{1}.$$

Future timestamps: no RGB image or pointmap available, replaced by learnable mask latents:

$$\tilde{c}_{t}=\left[m^{\mathrm{rgb}};m^{\mathrm{pm}}\right],\qquad t\geq T_{1}.$$

RGB mask latent $m^{\mathrm{rgb}}$ and pointmap mask latent $m^{\mathrm{pm}}$ are shared across future timestamps and broadcast spatially — they don't encode specific future appearance or geometry, only indicate the corresponding temporal slot is unobserved. Temporal RoPE assigns each slot a distinct time index, allowing the transformer to distinguish near- and long-horizon predictions.

flowchart LR
    A[Observed RGB I_0..I_6] --> B[RGB VAE E_rgb]
    A2[Pointmaps P_0..P_6
DepthAnything3] --> C[Pointmap VAE E_pm]
    B --> D[Observed Context Latent c_t]
    C --> D
    E[Ref Frame a=0 Encoded] --> F[Query Latent r_t
Repeated Across Time]
    G[Future Slots t≥T1] --> H[Learnable Mask Latent
m_rgb + m_pm]
    D --> I[Frozen Video DiT + LoRA]
    F --> I
    H --> I
    I --> J[Residual Track Latent z_hat]
    J --> K[Frozen Track Decoder D_track]
    K --> L[Future 3D Trajectory
X_hat_t = X_0 + D_track z_hat]

3.5 Training Strategy

Freeze base video model, TrackCraft3R adapter, VAE encoders/decoders. Train only: fresh LoRA adapter, I/O projections, prediction head, mask latents. Loss is L2 on decoded 3D trajectories for observed frames:

$$\mathcal{L}_{\mathrm{dec}}=\lambda_{\mathrm{obs}}\frac{\sum_{t<T_{1}}\sum_{q}v_{tq}\left\|\hat{X}_{t}(q)-X_{t}(q)\right\|_{2}^{2}}{\sum_{t<T_{1}}\sum_{q}v_{tq}+\epsilon}$$

Loss back-propagates through the frozen track decoder, but decoder parameters are not updated. Default $\lambda_{\mathrm{obs}}=0.25$.

4. Key Experiments

4.1 Main Comparison (SSv2 + OOD Phone)

Evaluated on 150 SSv2 held-out videos and 50 independent phone videos (OOD). Reports ADE, FDE, PWT@5cm in cm.

MotionForesight wins all SSv2 and OOD metrics without language or action labels. MolmoMotion uses a 1M-video corpus; MotionForesight uses only 40K SSv2 videos.

Figure 1: MotionForesight forecasts plausible motion in diverse everyday manipulation scenarios.

4.2 Long-Context Evaluation

Observe first 50%, predict remaining 50%. MotionForesight substantially outperforms MolmoMotion: ADE 10.20cm vs 11.90cm (no lang.) / 12.57cm (with lang.). Geometric grounding helps convert richer observed context into metrically consistent future motion.

Figure 4: Qualitative results for future 3D track predictions. From only the observed prefix, the model predicts lifting, translation, rotation, constrained sliding, and local non-rigid motion.

4.3 Data Scaling Ablation

The 40K model is strongest on all SSv2 metrics and highest OOD PWT. Greater data diversity improves the motion prior, though this gain isn't always monotonically reflected by single-modal metrics.

4.4 Motion-Conditional Dynamics Analysis

Standard metrics (ADE/FDE/PWT) compare deterministic predictions with a single recorded future, potentially favoring conservative predictions. Authors propose motion-conditional diagnostics:

• TVO (Trajectory-Vector Overlap): Compares per-frame predicted and ground-truth displacement vectors; high credit only when a point moves in the correct direction, by the correct amount, at the correct time;
• VVO (Velocity-Vector Overlap): Same overlap applied to frame-to-frame velocity; more sensitive to acceleration, turns, stops;
• MoveF1 / MoveIoU: Evaluates whether correct object points move; MoveIoU compares peak excursion per point;
• DQS (Dynamics Quality Score): Combines TVO and MoveF1, $\mathrm{DQS}_{c}=\sqrt{\mathrm{TVO}_{c}\,\mathrm{MoveF1}_{c}}$.

5. Limitations and Future Work

• Deterministic prediction: Current model predicts one future, but short interaction prefixes may support multiple physically plausible outcomes; multi-hypothesis or probabilistic forecasting could represent alternatives and expose calibrated uncertainty.
• Pseudo-ground-truth supervision: Errors from monocular depth, camera estimation, segmentation, and tracking can propagate into training and evaluation; inference remains sensitive to observed pointmap estimation errors.
• Limited training data: Only 40K SSv2 manipulation clips; robustness to head-mounted egocentric video, strong camera motion, long-horizon interactions, or non-manipulation dynamics not established.
• Downstream connection: Future work should connect predicted 3D tracks to downstream planning and control, testing whether object-centered motion priors from passive human video improve embodied decision-making.

6. Conclusion

MotionForesight demonstrates that large video priors can be efficiently redirected from "rendering future appearance" toward "forecasting explicit, actionable 3D dynamics" without language or action supervision. By applying a minimal modification to the TrackCraft3R tracking model — replacing future observations with learnable mask latents — and training only a lightweight LoRA adapter, the model converts retrospective 3D reconstruction into prospective prediction. Reference-anchored 3D tracks expose the metric variable needed for physical reasoning without restricting objects to a single rigid 6-DoF pose, allowing one representation to describe translation, rotation, articulated motion, and local deformation. Quantitative, motion-conditional, and qualitative results show the model captures meaningful interaction dynamics, scales with human video, and transfers to phone captures of home and office scenes.