PAPER DEEP DIVE
Exo2EgoPose: Leveraging Exocentric Demonstrations for Vision-Language guided Egocentric 3D Hand Pose Forecasting
Perceiving multimodal cues and forecasting fine-grained actions from an egocentric (Ego) perspective is vital for applications like robot manipulation. However, previous studies either rely mainly on under-informed visual inputs to predict coarse human motions or follow the VRM/VLA paradigm, which suffers from insufficient robot data and the gap between human and robot embodiments. We observe that 3D hand pose naturally serves as a unified representation to bridge human-robot actions. Hence, we investigate an under-explored Vision-Language guided Egocentric 3D Hand Pose Forecasting (VL-EHPF) task, which aims to predict future Ego 3D hand poses from visual observations, a language instruction, and pose states. To overcome the limited field-of-view and highly dynamic motions in the Ego view, we propose a framework dubbed Exo2EgoPose, which innovatively leverages holistic and stable exocentric (Exo) demonstrations as guidance to compensate for partial and dynamic Ego-view cues. Specifically, we introduce a Dual-level Exocentric Reconstruction Module (DERM), which incorporates the paired Exo videos as supervision to reconstruct their video-level and chunked frame-level representations, thereby modeling spatial contexts and temporal dynamics. Then, the Global-to-Local Modulation Module (GLMM) utilizes the reconstructed hierarchical Exo representations for progressive feature refinement via attention mechanisms and adaptive modulation, enabling comprehensive Exo guidance for accurate Ego hand pose forecasting. Extensive experiments on \textit{AssemblyHands}, \textit{Ego-Exo4D}, and our newly constructed \textit{EgoMe-pose} benchmarks show the superiority of our method, which outperforms state-of-the-art methods by a large margin. Moreover, it demonstrates an effective human-to-robot transfer capability and yields improvements on the \textit{CALVIN} dataset. Code will be released.
Exo2EgoPose: Leveraging Exocentric Demonstrations for Vision-Language guided Egocentric 3D Hand Pose Forecasting
Institution: UESTC | arXiv: 2607.15890v1
One-Sentence Summary
Exo2EgoPose leverages exocentric (Exo) demonstrations to guide egocentric (Ego) 3D hand pose forecasting via a Dual-level Exocentric Reconstruction Module (DERM) and Global-to-Local Modulation Module (GLMM), outperforming SOTA by large margins on three benchmarks and demonstrating effective human-to-robot transfer.
Background and Motivation
Forecasting fine-grained actions from egocentric perspective is vital for robot manipulation. Existing methods rely on under-informed visual inputs or suffer from insufficient robot data and human-robot embodiment gaps. 3D hand pose naturally serves as a unified representation bridging human-robot actions. The VL-EHPF task predicts future Ego 3D hand poses using visual observations, language instructions, and pose states. Core challenge: limited FOV and highly dynamic motions in Ego view. Solution: leverage paired Exo videos for holistic global context.
Figure 1: VL-EHPF task — predict future 3D hand poses from Ego view using Exo guidance.
Task Definition
At timestamp $t$, model $\mathcal{M}$ takes Ego frames $O_{t-T'+1:t} \in \mathbb{R}^{T' \times H \times W \times 3}$, language $l$, and hand poses $S_{t-T'+1:t} \in \mathbb{R}^{T' \times 42 \times 3}$, predicting future keypoints:
$$\hat{S}_{t+1:t+\bar{T}} = \mathcal{M}(l, O_{t-T'+1:t}, S_{t-T'+1:t})$$42 = 21 joints × 2 hands. Also predicts joint validity $\hat{V} \in \{0,1\}^{\bar{T} \times 42}$.
Method
Multimodal Tokenization
Modality-specific encoders: text $\mathcal{E}_L$, MAE $\mathcal{E}_M$ (pixel-level), DINOv2 $\mathcal{E}_D$ (semantic), pose $\mathcal{E}_P$:
$$f_L = \mathcal{E}_L(l), \quad F_M = \mathcal{E}_M(O), \quad F_D = \mathcal{E}_D(O), \quad F_P = \mathcal{E}_P(S)$$
Figure 2: Exo2EgoPose framework — tokenization + DERM + GLMM + Transformer.
Dual-level Exocentric Reconstruction (DERM)
Reconstructs video-level and chunked frame-level Exo representations from Ego inputs. Learnable queries $Q = \{q_p; q_v; q_f; q_e\}$. Transformer processing:
$$\{q'_p, q'_v, q'_f, q'_e\} = \mathcal{T}(\{w, M, D, P\}, q_p, q_v, q_f, q_e)$$Video-level Exo representation via frozen MAE + Perceiver Resampler:
$$v^{\text{exo}} = \frac{1}{T' + \bar{T}} \sum_{i=t-T'+1}^{t+\bar{T}} v_i^{\text{CLS}}$$Global-to-Local Modulation (GLMM)
Uses reconstructed hierarchical Exo representations for progressive Ego feature refinement via attention and adaptive modulation units (AMU). GMM captures temporal trends (MPJVE), LMM improves spatial positions (MPJPE). Adaptive modulation:
$$\hat{F} = \gamma \odot F + \beta, \quad \gamma = \text{MLP}(v^{\text{exo}}), \quad \beta = \text{MLP}(v^{\text{exo}})$$MPJPE metric: $\text{MPJPE} = \frac{1}{N \times 42} \sum_{n=1}^{N} \sum_{j=1}^{42} \| \hat{\mathbf{p}}_{n,j} - \mathbf{p}_{n,j} \|_2$
Figure 3: DERM and GLMM architecture — dual-level Exo reconstruction + global-to-local modulation.
graph TD
A["Input: Ego frames + Language + Pose"] --> B["Multimodal Tokenization
MAE + DINOv2 + Text + Pose encoders"]
B --> C["Multimodal Transformer"]
D["Paired Exo video"] --> E["DERM: Dual-level Reconstruction
Video-level + Frame-level Exo"]
E --> F["GLMM: Global-to-Local Modulation
GMM + LMM + AMU"]
C --> F
F --> G["Predict future 3D hand poses
42 keypoints × T steps"]
G --> H["Human-to-Robot Transfer
CALVIN validation"]
Results
| Method | AssemblyHands MPJPE ↓ | Ego-Exo4D MPJPE ↓ | EgoMe-pose MPJPE ↓ |
|---|---|---|---|
| Random baseline | Higher | Higher | Higher |
| GR-1 | 39.57 | — | — |
| AR-VRM (+Exo) | 32.93 | — | — |
| Exo2EgoPose | 25.37 | Best | Best |
vs GR-1: -14.20/-2.34 MPJPE/MPJVE on AssemblyHands, -7.85/-0.38 on Ego-Exo4D, -9.66/-8.00 on EgoMe-pose. vs AR-VRM: -7.56/-1.51, -8.02/-0.27, -6.62/-6.63.
Ablation Study
| Config | MPJPE ↓ | MPJVE ↓ | Notes |
|---|---|---|---|
| Full model | 25.37 | 6.29 | GMM+LMM+VER+CFER |
| Vanilla Transformer (no GMM/LMM) | 26.44 | 6.57 | Cross-attn+AMU superior |
| w/o GMM | 27.03 | 6.56 | GMM captures temporal trends |
| w/o LMM | 27.26 | 6.48 | LMM improves spatial positions |
| w/o GLMM (all) | 30.11 | 7.76 | Exo modulation critical |
| w/o video-level Exo recon | 32.94 | 8.07 | Video-level Exo important |
| w/o frame-level Exo recon | 33.22 | 8.06 | Frame-level Exo important |
| w/o all Exo | 39.70 | 8.87 | Exo demonstrations essential |
Figure 4: Predicted Ego 3D hand poses — Exo2EgoPose more accurate.
Human-to-Robot Transfer
Effective transfer to CALVIN dataset. 3D hand pose as unified bridge representation. Transfer via feature alignment:
$$\mathcal{L}_{\text{transfer}} = \|\phi_{\text{human}}(S) - \phi_{\text{robot}}(a)\|^2$$
Figure 5: Human-to-robot transfer — 3D hand pose as unified bridge representation.
The newly constructed EgoMe-pose benchmark focuses on egocentric hand pose prediction evaluation. Model complexity analysis shows Exo2EgoPose has comparable parameters and inference speed to baselines, with negligible overhead from Exo guidance. Hyperparameter sensitivity analysis demonstrates robustness to observation window $T'$ and prediction steps $\bar{T}$.
Limitations
- Requires paired Exo videos, limiting applicability in scenes without Exo cameras
- Exo video quality affects guidance effectiveness
- 42-keypoint hand representation may be insufficient for full hand geometry
- Human-to-robot transfer still has embodiment gap requiring further study
Conclusion and Outlook
Exo2EgoPose leverages exocentric demonstrations for egocentric 3D hand pose forecasting. DERM constructs Ego-Exo cross-view correspondence via dual-level reconstruction; GLMM injects Exo guidance through global-to-local modulation. Outperforms SOTA by large margins on three benchmarks and demonstrates human-to-robot transfer capability. 3D hand pose as a unified bridge representation offers a new path from human demonstrations to robot manipulation.
Key Insight: 3D hand pose naturally serves as a unified representation bridging human-robot actions — predictable from human Ego view and usable as action priors for robot manipulation.
SOURCE LINKS
