BeyondRetarget: Learning Executable Humanoid Motions Directly from Monocular Video

Figure 1: BeyondRetarget encodes the video into a shared motion space and directly decodes robot-specific trajectories — no explicit human intermediate representation.

Background: Structural Flaws of the Two-Stage Pipeline

Executable whole-body motion data is the foundation of humanoid skill learning and whole-body control. Traditional acquisition relies on optical motion capture (Vicon) or wearable inertial sensors — expensive hardware in controlled venues, impossible to scale. Monocular RGB video, in contrast, is abundant and nearly free to collect, promising a scalable low-cost channel for robot motion data.

Existing work almost universally follows a two-stage route: first reconstruct explicit SMPL human motion from video (WHAM, GVHMR, etc.), then retarget it to robot joint space (GMR, NMR, etc.). This works, but has three structural problems:

First, the intermediate representation is human-centric. SMPL describes human bodies, while humanoids differ substantially in body proportions, joint degrees of freedom, and dynamics. Describing robot motion in a human-centric space is inherently awkward, forcing the retargeting stage to "wrench" the motion into shape.

Second, errors accumulate one-way. Human motion estimation errors from stage one propagate intact into retargeting, compounding with retargeting errors; joint optimization cannot eliminate what upstream has already fixed. The paper's Oracle experiment proves it directly: even feeding perfect SMPL ground truth into retargeting still incurs non-negligible error.

Third, robot-specific binding. Learned retargeting (e.g., NMR) couples to one target robot; adapting to a new morphology requires fresh robot data.

BeyondRetarget's answer is decisive: since the intermediate stage is the problem, remove it entirely — stop reconstructing "how the human moved" and predict directly "how the robot should move."

Method

Problem Formulation

Given a monocular RGB video \(\mathcal{I}=\{\mathbf{i}_t\}_{t=1}^{n}\) of length \(n\) and a target robot configuration \(r\), directly estimate the robot's complete motion:

$$\hat{\mathcal{M}}^{(r)}=\{\hat{\mathbf{p}}_{t},\ \hat{\mathbf{o}}^{(r)}_{t},\ \hat{\mathbf{q}}^{(r)}_{t}\}_{t=1}^{n}$$

where \(\hat{\mathbf{p}}_t\in\mathbb{R}^3\) is root translation, \(\hat{\mathbf{o}}^{(r)}_t\in\mathbb{R}^6\) root orientation (6D rotation), and \(\hat{\mathbf{q}}^{(r)}_t\in\mathbb{R}^{d_r}\) the robot's \(d_r\)-dimensional whole-body DoF vector. The output dimension \(d_r\) varies per robot — this is what "robot-oriented" means: the representation serves each robot's joint space directly.

Architecture: Shared Representation + Robot-Specific Decoders

Figure 2: A frozen visual encoder plus temporal aggregation (TCAM) yields a shared motion representation; a robot-specific decoder outputs joint trajectories directly, followed by contact-aware refinement.

The architecture has three blocks:

(1) Visual and temporal encoding. A frozen visual encoder (4DHumans-style) encodes each frame; a Temporal Context Aggregation Module (TCAM) models cross-frame context, producing the shared motion representation \(\mathbf{u}_t\). This representation bridges human and humanoid motion spaces without belonging to either — it captures cross-morphology motion structure.

(2) Robot motion decoder. For target robot \(r\), decoder \(\Psi_r\) maps \(\mathbf{u}_t\) directly to robot motion, parameterized as a residual around a robot-specific nominal state:

$$[\hat{\mathbf{p}}_{t},\ \hat{\mathbf{o}}^{(r)}_{t},\ \hat{\mathbf{q}}^{(r)}_{t}]=\bm{\mu}^{(r)}+[\Delta\mathbf{p}_{t},\ \Delta\mathbf{o}^{(r)}_{t},\ \Delta\mathbf{q}^{(r)}_{t}]$$

where \([\Delta\mathbf{p}_t,\Delta\mathbf{o}^{(r)}_t,\Delta\mathbf{q}^{(r)}_t]=\Psi_r(\mathbf{u}_t)\) is the predicted residual and \(\bm{\mu}^{(r)}\in\mathbb{R}^{3+6+d_r}\) a learnable nominal state. Residual parameterization means the network only learns "deviation from the rest pose," naturally accommodating robots with very different DoF counts and default stances.

The decoder is an MLP trunk with three branches: root translation, root orientation, and a robot-specific DoF branch. Adding a new robot reuses the pretrained visual-temporal representation, MLP trunk, and root translation branch; only the DoF and root orientation branches are retrained — about one hour on a single RTX 4090 per new humanoid (the paper adapts Unitree G1/R1/H1, Fourier GR1-T1/GR2-V3, Booster T1, Tienkung, and Atlas — 8 humanoids in total).

Unified Motion Supervision: Non-Uniform Morphology Scaling

The hardest problem: no ground-truth humanoid trajectories exist in training data — only SMPL human annotations, whose joint dimensions and kinematic meanings differ entirely. The solution is non-uniform morphology alignment: "pinch" human keypoints into robot proportions.

Figure 3: Unified human-robot supervision. Human keypoints are rescaled link-by-link to robot morphology; keypoint positions and directed limb segments jointly supervise robot pose.

Specifically, define a set \(\mathcal{J}\) of 18 shared semantic keypoints covering pelvis, torso, shoulders, elbows, wrists, hands, hips, knees, ankles, and feet. Human-side keypoints \(\mathbf{a}_{t,j}\) come from SMPL joints; robot-side counterparts are \(\hat{\mathbf{x}}_{t,j}\). Taking the pelvis as root, each skeleton link is scaled independently to build the robot-proportioned target:

$$\mathbf{y}_{t,0}=\mathbf{a}_{t,0},\qquad \mathbf{y}_{t,j}=\mathbf{y}_{t,\pi(j)}+\frac{\ell^{(\mathrm{r})}_{j}}{\ell^{(\mathrm{h})}_{j}}\left(\mathbf{a}_{t,j}-\mathbf{a}_{t,\pi(j)}\right)$$

where \(\pi(j)\) is the parent of keypoint \(j\), and \(\ell^{(\mathrm{h})}_j\), \(\ell^{(\mathrm{r})}_j\) the human and robot link lengths. The scaling is per-link, non-uniform: a robot with longer legs and shorter arms rescales each accordingly — not one global factor. Ablations show this contributes most: removing it degrades MAMPJPE from 26.36 to 36.08 mm.

Supervision has two layers. The first is a keypoint geometry loss: predicted robot keypoints are Procrustes-aligned to the scaled human keypoints with a Smooth L1 penalty:

$$\mathcal{L}_{\mathrm{geo}}=\frac{1}{n|\mathcal{J}|}\sum_{t=1}^{n}\sum_{j\in\mathcal{J}}\rho\left(\tilde{\mathbf{x}}_{t,j}-\mathbf{y}_{t,j}\right)$$

The second is a directed link direction loss. Keypoint positions under-constrain local limb orientation, so a set \(\mathcal{D}\) of directed semantic links (upper arms, forearms, thighs, shanks, torso) is supervised by the cosine discrepancy between predicted and target link vectors:

$$\mathcal{L}_{\mathrm{dir}}=\frac{1}{n|\mathcal{D}|}\sum_{t=1}^{n}\sum_{d\in\mathcal{D}}\left(1-\hat{\mathbf{v}}^{\top}_{t,d}\mathbf{v}_{t,d}\right)$$

The full objective adds root translation/orientation losses, cross-frame temporal consistency \(\mathcal{L}_{\mathrm{temp}}\), and a robot joint-limit penalty \(\mathcal{L}_{\mathrm{lim}}\):

$$\mathcal{L}_{\mathrm{motion}}=\lambda_{\mathrm{geo}}\mathcal{L}_{\mathrm{geo}}+\lambda_{\mathrm{dir}}\mathcal{L}_{\mathrm{dir}}+\lambda_{\mathrm{trans}}\mathcal{L}_{\mathrm{trans}}+\lambda_{\mathrm{ori}}\mathcal{L}_{\mathrm{ori}}+\lambda_{\mathrm{temp}}\mathcal{L}_{\mathrm{temp}}+\lambda_{\mathrm{lim}}\mathcal{L}_{\mathrm{lim}}$$

The upshot: human annotations supervise robot motion directly, in a unified geometric space rather than either parameter space.

Contact-Aware Refinement

Visually accurate poses are not necessarily physically executable — robot motion executability hinges on temporal consistency and contact stability. The refinement module uses a two-level state decision:

Figure 4: Contact-aware kinematic refinement. Lower limbs are divided into contact/support states, each triggering different-strength post-processing.

First, freeze the motion estimator and train a contact predictor (supervised by pressure data from MotionPRO) that fuses motion and visual features to predict per-foot contact probability \(\hat{c}_{t,f}\in[0,1]\). Then a two-level decision: binarize probability into contact state \(c_{t,f}\in\{0,1\}\); obtain foot horizontal velocity \(v^{xy}_{t,f}\) via forward kinematics, and count a foot as supporting only if it is in contact AND nearly stationary:

$$s_{t,f}=c_{t,f}\wedge\left(\left\|v^{xy}_{t,f}\right\|_{2}\leq\tau_{v}\right)$$

This distinction is deliberate: a foot touching ground while dragging is physically different from a foot stably bearing weight. Contact state triggers root height correction (eliminating penetration/floating); support state further triggers root horizontal correction + lower-body inverse kinematics to guarantee zero support-foot sliding. To avoid abrupt switching artifacts, IK correction residuals pass through a temporal smoothing filter before being added back:

$$\Delta\mathbf{q}^{\mathrm{IK}}_{t}=\mathbf{q}^{\mathrm{IK}}_{t}-\hat{\mathbf{q}}_{t},\qquad \hat{\mathbf{q}}_{t}\leftarrow\hat{\mathbf{q}}_{t}+\operatorname{Filt}\left(\Delta\mathbf{q}^{\mathrm{IK}}_{t}\right)$$

Ablations: removing refinement worsens violation rate (foot floating/penetration) from 6.17% to 22.18% and foot slide from 1.71 to 4.51 mm/frame — motion looks fine but the robot physically cannot stand it.

Experiments

Comparison with Two-Stage Methods (Unitree G1)

Key readings:

End-to-end wins across the board. MAMPJPE 26.36 mm is ~33% lower than the best two-stage combo (39.19); root trajectory error drops from hundreds of mm to 80.74. The most striking gain is robustness: every two-stage method suffers severe pose collapse (windows exceeding 65/100 mm MAMPJPE); BeyondRetarget records zero on both Fail65 and Fail100 — not a single collapse.

Retargeting itself introduces error. The Oracle rows feed perfect SMPL ground truth into the retargeting module; methods with estimated inputs still lag far behind, and BeyondRetarget even beats GT→NMR — evidence that even a perfect upstream cannot lift the two-stage ceiling imposed by retargeting.

Physical simulation execution. Executing generated motions with the general motion tracker SONIC in MuJoCo, BeyondRetarget achieves 347/362 (95.86%) success with 35.04 mm executed error — both best.

Qualitatively (Figure 5), two-stage pipelines misplace limbs or collapse on large-amplitude arm swings; on fine-grained motions like "hands behind the back," all four two-stage baselines place the hands in front of the body — the ambiguity of the intermediate representation laid bare — while BeyondRetarget correctly reconstructs the behind-the-back pose.

Multi-Robot Adaptation

Eight morphologically diverse humanoids share one representation, with errors in a reasonable 26.36–72.05 mm band; on all four robots supported by both methods, BeyondRetarget beats GVHMR→GMR. Qualitative results further show generalization to in-the-wild Internet videos — the unified supervision genuinely transfers the shared visual motion representation across embodiments.

Real-Time Performance and Real Robot

The end-to-end architecture with a streamlined model achieves ~7× lower latency than two-stage pipelines (192.8 ms vs ~1.3 s), higher throughput, and lower memory — directly enabling real-time visual teleoperation: a human acts on camera and the robot follows live (Figure 6b). On the real robot, for a stretching motion BeyondRetarget raises the left arm overhead with the hand kept near the thigh while all four two-stage baselines show clear arm discrepancies; during single-arm circling and hands-on-head motions it maintains stable body motion and foot support while all baselines destabilize.

Ablations

Each component has its job: non-uniform alignment is the biggest contributor to pose accuracy (+9.7 mm if removed); the direction loss underpins limb orientation (EDE 15.29→21.78 without it); the temporal loss suppresses jitter and foot slide; contact refinement is the sole guardian of physical plausibility (violations nearly 4× without it) — notably it even slightly raises MAMPJPE (25.73→26.36), trading a little geometric accuracy for execution stability: an "executability-first" design choice.

Significance and Limitations

The industry significance is a data lever: inexhaustible Internet human video becomes executable motion data for 8 humanoids via one end-to-end model, with new-robot adaptation cost reduced to "one hour on a single GPU." This opens a genuinely scalable channel for humanoid imitation learning and whole-body control.

The authors are candid about limitations: constrained by model capacity and training data distribution, global trajectory estimation still errs on large-range motions or with visibly moving cameras; the current method does not cover complex terrains or contact-rich human-environment interaction. Future work: more diverse data, scene geometry and physical contact modeling, so robots can learn more challenging scene-conditioned motion skills directly from video.