PAPER DEEP DIVE
Learning Diverse Humanoid Tasks via Synthetic Video Scenarios without Real World Data
The human-like morphology of humanoid robots grants them exceptional potential for agile and versatile motor capabilities, but it also introduces significant challenges in acquiring complex skills. Traditional Learning-from-Demonstrations methods are often constrained by the high cost of collecting real-world data, the difficulty of capturing motion-specific behaviors, and the limited diversity of demonstrations across individuals. Moreover, even for the same task, humans may execute the motion in multiple distinct ways. In this paper, we propose a new framework that leverages the power of Generative AI to convert textual prompts into realistic and diverse sequences of human body movements, enabling the robot to observe multiple variations of how a single task can be performed. These synthetic demonstrations are then used as a training resource, allowing the robot to learn a broad range of task-execution styles without requiring direct human intervention. We evaluate the proposed method across four simulation scenarios. Experimental results show that the robot not only completes the tasks successfully but also demonstrates strong adaptability to complex variations in motion.
Learning Diverse Humanoid Tasks via Synthetic Video Scenarios without Real World Data
Authors: Yun-Hao Tsai, Cong-Thanh Vu, Yen-Chen Liu | Institution: National Cheng Kung University, Taiwan | arXiv: 2607.21648v1
One-Sentence Summary
This paper leverages Generative AI (Google Veo 3.1) to produce human motion videos from text prompts, converts them to humanoid reference motions via pose extraction and retargeting, and trains a Unitree G1 RL tracking policy to learn diverse tasks — no real-world motion capture data required. Validated on four simulation scenarios including lie-and-stand, boxing, and pick-and-place.
Background and Motivation
Humanoid robots have potential in human environments, but acquiring complex skills is challenging. Traditional LfD methods are constrained by: high cost of real-world data collection, difficulty of motion capture for specific behaviors, and limited demonstration diversity across individuals. Even for the same task, different people perform actions differently.
Figure 1: Multi-modal prompts guide Veo 3.1 to generate human motion videos → pose extraction and retargeting → motion chaining → RL motion tracking.
DeepMimic and AMP frameworks excel in physics simulation but heavily rely on human demonstration data, especially for complex tasks requiring high-precision motion capture. Generative AI advances offer new opportunities for synthesizing complex human behaviors.
Framework Architecture
Three components: (1) converting generated videos to humanoid motion; (2) motion chaining; (3) learning motion policies.
Robot State Representation
$$\mathbf{q}=\left[p_{\text{root}},\,q_{\text{root}},\,\theta_{1},\ldots,\theta_{n}\right]^{T}\in\mathbb{R}^{n+7}$$where $p_{\text{root}} \in \mathbb{R}^3$ is global root position, $q_{\text{root}} \in \mathbb{S}^3$ is root orientation (unit quaternion), $\theta_i$ are joint angles. Strict tracking is impractical due to high DoF and morphological differences; a DeepMimic-inspired RL framework relaxes strict tracking while encouraging human-like motion.
Video to Reference Motion
Structured prompts ensure consistent visual conditions. Two-stage processing: (1) SMPL-X parameter estimation via enhanced Vision Transformer; (2) GMR retargeting to humanoid — aligning initial poses, scaling limb lengths, mitigating foot sliding/ground penetration, final IK optimization under joint-limit constraints.
Figure 2: Instruction prompt defines video generation requirements; user prompt specifies desired content.
Motion Stitching
Each sequence from independent videos has its own local coordinate system. Direct concatenation causes physical discontinuities. Solution: (1) root coordinate alignment — transform second sequence to match first's terminal root pose; (2) joint-configuration smoothing — insert transition buffer with interpolated joint angles.
flowchart TD
A["Text prompts
(instruction + user)"] --> B["Veo 3.1 video generation
multiple variants per task"]
B --> C["SMPL-X pose extraction
Vision Transformer"]
C --> D["GMR retargeting to Unitree G1
IK optimization + joint limits"]
D --> E["Motion chaining: root alignment + smoothing"]
E --> F["RL tracking policy training
PPO + asymmetric actor-critic"]
F --> G["Deployment: diverse task execution"]
RL-Based Motion Tracking
Observation Space
Root-relative state representation. Per-step observation:
$$o_{t}=\left[p^{rel}_{1:n},\;q^{rel}_{1:n},\;v^{rel}_{1:n},\;\omega^{rel}_{1:n},\;e_{1:n},\;a_{t-1}\right]$$Per-link tracking error:
$$e_{1:n}=\left(p_{1:n}^{rel}-p_{1:n}^{ref},\;q_{1:n}^{rel}\ominus q_{1:n}^{ref}\right)$$Critic receives privileged observation $o_{\text{priv},t}$ with noise-free kinematics and target references.
Reward Function
$$r_{t}=r^{I}_{t}-r_{\text{limit}}-r_{\text{smooth}}-r_{\text{contact}}$$Motion tracking reward:
$$r^{I}_{t}=w_{p}r^{p}_{t}+w_{v}r^{v}_{t}+w_{e}r^{e}_{t}+w_{c}r^{c}_{t}$$Pose reward:
$$r^{p}_{t}=\exp\Bigg(-\alpha_{p}\sum_{i=1}^{n}\|q_{i}^{rel}\ominus q_{i}^{ref}\|^{2}\Bigg)$$Velocity reward:
$$r^{v}_{t}=\exp\Bigg(-\alpha_{v}\sum_{i=1}^{n}\|\omega_{i}^{rel}-\omega_{i}^{ref}\|^{2}\Bigg)$$End-effector reward:
$$r^{e}_{t}=\exp\Bigg(-\alpha_{e}\sum_{e\in E}\|p_{e}^{rel}-p_{e}^{ref}\|^{2}\Bigg)$$Center-of-mass reward:
$$r^{c}_{t}=\exp\Big(-\alpha_{c}\|p_{c}^{rel}-p_{c}^{ref}\|^{2}\Big)$$Weights: $w_p=0.65, w_v=0.1, w_e=0.15, w_c=0.1$; scaling: $\alpha_p=2.0, \alpha_v=0.1, \alpha_e=40.0, \alpha_c=10.0$. Regularization: joint limit, action smoothness ($r_{\text{smooth}}=w_{\text{smooth}}\|a_t-a_{t-1}\|^2$), self-collision penalties.
Experimental Results
Trained in NVIDIA Isaac Lab, 4096 parallel agents, Intel Xeon W5-3435X + RTX 4000 Ada, PPO with asymmetric actor-critic (3-layer FC [512,256,128], ELU activations).
Motion Generation Evaluation
50 everyday task prompts tested with Google Veo 3 API, 10 videos per prompt. Most tasks produce accurate, diverse motion. High-speed actions (backflips) exhibit temporal inconsistencies.
Policy Evaluation
| Task | MAE range (m) | 0.5kg payload effect | Notes |
|---|---|---|---|
| Lie-and-stand | 0.04~0.07 | Slight upper-body increase | Full-body transition |
| Boxing | 0.04~0.07 | Notable upper-body increase | Dynamic upper-body |
| Pick-and-place | 0.04~0.07 | Lower body stable, upper torque ↑ | Coordinated manipulation |
Figure 3: Qualitative results — lie-and-stand, boxing, pick-and-place via RL tracking.
Figure 4: Wrist/ankle vertical trajectory alignment, joint torque profiles, with/without 0.5kg payload.
Figure 5: Upper/lower body joint position MAE across three tasks, loaded vs. unloaded.
Randomization and Robustness
Minimal randomization set applied during training: ground friction/restitution, nominal joint configuration (modeling joint offset errors), torso center of mass, and random velocity perturbations for disturbance robustness.
| Training Config | Value | Notes |
|---|---|---|
| Parallel agents | 4096 | NVIDIA Isaac Lab CUDA-accelerated |
| RL algorithm | PPO | Asymmetric actor-critic |
| Network structure | [512, 256, 128] | 3-layer FC, ELU activations |
| Video generation | Google Veo 3.1 | 10 variants per task |
| Pose extraction | SMPL-X + ViT | With bbox, body-shape priors, temporal module |
| Retargeting | GMR | IK optimization + joint limits |
| Test prompts | 50 | Everyday task prompts |
Motion Generation Success Rate Analysis
Evaluation of 50 everyday task prompts shows most tasks (carrying, walking, grasping) produce physically plausible, correctly represented motion. However, high-speed dynamic actions (backflips, rapid rotation) have lower success rates with temporal inconsistencies — non-physical teleportation or velocity spikes mid-motion. This reflects current video generation models' inherent limitation with rapid, highly dynamic motions: sparse training data distribution and harder temporal consistency constraints at high speeds.
Comparison with DeepMimic/AMP
Traditional DeepMimic relies on expensive MoCap data with single demonstrations limiting behavioral diversity. AMP extracts motion priors via adversarial learning but still requires real motion data. This work shifts data sourcing from physical capture to generative models, generating 10 variants per task that naturally capture multiple ways humans perform the same task. The trade-off is that generated videos have lower physical consistency than MoCap — some motions may contain minor non-physical segments — but RL tracking policy randomization and regularization absorb this noise to some extent.
Limitations
- High-speed dynamic motions (backflips) show temporal inconsistencies in generated videos.
- Only validated in simulation; no sim-to-real transfer demonstrated.
- Limited physical consistency in generated motions; some may violate physics constraints.
- 50 everyday tasks is a limited evaluation scope; scalability to complex tasks unverified.
Conclusion and Outlook
This framework leverages Generative AI to synthesize human motion data for humanoid robot training without traditional motion capture. The complete pipeline — prompt-driven video generation → retargeting → motion chaining → RL tracking — enables scalable, diverse motion acquisition. The motion chaining mechanism allows a single policy to execute extended compound behaviors. Simulation results demonstrate successful learning of dynamic and manipulation tasks with accurate motion tracking and stability.
Key insight: "Replace real demonstrations with generated video" — shifting the data collection bottleneck from the physical world to generative models provides a scalable new paradigm for humanoid skill learning. Future work: sim-to-real transfer, enhanced physical consistency, extension to more complex tasks.
Deep analysis generated by RobotWorld paper-detail-generator based on full-text reading | arXiv:2607.21648v1
SOURCE LINKS
