Light-Loco-Parkour: Versatile Perceptive Whole-Body Locomotion via Multi-Skill Distillation

Light-Loco-Parkour: Versatile Perceptive Whole-Body Locomotion via Multi-Skill Distillation

Paper Info · Title: Light-Loco-Parkour: Versatile Perceptive Whole-Body Locomotion via Multi-Skill Distillation · Authors: Hongming Chen, Zhuoran Li, Hongxi Wang, Jiangpeng Hu, Ziliang Li, Peize Liu, QingRui Zhao, Xuhao Liu, Liang Pan, Ximin Lyu, Yuntao Ma, Tingxiang Fan · Affiliation: Light Origins · arXiv: 2608.02653 (Aug 2026) · Project: light-loco-parkour.github.io · Code: Organization established, code release pending

One-Sentence Summary

LightLP uses a single deployable policy, conditioned only on onboard depth and a velocity command, to let a humanoid robot autonomously decide when to walk, balance, climb, step down, or vault obstacles — with no reference input, skill label, or runtime motion graph.

1. Background and Motivation

Humanoid robots are designed to operate in cluttered, human-centric environments that wheeled and tracked machines cannot reach. Moving through such spaces is rarely a matter of the legs alone: traversing an obstacle often demands the entire body, with the hands vaulting, the knees bracing, and the torso leaning in continuous coordination with what the robot perceives. A humanoid that can sense the terrain ahead and recruit its whole body to negotiate it would unlock applications from search-and-rescue over rubble, to inspection of cluttered industrial sites, to operation in everyday human surroundings.

Recent learned controllers have demonstrated several humanoid capabilities: tracking whole-body reference motions, walking on real hardware, supporting teleoperated whole-body tasks. Most demonstrated capabilities, however, remain limited to flat or mildly uneven terrain. Traversal of rough, obstacle-rich environments remains uncommon, especially when success requires non-foot contacts.

Progress toward this setting has mainly followed two learning paradigms. The first is motion imitation: learning from a human-motion prior, feeding a retargeted reference trajectory into the policy for tracking. Representative works include BeyondMimic, which models actuators from first principles; ASAP, which learns a residual delta-action model from real-world rollouts; and KungfuBot, which pairs physically constrained motion processing with an adaptive tracking curriculum. These systems transfer agile skills such as cartwheels and kicks zero-shot to a Unitree G1. At larger scale, SONIC trains a single tracking policy on hundreds of hours of motion-capture data. However, these methods share a blind spot: skills make almost no contact with objects off the ground, fundamentally because human references rarely capture a motion together with the scene that shaped it.

The second paradigm is locomotion control: training robots via reinforcement learning to perceive terrain and react. These policies track velocity well in open terrain but are nearly helpless when obstacles require coordination of arms, torso, and knees — the locomotion paradigm inherently focuses on the feet and leaves the upper body idle.

The two lines excel in different domains but remain disconnected: imitation methods have whole-body motion but no terrain generalization; locomotion control has terrain perception but no whole-body engagement. LightLP's starting point is to bridge these two lines with a single policy — letting the same policy track velocity in open terrain, autonomously switch to whole-body traversal at obstacles, and resume walking afterward, all inferred from depth and a velocity command alone.

2. Preliminaries

POMDP formulation. LightLP formulates whole-body perceptive locomotion control as a partially observable Markov decision process. At each control step, the policy $\pi_\theta(a_t \mid o_t)$ maps an observation $o_t$ to an action $a_t$, optimized with PPO to maximize the expected discounted return:

$$\mathbb{E}\left[\sum_{t}\gamma^{t}r_{t}\right]$$

where $\gamma$ is the discount factor and $r_t$ is the reward specified per module. All training is carried out in the IsaacLab simulator.

Teacher-student distillation. The core architectural pattern of LightLP is the teacher-student scheme: a teacher is trained in simulation with privileged information (clean height scan), then distilled to a student using only deployable observations (onboard depth). The key design principle is "reasoning backward from the student" — the teacher's actor only receives quantities the student can itself infer, avoiding the information gap that would make distillation fail.

Reference State Initialization (RSI). Originating from DeepMimic, this technique resets each episode to a random reference frame, spreading exploration across the whole motion and exposing the policy to high-momentum mid-skill states (such as the apex of a vault and the push-off of a climb), without which dynamic skills fail to emerge.

3. Method

3.1 System Overview: Four-Module Pipeline

LightLP comprises four modules: perceptive locomotion learning (Section IV), data augmentation (Section V-A), skill learning and generalization (Section V-B), and transition learning (Section V-C), followed by a final depth distillation pass that ports the height-scan policy onto the onboard depth camera (Section VI).

Figure 1: LightLP system overview. (a) RL stage trains perceptive-locomotion and skill teachers; (b) multi-expert DAgger distills into a single height-scan policy with transition fine-tuning; (c) depth distillation for onboard deployment.

The core insight of this design is decoupling and divide-and-conquer: first train each module's teacher separately with privileged information, then progressively distill to deployable observations, and finally use transition learning to chain the modules into an end-to-end policy. Each step solves only one sub-problem, reducing the difficulty of simultaneously learning perception, skills, and switching.

3.2 Observation and Action Spaces

During teacher training, the observation includes privileged information, the velocity command $(v_x, v_y, \omega_z)$, and a local height scan. The deployable student is distilled from this height-scan teacher, using onboard depth in place of the privileged scan. The action $a_t$ sets target joint positions tracked by a PD controller, and the policy runs onboard at 50 Hz.

The teacher's network is entirely MLP-based with no recurrence, so a single observation carries no memory of the recent past. LightLP therefore stacks the last five observation frames to supply temporal context, enabling the policy to implicitly infer latent dynamics and contact state it cannot sense directly. The student, by contrast, encodes depth with an MLP, fuses it with the proprioception embedding, passes the result through an RNN, and decodes its hidden state with a final MLP into the action. The recurrence builds memory across frames so that the policy can implicitly infer unmeasured state such as the current velocity $(v_x^c, v_y^c, \omega_z^c)$ and retain terrain that has scrolled out of view.

3.3 Asymmetric Actor-Critic

The teacher uses an asymmetric actor-critic architecture: the critic accesses privileged information that the actor does not, yielding more accurate value estimates that help the teacher learn better. Granting privileged input to the teacher's actor would speed learning and yield a stronger-performing teacher, yet it would leave the student — which never sees that input — unable to infer the teacher's observation at distillation, making it unable to reproduce the teacher's behavior. This information gap is well documented in the literature.

LightLP therefore reasons backward from the student when designing the teacher's observation: the actor reads only quantities the student can itself infer — a noise-free height scan and proprioception subjected to mild domain randomization. Because the target terrains (stepping stones, balance beams) demand precise foot placement, the actor additionally receives a privileged contact flag. The critic receives clean, noise-free inputs augmented with oracle information. This "information alignment" principle ensures that teacher and student make decisions on the same informational basis during distillation.

3.4 Perceptive Locomotion Learning: Rewards and Environment

Perceptive locomotion training includes rich reward terms, termination conditions, terrain curriculum, and event domain randomization. Training terrains cover stairs, ramps, stepping stones, balance beams, and other types. Domain randomization covers friction, mass, motor torque, and other physical parameters to ensure sim-to-real transfer robustness.

Figure 2: Perceptive locomotion training terrain types, each column a distinct configuration.

A key design is the foot-mounted height scan illegal-footstep penalty: a height scan mounted at the foot detects whether the foothold is legal, penalizing unsafe placements. This is especially important in sparse-foothold scenarios like stepping stones.

Figure 3: Foot-mounted height scan computing the illegal-footstep penalty for safe stepping-stone traversal.

3.5 Data Augmentation: From Sparse Seeds to Terrain-Paired References

This is LightLP's first core innovation. Traditional methods require large motion-capture datasets to learn diverse skills, but LightLP starts from a single seed motion and generates dynamically feasible, terrain-paired references through iterative self-augmentation.

The seed motion can come from video (via GVHMR to recover SMPL) or motion capture. However, neither GVHMR nor GMR is aware of the obstacle, so the seed reference inevitably collides with it: the supporting hand sinks into the top face, and a knee clips through the front edge. LightLP uses the simulator itself as a retargeting resource — a reference tracked under physics comes out both in seamless contact with the obstacle and dynamically feasible.

The Object-Interaction Mimic stage extends BeyondMimic from free-space imitation to contact-rich, object-interacting skills. Three key modifications: (1) tracking the global root position instead of only the root-relative pose, anchoring every contact to a fixed location in the scene; (2) adding two privileged observations — distance to obstacle and its size — so the policy can tell how far the true surface lies from an imperfect seed; (3) adding a global-frame position tracking term for the body part meant to make contact.

The iterative self-augmentation loop works as follows: at iteration $i$, train policy $\pi_i$ on current reference $r_i$ and obstacle $d_i$, roll out K candidate trajectories in simulation, select the best-scoring $\hat{r}_i^\star$, and lift both the trajectory and the obstacle by $m$ cm ($m \in [5, 10]$ cm) to form the next iteration's $(r_{i+1}, d_{i+1})$. Formally:

$$\pi_i = \arg\min_{\pi}\,\mathcal{L}_{\text{mimic}}(\pi;\,r_{i},d_{i})$$ $$\hat{r}_{i}^{(k)} \sim \mathrm{Rollout}(\pi_{i},\,d_{i}),\quad k\in[K]$$ $$\hat{r}_{i}^{\star} = \arg\max_{k\in[K]}\,\mathrm{Score}\big(\hat{r}_{i}^{(k)},\,r_{i}\big)$$ $$(r_{i+1},\,d_{i+1}) = \big(\hat{r}_{i}^{\star}+m\,\mathbf{e}_{z},\;\;d_{i}+m\,\mathbf{e}_{z}\big)$$

where $\mathbf{e}_z$ is the gravity-axis unit vector. Because the rollout is produced by the policy under the physics engine and the robot's actuator limits, it is dynamically and kinematically feasible by construction, even though its kinematic shape gradually drifts from the original SMPL trajectory. Each iteration lifts 5-10 cm, extracting a slightly harder version of the same skill. The accumulated (motion, terrain) pairs form the augmented dataset.

Figure 4: Red markers indicate body parts where the global position reward is applied: wrist for climb-and-step, elbow for speed-vault, pelvis for reverse-vault.

3.6 Skill Learning and Generalization: Height-Scan Distillation

Although the Object-Interaction Mimic stage yields a feasible reference for each skill, the resulting experts are inherently neither generalizable nor deployable: the mimic expert is fed obstacle geometry directly and lacks odometry, so it cannot anchor itself to a global reference frame. Each expert reproduces a single motion tied to one obstacle, rather than a behavior that transfers across geometry or runs from onboard sensing alone.

LightLP wants a whole-body skill that runs on the same observation as the perceptive locomotion policy: proprioception, a local height scan, and the velocity command. Exteroception and the velocity command serve as conditional inputs that tell the policy which task to perform at each moment, without any explicit skill label. From the continuum of references produced by data augmentation, LightLP picks the iteration with the most demanding obstacle for each skill — furthest, tallest, closest to joint torque and angular-velocity limits — and distills this hardest expert directly onto the perceptive locomotion observation. Using the hardest expert lets the distilled policy inherit the full operating envelope rather than only the easy interior of the curriculum.

Distillation uses DAgger, collecting on-policy rollouts from the student and supervising them with the expert's actions at matching states. However, distilling on the student's own sampled states alone yields a brittle policy: once a rollout drifts off the expert's distribution, imitation offers no signal for recovery. LightLP therefore opens a PPO loss alongside the imitation term, so the policy retains the ability to recover even when tracking is imperfect. The final objective combines expert distillation with PPO:

$$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{DAgger}}+\lambda_{\text{RL}}\,\mathcal{L}_{\text{PPO}}(r)$$ $$r = w_{\text{task}}\,r_{\text{task}}+w_{\text{goal}}\,r_{\text{goal}}+w_{\text{mimic}}\,r_{\text{mimic}}$$

where the imitation term is:

$$\mathcal{L}_{\text{DAgger}}(\pi_{\theta}) = \mathbb{E}_{s\sim d^{\pi_{\theta}}}\left[\left\|\pi_{E}(s_{\text{priv}})-\pi_{\theta}(s^{\text{stu}})\right\|_{2}^{\,2}\right]$$

penalizing the gap between the expert action under the privileged observation $s_{\text{priv}}$ and the student action under the deployable observation $s^{\text{stu}}$. $\mathcal{L}_{\text{PPO}}$ is the standard clipped surrogate driven by the composite reward $r$, and $\lambda_{\text{RL}}$ trades off the two. The reward, domain randomization, and terminations are inherited from the Object-Interaction Mimic stage, with only the observation changing to the height scan and velocity command.

The reward terms include: a terminal base-pose goal reward $r_{\text{goal}} = -\lVert p_T - p^\star\rVert_2 - \alpha\lVert\theta_T - \theta^\star\rVert$, a joint-angle mimic reward $r_{\text{mimic}} = \exp(-\beta\lVert q_t - \hat{q}_t\rVert_2^2)$, and a reference-deviation termination $\lVert p_t - \hat{p}_t\rVert_2 > 25$ cm.

3.7 Skill Generalization

After distillation, the policy $\pi_i$ reproduces its skill only on the single obstacle its expert was trained on. The generalization stage turns it into one that handles the whole range of obstacles collected during data augmentation. Training continues across that range, but now without an expert to imitate. Dropping the expert and asking for generalization rather than reproduction drives changes to observation and reward. Each skill trains on its own terrain, but terrain parameters are randomized within the range covered by the augmented dataset, shifting the policy from "reproducing one configuration" to "adapting to geometric variation."

3.8 Transition Learning: Multi-Expert Distillation and Fine-Tuning

This is LightLP's third core innovation — letting skill transitions emerge from sparse reward, without one-hot skill labels, hand-coded state machines, or runtime motion generators.

The multi-expert distillation stage distills the perceptive locomotion teacher and each skill teacher into a single height-scan policy. DAgger collects on-policy rollouts supervised by the matching expert determined by the current terrain. This lets a single policy learn to walk in open terrain and execute whole-body skills at obstacles.

The transition fine-tuning stage introduces a reward-only transition group. The policy trains in scenes containing three regions — open terrain, transition zone, and skill zone — but the transition zone has no expert supervision. The policy must learn when and which skill to invoke from depth observation and velocity command alone. The AMP prior switches from locomotion to skill at the trigger position, helping maintain motion quality.

Figure 5: Transition group training. The robot is reset across open terrain, transition zone, and skill zone; the transition zone has no expert supervision.

t-SNE visualization reveals the internal representation of this mechanism: with the transition group, the RNN hidden state can leave the perceptive-locomotion region and enter the parkour-skill region; without it, the hidden state stays confined to the locomotion region, and the policy keeps walking or stepping in place instead of entering skill mode. This proves the recurrent policy learns to encode which mode to enter from the observed terrain and velocity command alone.

Figure 6: t-SNE visualization of RNN hidden states during transition courses, colored by behavior type: flat locomotion, transition, parkour.

3.9 From Height Scan to Onboard Depth

The final distillation ports the height-scan policy onto a recurrent depth policy using onboard-style depth. The student encoder processes depth images, fuses with proprioception and velocity command through an RNN, and decodes with an MLP into the action. The decoder is used only at training time and dropped at deployment.

Crucially, depth noise and latency modeling is essential. Distilling onto clean simulated depth would transfer poorly, because the hardware RealSense D435 returns far noisier and slower images. Each frame is corrupted by range-dependent Gaussian noise with standard deviation $0.005 + 0.02\,d$ (where $d$ is the measured depth in meters) and a global depth-scale jitter of $\pm 5\%$. Persistent block dropout is applied: small patches refreshed every few frames are set to the camera's maximum range to mimic the invalid regions the D435 produces, biased toward top and bottom rows. The camera's measured timing is also replayed: 30-60 ms processing latency at 27-33 Hz frame rate. A short fine-tune under this noise-and-latency model yields the deployed policy.

flowchart LR
    A["Seed motion
video/mocap"] --> B["Object-Interaction Mimic
global position tracking+RSI"]
    B --> C["Iterative Self-Augmentation
lift 5-10cm per iter"]
    C --> D["Hardest expert selection"]
    D --> E["Height-Scan Distillation
DAgger+PPO"]
    E --> F["Skill Generalization
across obstacle range"]
    F --> G["Multi-Expert Distillation
single policy"]
    G --> H["Transition Group Fine-Tune
no expert supervision"]
    H --> I["Depth Distillation
noise+latency model"]
    I --> J["Onboard Deployment
50Hz/Orin Nano"]

Figure 7: Complete LightLP training pipeline, nine steps from seed motion to onboard deployment.

4. Experimental Results

4.1 Platform: Lightbot 0

All experiments are conducted on Lightbot 0, a custom-built compact humanoid developed at Light Origins. The platform stands 90 cm tall, weighs 18.9 kg, and exposes 21 actuated degrees of freedom distributed across the legs, torso, and arms, with parallel four-bar ankle linkages on both legs. Joint actuation is supplied by quasi-direct-drive motors with two torque tiers: 45 N·m for the waist and lower-body joints and 15 N·m for the arm joints, with a uniform peak angular velocity of 9.42 rad/s. These limits are substantially below those of 1.3 m-class platforms used in concurrent humanoid parkour work; the limited actuation is precisely why the data augmentation pipeline cannot rely on direct mocap retargeting from full-size humans.

Figure 8: Lightbot 0, the custom-built 90 cm humanoid with 21 DoF, chest-mounted depth camera, and pelvis IMU.

The onboard sensor stack consists of a chest-mounted Intel RealSense D435 depth camera tilted 30° downward, a 6-axis IMU at the pelvis, and joint encoders. All policies run from this onboard stack alone, with no LiDAR, motion capture, or external state estimation. The deployed policy executes at 50 Hz on an onboard NVIDIA Jetson Orin Nano, a compact 7-25 W edge module rated at 67 INT8 TOPS — a far lighter compute budget than the dedicated units required by methods that run a motion generator at inference.

4.2 Real-World Experiments

LightLP is deployed zero-shot indoors and outdoors, tracking velocity commands from onboard depth and proprioception alone. At deployment, it needs no odometry, motion capture, or external mapping. Seven capabilities are demonstrated:

Skill	Key Challenge	Highlight
Reverse-vault	360° aerial rotation + pelvis contact redirect	Two consecutive rotations dissipate impact
Speed-vault	3.14 m/s peak velocity + transient single-arm support	Running approach + flight stabilization
Climb-and-step	Chest-height hand contacts + bilateral arm support	Succeeds on unseen 50 cm pommel horse
Plank bridge	Extremely narrow support surface	Continuous posture regulation
Stepping stones	Discrete footholds + limited field of view	Memory of prior terrain geometry
High platform	30 cm step ≈ 33% of standing height	Prolonged single-leg stance
Outdoor curbed stairs	Real-world depth noise + terrain uncertainty	Zero-shot stair ascent/descent

Table 1: Seven real-world deployment capabilities and their key challenges.

Figure 9: Real-world deployment on Lightbot 0, each run zero-shot from onboard depth and proprioception.

The speed-vault reaches a peak velocity of 3.14 m/s — a highly challenging dynamic performance for a 90 cm, 18.9 kg compact humanoid. In climb-and-step, replacing the training box with a 50 cm pommel-horse-like trapezoidal obstacle never seen in training, the same skill still succeeds on the sloped, non-box profile, demonstrating that the policy keys on perceived terrain geometry rather than memorizing a fixed obstacle shape. The high platform at 30 cm represents approximately 33% of the robot's standing height, requiring prolonged single-leg stance and a large range of motion.

4.3 Simulation and Benchmark

Reference generation comparison. Starting from the same seed motion, LightLP's references are compared against GMR and OmniRetarget. Key metrics include no penetration (the body must not pass through the obstacle). Direct GMR retargeting produces penetration because it is unaware of the obstacle; OmniRetarget optimizes at the reference level but cannot guarantee dynamic feasibility. LightLP's references, tracked under the physics engine, are both penetration-free and dynamically feasible.

Method	No Penetration	Dynamically Feasible	Scene-Paired
GMR	No	No	No
OmniRetarget	Partial	No	Yes
LightLP (ours)	Yes	Yes	Yes

Table 2: Reference generation method comparison. LightLP is the only method satisfying all three criteria.

Skill and locomotion benchmark. Compared against baseline methods on climb-and-step and speed-vault, LightLP maintains higher success at larger normalized obstacle heights, indicating stronger generalization across obstacle scales and pushing behavior closer to the robot's physical limits. This is attributed to the high-precision, dynamically feasible references combined with the final task-reward fine-tune.

For perceptive locomotion, LightLP is compared with BeamDojo (designed for sparse footholds like stepping stones) and CReF. Stairs are divided into low (15 cm), middle (25 cm), and high (35 cm) levels; CReF reaches only the middle level and reports lower success than the LightLP student policy. Stepping stones with height variation and balance beams with yawed approaches still maintain high success.

4.4 Transition Ablation

The transition ablation is one of the most convincing experiments, directly validating the necessity of the transition group:

Setting	Transition Success Rate (%)
Plane locomotion + skills, w/o transition group	0
Rough perceptive locomotion + skills, w/o transition group	33
w/o AMP	51
Ours (with transition group)	98

Table 3: Transition ablation and command-adherence test. The transition group raises success from 0% to 98%.

A policy trained only on flat locomotion and isolated whole-body skills without the transition group never learns how to connect them, achieving 0% success. Even when rough-terrain perceptive locomotion is added, removing the transition group leaves the policy without a learned composition strategy: in front of an obstacle, it often continues the locomotion behavior, collides with the wall, or repeatedly steps in place — only 33% success. With the transition group, the policy identifies from depth and velocity command which skill is needed and completes the obstacle traversal, reaching 98% success.

Removing the AMP prior drops success to 51% — the policy can still react to terrain cues, but the unconstrained objective produces unnatural motions and awkward contacts that are difficult to transfer to hardware. AMP keeps the policy close to the learned skill manifold, yielding both reliable transitions and physically plausible whole-body motion.

Command adherence test. To verify that the policy truly acts on the command rather than being drawn to any obstacle in view, a reverse-away command is issued in front of an obstacle. LightLP retreats as commanded, confirming that its behavior is decided jointly by the velocity command and the depth observation rather than by the mere presence of an obstacle — a known failure mode in reference-bound whole-body policies.

5. Limitations

Three limitations stated by the authors: First, the seed-collection step of the data pipeline still requires a human-in-the-loop to align each video with a candidate obstacle, and this manual step is a real ceiling on how fast the skill set can be expanded. Second, the transition policy currently couples a small, discrete set of skills, and degraded behavior is observed when obstacles overlap or appear in close succession — a problem the current sparse-reward setup does not fully resolve. Third, the fixed chest-mounted depth camera limits both whole-body manipulation of the scene and the perceptive coverage available to the locomotion controller, particularly when the upper body occludes the camera mid-skill.

Independent assessment: From a methodological perspective, the iterative self-augmentation's fixed 5-10 cm per-iteration lift may not be optimal — harder skills might require adaptive step sizes or difficulty-feedback-driven dynamic adjustment. Additionally, the nine-step pipeline involves multiple training stages and hyperparameters, making end-to-end tuning complex and reproducibility challenging. Finally, validation on only a single 90 cm platform means larger or smaller humanoids (with different torque-to-weight ratios) may require re-tuning reference generation and curriculum parameters.

6. Conclusion and Future Work

LightLP's central contribution is a single deployable policy that reads onboard depth and a velocity command, then selects and executes the appropriate behavior across both perceptive locomotion and whole-body parkour skills, without a reference input, skill label, hand-coded gate, or runtime motion graph. This capability is enabled by three components: high-precision reference-object pairs that align whole-body motion with the terrain contacts required for each skill, RL fine-tuning that turns each skill into a terrain-conditioned behavior, and transition-group training that lets switching between locomotion and skills emerge from sparse reward.

Deployed zero-shot on Lightbot 0, the policy traverses climb-and-step, reverse-vault, and speed-vault obstacles in both indoor and outdoor environments, showing that whole-body contact and perceptive locomotion can be unified within one learned controller. This marks a step beyond isolated terrain-specific behaviors toward a unified controller that can coordinate stepping, balancing, climbing, and vaulting within one learned policy.

Future directions include: scaling the pipeline to much larger reference sets and studying whether richer transition behavior emerges as data grows; addressing the perceptual bottleneck through active perception (camera or head gaze controlled by the policy itself) or by giving the policy an explicit short-term memory of the scene, so that mid-skill occlusion does not erase what the robot already saw on approach.

Golden Quote

"Let the policy decide when to walk and when to climb — not because we wrote the rules, but because it learned to unify perception and decision from reward."