Gallant: Voxel Grid-based Humanoid Locomotion and Local-navigation across 3D Constrained Terrains

Gallant: Voxel Grid-based Humanoid Locomotion and Local-navigation across 3D Constrained Terrains

Authors: Qingwei Ben, Botian Xu, Kailin Li, Feiyu Jia, Wentao Zhang, Jingping Wang, Jingbo Wang, Dahua Lin, Jiangmiao Pang

Affiliations: Shanghai AI Lab · CUHK · USTC · UTokyo · SJTU

Paper: https://arxiv.org/abs/2511.14625

---

One-sentence Summary

Gallant uses voxelized LiDAR data as a lightweight structured perceptual representation, paired with a z-grouped 2D CNN and end-to-end PPO policy, enabling a humanoid to traverse ground, lateral, and overhead 3D-constrained terrains, achieving near-100% success on stair climbing and elevated-platform scenarios for the first time.

Background and Motivation

Robust humanoid locomotion requires accurate, globally consistent perception of the surrounding 3D environment. Existing perception modules, mainly based on depth images or elevation maps, offer only partial, locally flattened views, failing to capture full 3D structure. Depth cameras offer lower latency but narrow FoV and limited range, impeding reasoning about spatially extended environments. 3D LiDAR provides detailed geometry with wide FoV, but raw point clouds are sparse and noisy, bottlenecking sample-efficient learning and real-time inference.

Elevation-mapping approaches compress 3D point clouds into 2.5D height fields, providing a bird's-eye height estimate per ground cell. But this projection discards vertical and multi-layer structure (overhangs, low ceilings, mezzanines, stair undersides), and the reconstruction stage can introduce algorithm-specific distortions and latency, further decoupling perception from control. When a robot must traverse lateral clutter, overhead constraints, or narrow passages, 2.5D elevation maps fundamentally cannot represent these 3D structures.

Gallant addresses these limitations with a voxel-grid-based perception-learning framework. It uses a robot-centric voxel grid derived from LiDAR point clouds, preserving multi-layer scene structure over a large FoV while aggregating raw points into voxels to reduce dimensionality and smooth noise. A z-grouped 2D CNN treats height slices as channels, exploiting sparsity for compact features with a favorable accuracy-compute trade-off over heavier 3D CNNs.

Core Contributions

1. Voxel grid perception: Converts LiDAR point clouds into a robot-centric binary occupancy tensor $X \in \{0,1\}^{C \times H \times W}$ ($C=40$ height slices, $H=W=32$), preserving multi-layer 3D structure while reducing dimensionality.
2. Z-grouped 2D CNN: Treats the z-axis as the channel dimension, applying 2D convolutions over the x-y plane. Reduces compute and memory by roughly a factor of $k$ versus 3D kernels while capturing vertical patterns.
3. Efficient LiDAR simulation: A lightweight raycast-voxelization pipeline using NVIDIA Warp, precomputing per-mesh local-frame BVHs and transforming only rays during simulation, enabling scalable parallel LiDAR simulation in dynamic environments.
4. Full-space obstacle traversal: A single policy extends beyond ground-level obstacles to lateral clutter, overhead constraints, multi-level structures, and narrow passages.

Method Details

Problem Formulation

Humanoid perceptive locomotion is formulated as a POMDP $\mathcal{M}=(\mathcal{S}, \mathcal{A}, \mathcal{O}, P, \mathcal{R}, \Omega, \gamma)$, trained with PPO. The environment is divided into 8 m × 8 m blocks; each episode starts at the center with a goal $\mathbf{G}$ sampled on the perimeter, 10 s horizon. Observations include command (goal position, elapsed/remaining time), action history, proprioception (angular velocity, projected gravity, joint positions/velocities), perception (voxel grid), and privileged inputs (linear velocity, height map, critic-only). The reward uses goal-reaching:

$$r_{\text{reach}} = \frac{1}{1+\|\mathbf{P}_t\|^2} \cdot \frac{\mathds{1}(t > T - T_r)}{T_r} \quad (T_r = 2s)$$

The objective maximizes $J(\pi) = \mathbb{E}[\sum_{t=0}^{H-1} \gamma^t r_t]$. Episodes end on fall, harsh collision, or timeout.

Efficient LiDAR Simulation

Most GPU simulators lack native efficient LiDAR simulation or are limited to a single static mesh. Gallant implements a lightweight raycast-voxelization pipeline using NVIDIA Warp. Traditional raycasting builds a BVH over scene geometry, costly to update at every step. This method precomputes a BVH for each mesh in its local frame; during simulation the ray origin $\mathbf{p}$ is transformed and only the rotation is applied to direction $\mathbf{d}$:

$$\text{raycast}(TM, \mathbf{p}, \mathbf{d}) = T^{-1}\text{raycast}(M, T^{-1}\mathbf{p}, R^{-1}\mathbf{d})$$

Each step computes ray-mesh intersections for every mesh $M$ with transform $T_t$, parallelized via a Warp kernel of shape $(N_{\text{envs}}, N_{\text{meshes}}, N_{\text{rays}})$. Domain randomization includes LiDAR pose perturbation, hit-position noise, 10 Hz update with 100-200 ms delay, and 2% random voxel dropout.

Voxel Representation and 2D CNN Perception

LiDAR point clouds are converted into a fixed-size robot-centric voxel grid. Returns from two torso-mounted LiDARs are transformed into a unified torso frame. The perception volume is $\Omega = [-0.8, 0.8]\text{m} \times [-0.8, 0.8]\text{m} \times [-1.0, 1.0]\text{m}$, resolution $\Delta = 0.05\text{m}$, yielding a $32 \times 32 \times 40$ grid. Each voxel is 1 if at least one point lies inside, 0 otherwise, producing $X \in \{0,1\}^{C \times H \times W}$.

Due to LiDAR's line-of-sight nature and terrain structure, the voxel grid is highly sparse: most $(x,y)$ columns contain only 1-2 occupied z-slices. The method treats z as the channel dimension and applies 2D convolutions over x-y:

$$Y_{o,v,u} = \sigma\left(\sum_{c=0}^{C-1}\sum_{\Delta v, \Delta u} \mathbf{W}_{o,c,\Delta v, \Delta u} \cdot X_{c,v+\Delta v, u+\Delta u} + b_o\right)$$

This reduces compute and memory by roughly a factor of $k$ versus a 3D kernel of size $k^3$, while capturing vertical patterns critical for locomotion and supporting real-time onboard inference.

Figure 1: Method overview. (a) Curriculum training over 8 terrains; (b) efficient LiDAR simulation; (c) 2D CNN on voxel grid; (d) latent-aware PPO for zero-shot sim-to-real.

Terrain Design

Eight representative terrain types: Plane (basic walking), Ceiling (overhead constraints, crouching), Forest (sparse lateral clutter, weaving), Door (narrow gaps, lateral clearance), Platform (stepable surfaces, inter-platform traversal), Pile (fine-grained support reasoning), Upstair/Downstair (vertical elevation). Curriculum difficulty increases progressively: $\mathbf{p}_\tau(s) = (1-s)\mathbf{p}_\tau^{\min} + s\mathbf{p}_\tau^{\max}$, with promotion on success and demotion on failure.

Terrain	Parameter	Easiest	Hardest
Ceiling	Height (m) ↓	1.30	1.00
Forest	Min tree distance (m) ↓	2.0	1.0
Door	Wall distance (m) ↓	2.00	1.00
Platform	Height (m) ↑	0.05	0.35
Pile	Cylinder distance (m) ↑	0.35	0.45
Upstair	Step height (m) ↑	0.00	0.20

Method Overview

flowchart TD
    A[Two torso LiDAR point clouds] --> B[Transform to unified torso frame]
    B --> C[Voxelization 32x32x40 binary occupancy]
    C --> D["Z-grouped 2D CNN z-axis as channels"]
    D --> E[Perception features]
    F[Proprioception history] --> G[PPO Actor-Critic policy]
    E --> G
    H[Goal position command] --> G
    G --> I[Joint actions]
    I --> J[Humanoid motion]
    subgraph Sim Training
        K[Efficient LiDAR sim Warp] --> C
        L[Domain rand noise/latency/dropout] --> C
        M[Curriculum 8 terrains] --> G
    end

Experimental Results

Simulation Experiments

Gallant achieves near-100% success across 8 terrains. Key ablations cover perception network (Sparse-3D-CNN, 3D-CNN, Sparse-2D-CNN, Gallant), perceptual interface (height-map only, voxel only, Gallant), and voxel resolution (10 cm, 2.5 cm, 5 cm).

Ablation	Variant	Key result
Network	Sparse-3D-CNN	Ceiling 86.7%, Door collisions 143.5
Network	3D-CNN	Ceiling 97.5% but Door collisions 69.6
Network	Gallant (z-grouped 2D CNN)	Ceiling 97.1%, Door collisions only 27.7
Interface	Height-map only	Ceiling only 5.3% — cannot represent overhead constraints
Interface	Voxel only	Ceiling 96.9% but Door collisions 281.4
Interface	Gallant (voxel + height map)	Balanced optimal across all terrains
Resolution	10 cm	Insufficient precision, Ceiling 75.5%
Resolution	2.5 cm	Ceiling drops to 13.3%, over-fine increases sparsity
Resolution	5 cm (Gallant)	Optimal balance, Ceiling 97.1%

The height-map-only variant's 5.3% Ceiling success proves 2.5D elevation fundamentally cannot represent overhead constraints — the core motivation for voxels. The voxel-only variant's 281.4 Door collisions (vs Gallant's 27.7) show height maps provide valuable fine ground detail for precise lateral clearance. The 5 cm resolution is the Pareto optimum: 10 cm loses fine structure, 2.5 cm increases sparsity and reduces performance.

Figure 2: The 8 terrain types used in simulation training (hardest settings).

Real-World Experiments

Gallant deploys zero-shot to real humanoid hardware, traversing diverse obstacles including ground, lateral, and overhead challenges. Improved end-to-end optimization enables near-100% success on stair climbing and elevated-platform scenarios for the first time. The full-stack pipeline — from data generation to perception to control — is foundational to full-space traversal.

Figure 3: Real-world deployment traversing diverse 3D-constrained terrains.

POMDP定义

$$ \mathcal{M}=(\mathcal{S},\mathcal{A},\mathcal{O},P,\mathcal{R},\Omega,\gamma) $$

期望回报

$$ J(\pi)=\mathbb{E}[\sum_{t=0}^{H-1}\gamma^{t}r_{t}] $$

Limitations

1. Fixed voxel resolution: 5 cm is a precision-sparsity trade-off; for small obstacles or thin structures it may be insufficient. Adaptive or multi-scale voxels could help but add complexity.
2. Limited perception range: The volume $\Omega$ covers only 1.6 m × 1.6 m around the robot. Long-range path planning or large-scale navigation needs a higher-level planner.
3. LiDAR dependency: The framework relies on two torso LiDARs, with hardware cost and power implications. Domain randomization narrows the sim-to-real gap, but specific real LiDAR noise patterns (rain/fog, specular reflection) may not be fully modeled.

Conclusion and Outlook

Gallant is a voxel-grid-based perception-learning framework for humanoid locomotion and local navigation in 3D-constrained terrains. It leverages voxelized LiDAR as a lightweight structured representation, a z-grouped 2D CNN mapping to control, and fully end-to-end optimization. Efficient LiDAR simulation supports scalable training and sim-to-real consistency. Experiments show broader perceptual coverage lets a single policy extend beyond ground-level obstacles to lateral clutter, overhead constraints, multi-level structures, and narrow passages, achieving near-100% success on stairs and elevated platforms for the first time.

The core insight is that 2.5D elevation maps fundamentally cannot represent 3D vertical and multi-layer structure, while full 3D CNNs are too costly. The z-grouped 2D CNN finds an elegant balance — channel mixing for vertical structure, 2D convolution for spatial context, at 1/k the cost of 3D kernels. The complementarity of voxel grids (overhead/lateral) and height maps (fine ground detail) is key to full-space traversal.

Golden Quote

When the map is no longer just the height beneath the feet but the full 3D occupancy above, beside, and ahead, the robot finally "sees" the entire space like a human — not just the ground.