PAPER DEEP DIVE
RAVEN: Reinforcement-Adaptive Visibility-Graph Planning for Robust Humanoid Navigation with Collision-Free MPC
Humanoid navigation in dynamic environments requires long-horizon planning while respecting short-horizon dynamic and safety constraints. Classical visibility-graph planners combined with model predictive control (MPC) can efficiently generate collision-free trajectories, but their performance depends on manually tuned parameters and accurate system modeling. In real robotic systems, control delays, state-estimation noise, and locomotion uncertainties can cause overshoot and constraint violations even when the nominal path is geometrically optimal. We propose RAVEN, a hierarchical reinforcement learning (RL)-MPC framework for robust humanoid navigation. Unlike prior approaches that use learning to tune cost weights or replace planning entirely, RAVEN employs RL to adapt the geometric construction of a visibility-graph planner by modifying obstacle inflation and related graph parameters. By directly reshaping the free-space geometry, the learned planner alters the topology of the global path to compensate for delay and tracking imperfections. A collision-free MPC layer then tracks the planned trajectory while explicitly enforcing velocity bounds and obstacle-avoidance constraints. By training under realistic delays and observation noise, RAVEN learns planning adaptations that improve robustness while retaining explicit long-horizon geometric planning and constrained optimization, in contrast to end-to-end learning approaches. We evaluate RAVEN against a manually tuned visibility-graph MPC baseline and a pure RL navigation policy. Results demonstrate reduced overshoot near obstacles, improved robustness in narrow passages, and more reliable navigation under delay and noise. These findings indicate that reinforcement-adaptive graph construction combined with constrained MPC provides an effective and interpretable alternative to end-to-end learning for robust humanoid navigation.
RAVEN: Reinforcement-Adaptive Visibility-Graph Planning for Robust Humanoid Navigation with Collision-Free MPC
Institution: UCLA RoMeLa | arXiv: 2607.15701v1
One-Sentence Summary
RAVEN is a hierarchical RL-MPC framework where an RL meta-policy adaptively adjusts obstacle inflation parameters of a visibility-graph planner to reshape free-space geometry, combined with collision-free MPC for trajectory tracking, achieving robustness to control delay and observation noise in humanoid navigation.
Background and Motivation
Humanoid navigation requires long-horizon planning with short-horizon dynamic and safety constraints. Classical visibility-graph + MPC pipelines generate collision-free trajectories but depend on manually tuned parameters and accurate modeling. Real robots face control delays, state-estimation noise, and locomotion uncertainty causing overshoot and constraint violations. End-to-end RL lacks structural priors, tending toward overly conservative reactive policies. RAVEN's key idea: RL adjusts planner geometric parameters (obstacle inflation) rather than directly controlling velocity.
Figure 1: RAVEN hierarchical framework — RL meta-policy adjusts visibility graph parameters, MPC tracks planned path.
Method
DAVG-cfMPC Foundation
DAVG computes kinematically feasible paths by discretizing environment into visibility graph. cf-MPC formulates collision avoidance as linearized soft constraint in convex QP. For obstacle $j$:
$$[\boldsymbol{x}_k - \boldsymbol{x}_{obs,j},\, \boldsymbol{y}_k - \boldsymbol{y}_{obs,j}] \cdot \boldsymbol{V}_{k,j} \geq \|\boldsymbol{V}_{k,j}\| (R_{obs,j} - \delta_j)$$where $\boldsymbol{V}_{k,j}$ is relative position, $\delta_j \geq 0$ is slack variable, $R_{obs,j}$ is inflation radius.
RL Meta-Policy
Outputs planning parameters (obstacle inflation values) rather than motion commands. Increasing inflation creates safer paths with larger clearance; decreasing allows aggressive passage through narrow gaps. Robot pose encoded as $[x, y, \sin\psi, \cos\psi]$ to avoid angular wrap-around. Asymmetric Actor-Critic: Actor receives noisy/delayed observations, Critic receives privileged (noisy + clean) observations.
Figure 2: RL meta-policy — observes environment state, outputs obstacle inflation parameters.
Reward Function
$$R = w_{\text{time}} + w_{\text{path}} \|\boldsymbol{p}_t - \boldsymbol{p}_{t-1}\|_2 + w_{\text{col}} \sum_i \mathbf{1}\{\|\boldsymbol{p}_t - \boldsymbol{o}_i\|_2 < r_{\text{pen}}\} + w_{\text{inside}} \sum_i \max(0, r_{\text{pen}} - \|\boldsymbol{p}_t - \boldsymbol{o}_i\|_2)$$ $$+ w_{\text{ar}} \sum_i (r_{t,i} - r_{t-1,i})^2 + R_{\text{succ}} \cdot \mathbf{1}\{\text{success}\} + R_{\text{fall}} \cdot \mathbf{1}\{\text{fall}\}$$Weights: $w_{\text{time}}=-5.0$, $w_{\text{path}}=-20.0$, $w_{\text{col}}=-8.0$, $w_{\text{inside}}=-80.0$, $w_{\text{ar}}=-0.5$, $R_{\text{succ}}=5000$, $R_{\text{fall}}=-50000$.
Meta-policy optimization objective:
$$\pi^* = \arg\max_\pi \mathbb{E}\left[\sum_t \gamma^t R(s_t, a_t, s_{t+1})\right], \quad a_t = \{r_{t,1}, \ldots, r_{t,K}\}$$
Figure 3: How adaptive obstacle inflation changes visibility graph and path topology.
graph TD
A["RL Meta-Policy
Observe: robot state + goal + obstacles"] --> B["Output: obstacle inflation params"]
B --> C["DAVG Visibility Graph
Adaptive inflation radius"]
C --> D["Shortest Path Computation"]
D --> E["cf-MPC Tracking
QP collision constraints"]
E --> F["Robot Execution
With delay and noise"]
F --> A
Results
Simulation: Delay Robustness
| 0s Delay | MPC | RL | RAVEN |
|---|---|---|---|
| Path Length (m) | 9.838 | 9.645 | 9.23 |
| Completion Time (s) | 11.43 | 11.55 | 11.28 |
| Max Penetration (m) | 0.049 | 0.016 | 0.019 |
| 0.06s Delay | MPC | RL | RAVEN |
|---|---|---|---|
| Path Length (m) | 11.25 | — | 9.33 |
| Completion Time (s) | — | 12.21 | 11.58 |
| Max Penetration (m) | 0.128 | 0.0 | 0.03 |
Under delay: MPC degrades severely (penetration 0.128m, path 11.25m); pure RL is safe but inefficient (12.21s); RAVEN balances both (0.03m penetration, 9.33m path, 11.58s time).
From control theory, RAVEN's design is a feedforward-feedback separation. MPC is pure feedback; delay $\tau$ causes overshoot. Meta-policy acts as feedforward compensator. Delay-induced position deviation:
$$\Delta \boldsymbol{p}(t) \approx \boldsymbol{v}(t) \cdot \tau + \frac{1}{2} \boldsymbol{a}(t) \cdot \tau^2$$Adaptive inflation must satisfy: $R_{obs}^{\text{adaptive}} \geq R_{obs}^{\text{nominal}} + \|\Delta \boldsymbol{p}\| + \epsilon_{\text{noise}}$
Figure 4: RAVEN adaptive path — increases inflation near obstacles for wider turns, decreases after narrow passage.
Hardware Experiments
Validated sim-to-real transfer on real humanoid robot. RAVEN more reliable in narrow passages and delayed control. Meta-policy dynamically increases obstacle inflation by observing delayed state, guiding MPC along more conservative trajectories.
Figure 5: Hardware experiments — RAVEN navigation deployment on real humanoid.
The key distinction from end-to-end RL is inductive bias. RAVEN preserves visibility graph's global optimal path structure, restricting RL to low-dimensional parameter space (obstacle count $K$), improving sample efficiency and interpretability. Topology-level adaptation — when inflation increases, certain graph edges become invisible, changing shortest path topology from narrow passage to detour. This is more effective than simple velocity adjustment.
Limitations
- Limited to static environments; dynamic obstacles (moving objects, pedestrians) need extension
- Simplified locomotion model; more complex humanoid models may require adjustment
- Meta-policy only adjusts obstacle inflation; other planning parameters unexplored
- Sim-to-real gap remains, especially for complex terrain and perception noise
Conclusion
RAVEN integrates RL with classical planning by allowing a meta-policy to adapt visibility-graph construction parameters while cf-MPC tracks trajectories under explicit constraints. Preserves interpretability and safety of optimization-based control while learning robustness to delay and noise. Outperforms static MPC (brittle) and pure RL (overly conservative) in both efficiency (shortest path 9.33m) and safety (0.03m penetration). Future work: dynamic environments and advanced locomotion models.
Key Insight: Rather than letting RL directly control the robot, let it adjust the planner's geometric construction — preserving classical planning's interpretability and safety guarantees while gaining learned adaptation to uncertainties.
SOURCE LINKS
