X-NavDP: Generalizing Navigation Diffusion Policy to Novel Behavior and Embodiments with Group Q-score Reweighted Matching

Background and Motivation

Visual navigation is a core capability for autonomous mobile robots. In recent years, the paradigm of pretraining diffusion policies on large-scale simulation datasets has matured—methods like NavDP use oracle global planners to generate expert trajectories and train diffusion models via imitation learning to produce navigation actions. This approach offers excellent data scalability and strong zero-shot generalization across diverse indoor/outdoor scenes and heterogeneous robot embodiments.

The fundamental contradiction, however, lies in the training-deployment gap: oracle global trajectories used during training are unavailable at deployment. The oracle planner possesses complete global map information, and the generated expert trajectories are essentially "God's eye view" optimal paths. Physical robots only have onboard local sensors (RGB-D cameras) and face constraints such as limited perception range and occluded obstacles. This domain gap makes pretrained policies fragile in complex scenarios.

Specifically, the limitations of pretrained policies manifest at three levels. First, expert trajectories are generated by an oracle suited to a single nominal robot—different embodiments (differential-drive wheeled, quadruped gait, humanoid walking) have vastly different kinematic constraints, and nominal trajectories fail to generalize. Second, expert trajectories only contain "forward-to-goal" successful paths, lacking recovery behaviors such as backing out of traps, lateral avoidance, and detouring around long obstacles—precisely what is most needed in real deployment. Third, the likelihood function of diffusion policies is intractable, making standard policy gradient methods (e.g., DPPO) unstable when backpropagating through the long reverse chain.

RL post-training is a natural remedy—online interaction with simulation environments can provide real experience under local observations. But prior work shows limited gains: DPPO produces chaotic trajectories that degrade performance, DSRL nearly collapses (optimizing only initial noise is insufficient for learning effective navigation behaviors), and while DPMD outperforms both, it becomes uninformative in hard states where all candidate trajectories share similarly low absolute Q-values. Cross-state normalization in the minibatch causes easy states with higher absolute returns to dominate the actor update, failing to provide meaningful gradients for learning recovery actions.

X-NavDP addresses exactly this bottleneck: how to achieve efficient exploration and stable policy iteration in RL post-training of diffusion policies. The answer consists of two complementary designs—self-bootstrapped perturbation exploration and group Q-score reweighted matching (GQRM)—plus embodiment-conditioned modulation for large-scale heterogeneous training.

Preliminaries

Diffusion Policy for Navigation: Given observation $\mathbf{o}_t$ (RGB-D images) and goal $\mathbf{g}$, the policy generates an action chunk $a^0=\{(x_i,y_i)\}_{i=1}^{H}$ containing $H$ future waypoints through iterative denoising. The forward diffusion process adds Gaussian noise over $K$ steps:

$$q(a^k|a^{k-1})=\mathcal{N}(a^k;\sqrt{1-\beta_k}\,a^{k-1},\beta_k\mathbf{I})$$

where $\beta_k$ is the noise schedule. The reverse process learns a noise prediction network $\boldsymbol{\epsilon}_\phi(a^k,k,\mathbf{o}_t,\mathbf{g})$ with weighted MSE loss:

$$\mathcal{L}_{\text{dp}}=\mathbb{E}_{k,\boldsymbol{\epsilon}}\left[w(k)\|\boldsymbol{\epsilon}-\boldsymbol{\epsilon}_\phi(a^k,k,\mathbf{o}_t,\mathbf{g})\|^2\right]$$

This formula means: at a randomly selected diffusion step $k$, the noise prediction network should accurately recover the added Gaussian noise $\boldsymbol{\epsilon}$. Here $w(k)$ is a timestep-dependent weight. At inference, action chunks are generated by iteratively denoising from pure noise $a^K\sim\mathcal{N}(\mathbf{0},\mathbf{I})$.

Policy Mirror Descent and DPMD: Policy mirror descent (PMD) regularizes each policy update through a KL-divergence proximal term. The improvement target is:

$$\pi_{\text{MD}}(\cdot|s)=\arg\max_{\pi(\cdot|s)}\;\mathbb{E}_{a\sim\pi(\cdot|s)}\left[Q^{\pi_{\text{old}}}(s,a)\right]-\lambda D_{\text{KL}}\!\left(\pi(\cdot|s)\,\|\,\pi_{\text{old}}(\cdot|s)\right)$$

This has the closed-form solution $\pi_{\text{MD}}(a|s)=\frac{\pi_{\text{old}}(a|s)\exp(Q^{\pi_{\text{old}}}(s,a)/\lambda)}{Z_{\text{MD}}(s)}$, where $Z_{\text{MD}}(s)$ is the partition function. When the policy is parameterized by a diffusion model, directly sampling from this target is intractable. DPMD bypasses this via reweighted score matching: the exponential Q-value term weights the score matching objective, assigning larger regression weights to higher-value actions and enabling policy improvement in score-function space.

Method

Figure 2: Overview of the X-NavDP pipeline. (a) NavDP backbone with FiLM-injected embodiment information. (b) Structured trajectory perturbation mixing goal-conditioned and goal-agnostic samples. (c) Within-group Q-score normalization for adaptive reweighted score matching. (d) Large-scale cross-embodiment RL training.

1. RL Post-Training Framework

X-NavDP is built on IsaacLab, supporting over 500 wheeled, quadruped, and humanoid robots interacting simultaneously across 50+ scenes. All rollouts are aggregated into a shared buffer for joint cross-embodiment policy optimization. Since the navigation policy outputs waypoint chunks rather than joint commands, the system employs a hierarchical control stack: the diffusion policy predicts short-horizon target trajectories, a unified MPC converts them into base velocity commands, and an embodiment-specific pretrained locomotion policy executes them. The MPC and locomotion controllers run at 25 Hz, while the navigation policy is invoked chunk-wise: each predicted trajectory is tracked for 3 seconds, and the cumulative reward over this window is stored as one macro-step transition. This design reduces inference overhead and aligns value learning with trajectory-level actions.

2. Self-Bootstrapped Perturbation

A pretrained goal-conditioned diffusion policy carries a strong navigation prior, so its goal-conditioned samples concentrate on a narrow set of forward-moving trajectories toward the goal. Online exploration is therefore overly conservative, especially in hard states requiring backing out, lateral motion, or short detours. The resulting rollouts rarely contain the recovery behaviors that RL needs to reinforce. Directly injecting Gaussian noise is simple but unstructured noise seldom induces meaningful navigation behavior and easily destroys temporal smoothness and dynamic feasibility.

X-NavDP's key insight: the same pretrained navigation policy, when conditioned on a goal-agnostic condition, produces scene-consistent trajectories that explore more aggressively than its goal-conditioned counterpart. The internal goal-agnostic branch is therefore used to perturb goal-conditioned trajectories, increasing behavioral diversity while preventing excessive deviation from the pretrained trajectory manifold.

For each observation, a goal-conditioned trajectory $\tilde{\boldsymbol{\tau}}_{\text{pointgoal}}$ and a goal-agnostic trajectory $\tilde{\boldsymbol{\tau}}_{\text{nogoal}}$ are drawn from the same visual context, and the mixed trajectory is constructed as:

$$\boldsymbol{\tau}_{\text{mixed}}=\mathbf{s}\odot\bigl(\tilde{\boldsymbol{\tau}}_{\text{pointgoal}}+\lambda\,\tilde{\boldsymbol{\tau}}_{\text{nogoal}}\bigr),\quad\mathbf{s}=\bigl((-1)^{B_{1}},(-1)^{B_{2}}\bigr)$$

$$B_{1},B_{2}\sim\operatorname{Bernoulli}(\epsilon),\quad B_{1}\perp B_{2}$$

where $\lambda$ is a signed mixing coefficient, $\odot$ is the Hadamard product, and $\mathbf{s}$ independently flips the $x$ and $y$ coordinates of the entire chunk with probability $\epsilon$. This enables the policy to produce lateral, backward-recovery, and detour trajectories while preserving the smooth, dynamically feasible structure of the original waypoints. In the codebase (generate_action_mix method, x_navdp_policy.py:501), additional mechanisms include trajectory length ratio scaling, line trajectory mixing, and directional flip probability control, further enriching candidate behavior diversity.

3. GQRM: Group Q-score Reweighted Matching

GQRM is X-NavDP's core innovation, turning the self-bootstrapped candidate group into a stable diffusion actor update. The original Q-weighted variant in DPMD becomes uninformative in hard states where all sampled candidates share similarly low absolute Q-values. DPMD normalizes Q-scores and computes exponential weights across a minibatch, mixing candidates from unrelated states—easy states with higher absolute returns dominate the actor update, failing to provide meaningful gradients for learning recovery actions in hard states.

GQRM normalizes values within each same-state candidate group drawn from $\pi_{\text{old}}(\cdot|s)$, rather than across unrelated states in a minibatch. For a denoised action $a_0\sim\pi_{\text{old}}(\cdot|s)$, the within-group statistics are:

$$\bar{Q}_{G}(s)=\mathbb{E}_{a_0\sim\pi_{\text{old}}(\cdot|s)}\!\bigl[Q(s,a_0)\bigr],\qquad\sigma_{G}(s)=\sqrt{\mathbb{E}_{a_0\sim\pi_{\text{old}}(\cdot|s)}\!\bigl[(Q(s,a_0)-\bar{Q}_{G}(s))^{2}\bigr]}$$

The group-normalized value used in the actor weight is:

$$\widetilde{Q}_{G}(s,a_0)=\mathrm{clip}\!\left(\frac{c\,\bigl(Q(s,a_0)-\bar{Q}_{G}(s)\bigr)}{\sigma_{G}(s)+\varepsilon},\;-h,\;h\right)$$

where $c$ scales the normalized advantage, $h$ caps its magnitude, and $\varepsilon>0$ guards against degenerate groups. The GQRM objective is:

$$\mathcal{L}_{\text{GQRM}}(\theta;\,s,t)=\mathbb{E}_{\substack{a_0\sim\pi_{\text{old}}(\cdot|s)\\a_t\sim q_{t|0}(\cdot|a_0)}}\!\left[\exp\!\bigl(\widetilde{Q}_{G}(s,a_0)/\lambda\bigr)\;\bigl\|s_{\theta}(a_t;\,s,t)-\nabla_{a_t}\log q_{t|0}(a_t|a_0)\bigr\|^{2}\right]$$

This formula states: within the score matching objective, the group-normalized Q-value exponent serves as the weight. Even in hard states with low absolute returns, if a candidate has positive relative advantage, it receives a weight greater than 1, shifting probability mass toward it. In practice, only top-k candidates with positive advantage $\widetilde{Q}_{G}(s,a_0)>0$ are retained, suppressing low-value sample noise and reducing computation. In the code (gqrm_trainer.py:148-158), the implementation uses num_samples=64, topk=5, norm_scale=2: 64 candidate trajectories are evaluated by Q-value, then the top-5 positive-advantage candidates are selected for reweighted score matching.

"Why same-state grouping matters" is discussed in Appendix A.2. Absolute Q-values differ greatly across states (open corridor vs. dead end), and cross-state normalization allocates most of the gradient budget to easy states. Same-state grouping ensures the normalization baseline is the candidate mean for that specific state, enabling the actor to distinguish "relatively better" candidates even in hard states and achieve meaningful policy improvement.

4. Cross-Embodiment Training and Closed-Loop Execution

Architecturally, X-NavDP modifies the NavDP model by adding an embodiment-modulated module. Given an embodiment index $e$, a learned robot embedding $\mathbf{z}_e=E_{\text{emb}}(e)$ is injected at two complementary points in the diffusion decoder. First, before decoding, it shifts action-token features:

$$\mathbf{u}_e=\mathbf{u}+f_{\Delta}(\mathbf{z}_e)$$

Then, after decoding, FiLM modulation generates trajectory features:

$$[\Delta\boldsymbol{\gamma}_e,\Delta\boldsymbol{\beta}_e]=f_{\text{FiLM}}(\mathbf{z}_e),\qquad\mathbf{h}_e=(1+\Delta\boldsymbol{\gamma}_e)\odot\mathbf{h}+\Delta\boldsymbol{\beta}_e,\qquad\hat{\boldsymbol{\epsilon}}_\phi=H_{\text{act}}(\mathbf{h}_e)$$

The embedding modulates both the pre-decoder action-token pathway and the post-decoder trajectory-feature pathway, allowing a single shared noise predictor to generate embodiment-specific trajectories without replicating policy weights across robots. In the code (x_navdp_policy.py:45), nn.Embedding(3, token_dim) corresponds to three embodiments (Dingo wheeled=0, Unitree G1 humanoid=1, Unitree Go2 quadruped=2).

Closed-Loop Temporal Guidance: During deployment, a lightweight temporal-consistency term inspired by Real-Time Chunking (RTC) reduces discontinuities between consecutive predictions. The guidance gradient is:

$$\mathbf{g}=\nabla_{\mathbf{x}_t^{(k)}}\bigl\langle\operatorname{sg}\!\bigl(\mathbf{w}_t\odot(\mathbf{y}_t-\hat{\mathbf{x}}_t^{(0,k)})\bigr),\;\hat{\mathbf{x}}_t^{(0,k)}\bigr\rangle$$

The guided reverse step becomes:

$$\mathbf{x}_t^{(k-1)}=\boldsymbol{\mu}_k+\sigma_k\,\mathbf{z}+\sqrt{\bar{\alpha}_k}\;\eta_{\text{guide}}\;\mathbf{g}$$

where $\mathbf{y}_t$ is the previously committed trajectory transformed into the current robot frame, $\mathbf{w}_t$ is a vector of decaying prefix weights, and $\operatorname{sg}$ stops gradient flow through the residual term. This encourages predictions to move closer to the previously committed trajectory without overriding the learned diffusion prior.

flowchart TB
    A[RGB-D Obs + Goal] --> B[Diffusion Policy NavDP Backbone]
    B -->|Goal-conditioned| C[pointgoal trajectory]
    B -->|Goal-agnostic| D[nogoal trajectory]
    C --> E[Mix + Direction Flip]
    D --> E
    E --> F[64 Candidate Trajectories]
    F --> G[Twin-Q Evaluation]
    G --> H[Within-Group Q Normalization]
    H --> I[Exp Weighting + top-k Selection]
    I --> J[Reweighted Score Matching]
    J --> K[Update Diffusion Actor]
    K --> B
    L[Embodiment Embedding FiLM] --> B

Experimental Results

Simulation Main Results

The simulation benchmark is built in IsaacLab using scenes from GRScenes-100. RL training uses 56 scenes (47 home + 9 commercial), and evaluation is conducted on 40 unseen scenes (20 home + 20 commercial). Three embodiments are evaluated: Dingo (differential-drive wheeled), Unitree Go2 (quadruped), and Unitree G1 (humanoid). Metrics are Success Rate (SR) and Success weighted by Path Length (SPL).

Table 1: Point-goal navigation results in IsaacLab across 40 indoor scenes. Each cell reports SR/SPL.

X-NavDP consistently outperforms NavDP across all embodiments and scenes, improving overall SR from 61.20% to 84.28% and SPL from 58.95% to 77.19%. The humanoid robot improvement is particularly striking: the pretrained policy achieves only 64.35%/50.70% in commercial/home scenes, and X-NavDP raises this to 84.25%/84.50%—the humanoid was NavDP's weakest embodiment, yet after RL post-training it becomes one of the strongest. This demonstrates that cross-embodiment training is not simple "averaging" but targeted improvement of each embodiment's weaknesses through structured exploration and within-group normalization.

Real-World Deployment

Table 2: Real-world point-goal deployment. Policy weights trained in simulation, deployed without sim-to-real fine-tuning. 10 trials per setting.

The real-world results are even more telling: NavDP nearly completely fails across all embodiments (max 40% success rate), while X-NavDP achieves 50%-80% success rates across all 9 settings. The average success rate jumps from 10% to 65%. Policy weights transfer directly from simulation to real robots without additional sim-to-real fine-tuning, thanks to RL post-training teaching the policy to perform recovery behaviors under local observations—precisely the failure modes caused by missing oracle trajectories in real deployment.

Figure 3: Qualitative trajectory comparison. X-NavDP produces recovery, detour, and safer behaviors absent from the NavDP policy. Visualization by VGGT-Ω.

RL Method Comparison Ablation

On a 24-scene subset, different RL fine-tuning methods are compared:

Table 3: RL method comparison in the 24-scene setting.

This ablation reveals several key findings. Policy gradient methods (DPPO) are unstable for diffusion navigation policies, performing even worse than the SFT baseline. DSRL nearly collapses, confirming that optimizing only initial noise is insufficient for learning effective navigation behaviors. DPMD-style reweighted score matching provides a much stronger baseline than DPPO and DSRL, and X-NavDP further improves through same-state group normalization—from 71.57% to 80.48%, a relative gain of approximately 12%. This validates the core value of within-group normalization: providing meaningful gradient signals in hard states.

Exploration Policy Ablation

Figure 4: Ablation study on exploration policies. No-goal perturbation and trajectory reversal are two critical designs for policy learning.

Removing both goal-agnostic perturbation and reverse exploration causes training to collapse entirely, underscoring the criticality of diverse exploration for diffusion policy learning. Using either goal-agnostic perturbation or reverse exploration alone recovers substantial performance, while the full exploration policy achieves the best overall SR/SPL by generating more diverse candidate behaviors. The paper also evaluates a variant that combines goal-agnostic and point-goal trajectories with a fixed weight of 1—this underperforms the full perturbation strategy because it lacks extrapolation ability and remains constrained within the pretrained policy's action manifold.

Reward Design

The RL training reward consists of six terms: a living regularizer (-0.025 per simulator step), path progress, goal arrival, collision, immobility, and humanoid stability. The path progress reward is based on $A^*$ search on the occupancy map, with obstacle-aware traversal cost for a free grid cell:

$$c=\mathrm{clip}(1+0.1/(d_{\text{obs}}+10^{-3}),1,1000)$$

where $d_{\text{obs}}$ is the cell's obstacle clearance in meters. This assigns higher costs to cells near obstacles, favoring paths with greater clearance. The progress reward is $r_{\text{prog}}=0.075\,\phi(\Delta_t,h_t,\delta\psi_t)$, where $\Delta_t$ is the $A^*$ path cost change, $h_t$ indicates forward heading, and $\delta\psi_t$ is the heading change. The piecewise design of $\phi$ is elegant: clear forward progress is rewarded, large turns used for recovery are neither penalized nor rewarded, and insufficient progress is penalized—ensuring recovery behaviors (such as U-turns) are not incorrectly punished.

Limitations

Author-stated limitations: The framework currently relies on short-term temporal context, limiting performance on tasks requiring long-term memory. G1 humanoid robots are particularly challenged in narrow passages—the bulky body and large turning radius make traversal difficult, and arms may contact nearby obstacles while walking. During detours around long obstacles, if the goal is on the opposite side of a wall, limited memory can cause the policy to lose trajectory consistency and return along its original path. Obstacle perception is imperfect, with some obstacles missed. The framework has been validated on three embodiments, each requiring a corresponding pretrained low-level locomotion controller—adapting to a new robot morphology requires additional preparation.

Independent assessment: First, the hierarchical control stack decouples navigation from locomotion, which simplifies training but may limit policy coordination—when the navigation policy predicts waypoints beyond the locomotion controller's tracking range, system behavior may become unpredictable. Second, GQRM's top-k selection (k=5) and 64-candidate sampling may still be insufficient to adequately cover effective behaviors in high-dimensional action spaces, especially in extremely hard states where the candidate group may all be low-value—within-group normalization can distinguish relative quality but the absolute signal remains weak. Third, the real-world evaluation uses only 10 trials per setting, providing limited statistical confidence, and the standard deviation of the 65% average success rate is unclear.

Conclusion and Future Work

X-NavDP demonstrates that diffusion navigation policies can undergo qualitative transformation through carefully designed RL post-training—not just numerical improvement (61%→84% simulation, 10%→65% real), but behavioral emergence: backing out of traps, detouring around long obstacles, and path selection—capabilities entirely absent from the pretrained policy. The two core innovations—self-bootstrapped perturbation exploration and group Q-score reweighted matching—address the fundamental questions of "how to explore" and "how to stably improve," respectively, and are complementary: the former generates diverse same-state candidate groups, while the latter leverages within-group relative advantage for targeted policy updates. Cross-embodiment FiLM modulation enables a single policy to serve three fundamentally different robot morphologies simultaneously.

Future directions include scaling to more diverse scenarios and robot morphologies, investigating tighter coordination between decoupled locomotion and navigation policies, and transferring this post-training framework to broader embodied navigation tasks such as vision-language navigation. The authors also note room for improvement in long-term memory and narrow-passage scenarios, which may require memory-augmented architectures or specialized narrow-passage training strategies.