PAVXploreRL: Physical-Action-Visual World Model Reinforcement Learning with Action Exploration

Paper: PAVXploreRL: Physical-Action-Visual World Model Reinforcement Learning with Action Exploration
Authors: Han Wang, Zijun Wang, Shuoshuo Xue, Rui Cao, Fen Jiao Cheng, Xiao Dang Liang, Roy Ka-Wei Lee
Affiliation: Singapore University of Technology and Design; Sun Yat-sen University
Link: arXiv:2607.16602v1 [cs.CV], July 2026
Code: ✅ Open-source at github.com/Social-AI-Studio/PAVXploreRL

1. Abstract

Action-conditioned world models are a key component of embodied AI, serving as scalable policy evaluators that reduce reliance on expensive real-world rollouts. To accurately capture diverse action-induced dynamics, such models should satisfy three key objectives — Physical Plausibility (P), Action Adherence (A), and Visual Fidelity (V), collectively PAV — while remaining robust to both in-distribution (ID) expert demonstrations and out-of-distribution (OOD) actions. Existing methods primarily rely on ID action-video pairs and pixel-level reconstruction losses, which do not explicitly optimize PAV objectives and lack OOD action exploration. This paper proposes PAVXploreRL, built on the latent world model VJEPA-2, with three complementary rewards: (1) raw VJEPA reward (physical consistency), (2) embodiment-focused VJEPA reward (action execution accuracy), (3) visual fidelity reward (perceptually realistic frames). Using VJEPA-2's latent predictive capabilities, rewards are computable for both ID action-video pairs and OOD actions generated by perturbing ID actions (no paired video needed). On Agibot and DROID benchmarks, the method improves over pretrained baselines by an average of 5.6%.

2. Background and Motivation

Action-conditioned world models take robot actions as input and predict future observations consistent with real-world interactions, serving as policy evaluators. Integrating them into policy training reduces real-robot reliance and improves task success rates.

Three limitations of existing methods:

• Insufficient physical plausibility: Commonly assessed by pretrained VLMs, but VLMs lack domain-specific training and have limited physical understanding. VJEPA, a latent world model, demonstrates superior physical understanding.
• Indirect action adherence evaluation: Inverse dynamics models (IDMs) infer actions from frame differences, creating an inverse information flow inconsistent with forward world models.
• ID-only action-video pairs: Existing world models train only on human expert success trajectories (ID), potentially overestimating the effectiveness of OOD actions from suboptimal policies, inflating success rates. Reliable world models must generalize across diverse policies.

Figure 1: PAVXploreRL vs prior RL. Unlike ID-only expert approaches, it enables both ID and OOD action exploration, replacing pixel/VLM rewards with a domain-adapted latent world model for direct PAV evaluation.

3. Core Method

3.1 World Model Architecture

The world model $\pi$ is defined as a conditional video predictor. At step $t=i$, given first frame $I_0$, history frames $I_{t\in(i-2T,i]}$, and future actions $A_{t\in[i,i+T)}$, the model predicts:

$$\pi\big(I_{0},I_{t\in(i-2T,i]},A_{t\in[i,i+T)}\big)=\hat{I}_{t\in(i,i+T]}\approx I_{t\in(i,i+T]}$$

The pretrained video diffusion model is adapted with: (1) action embeddings injected via AdaLN and time embeddings for action conditioning; (2) compact history representation for efficient long-horizon interaction. Training uses Diffusion-NFT (efficient RL without log-likelihood computation).

3.2 Reward Function Design

Core innovation: five reward components targeting different aspects of video quality, covering all three PAV objectives.

OOD Action Generation: Three perturbations applied to non-static GT action segments (per hand and action type):

• Temporal shift: segment boundaries $(s,e)$ perturbed within $\pm 0.6$ s;
• Structured magnitude bias: chunk-wise multiplicative scaling with $\pm 0.15$ deviations;
• Smooth temporal noise: Gaussian noise ($\sigma=0.02$) per step, temporally accumulated.

(1) Pixel-level Reward $r_{\text{pixel}}$: MSE, SSIM, LPIPS between predicted and GT frames, averaged after group normalization:

$$r_{\text{pixel}}=\frac{1}{3}\left(r_{\text{MSE}}+r_{\text{SSIM}}+r_{\text{LPIPS}}\right)$$

Normalization (applied to all forward rewards):

$$r_{i}^{\mathrm{norm}}=0.5+0.5\cdot\mathrm{clip}\!\left(\frac{r_{i}-\mu_{G}}{\sigma_{G}+\epsilon},-1,1\right)$$

(2) VJEPA Encoder Reward $r_{\text{VJEPA-E}}$: Uses VJEPA-2 encoder $E(\cdot)$ for physical understanding, measuring latent similarity between predicted and GT frames (ID branch only):

$$r_{\text{VJEPA-E}}=-\left\|E(\hat{I}_{t\in(i,i+T]})-E(I_{t\in(i,i+T]})\right\|_{1}$$

(3) VJEPA Predictor Reward $r_{\text{VJEPA-P}}$: Uses action-conditioned VJEPA-2 predictor $P(\cdot)$ to predict future latents from start frame $I_i$ and actions $A$ — no GT video needed, enabling the OOD branch:

$$r_{\text{VJEPA-P}}(A)=-\left\|E(\hat{I}_{t\in(i,i+T]})-P\big(E(I_{i}),A_{t\in[i,i+T)}\big)\right\|_{1}$$

Static regularization with zero-action $A_0$ to prevent static-favoring:

$$r_{\text{VJEPA-P}}=r_{\text{VJEPA-P}}(A)-\lambda_{\text{static}}r_{\text{VJEPA-P}}(A_{0})$$

(4) Embodiment-Focused VJEPA Reward: Encoder and predictor rewards computed on both raw frames and embodiment-focused frames (segmented robot regions):

$$r_{\text{VJEPA-E/P}}=r_{\text{VJEPA-E/P}}^{\text{raw}}+\lambda_{\text{emb}}r_{\text{VJEPA-E/P}}^{\text{emb}}$$

(5) Visual Fidelity Reward $r_{\text{VF}}$: Combines frame-level and temporal no-reference metrics (NIQE, SSIM, temporal consistency) for sharpness, perceptual quality, motion consistency, structural coherence, and color stability.

Final Reward Combinations:

$$r_{\text{ID}}=\lambda_{\text{pixel}}\,r_{\text{pixel}}+\lambda_{\text{VJEPA}}\,r_{\text{VJEPA-E}}+\lambda_{\text{VF}}\,r_{\text{VF}}$$

$$r_{\text{OOD}}=(1-\lambda_{\text{VF}})\,r_{\text{VJEPA-P}}+\lambda_{\text{VF}}\,r_{\text{VF}}$$

Figure 2: PAVXploreRL framework overview. ID branch uses pixel+VJEPA-encoder+VF rewards; OOD branch uses VJEPA-predictor+VF rewards. Rewards weight the diffusion policy optimization loss.

3.3 VJEPA-2 Training Pipeline

Two modifications to adapt VJEPA-2 to the robotic domain:

• Dual input streams: Raw images + embodiment-focused images (segmented robot regions) for explicit agent structure modeling;
• Multi-step autoregressive prediction: Accumulated loss across the entire predicted frame chunk $T$, enabling prediction from start frame and actions only. Encoder frozen; action-conditioned predictor trained from scratch.

Two-stage training: Stage 1 — standard one-step future prediction; Stage 2 — random sampling of raw/embodiment frames + autoregressive rollout with accumulated loss for long-horizon consistency.

flowchart TD
    A[History Frames + Actions] --> B[World Model
Video Diffusion + AdaLN]
    B --> C[Predicted Future Frames]
    C --> D{Reward Evaluation}
    D -->|ID Branch| E[Pixel Reward r_pixel
MSE+SSIM+LPIPS]
    D -->|ID Branch| F[VJEPA Encoder r_VJEPA-E
GT vs Predicted]
    D -->|OOD Branch| G[VJEPA Predictor r_VJEPA-P
No GT, Latent Prediction]
    D -->|Both| H[Visual Fidelity r_VF
No-Reference Metrics]
    D -->|Both| I[Embodiment Reward
Segmented Robot Region]
    E --> J[Final Reward r_ID / r_OOD]
    F --> J
    G --> J
    H --> J
    I --> J
    J --> K[Diffusion-NFT
Weighted Policy Optimization]
    K --> B

4. Key Experiments

4.1 Main Results

9.6-second autoregressive trajectories (3 sequential 3-step action chunks) on Agibot and DROID:

RL consistently improves the pretrained model on both datasets, particularly in SSIM, LPIPS, and flow consistency. On DROID, PT trails some baselines but RL training enables it to surpass them.

Figure 3: Qualitative comparison of long-horizon rollouts on Agibot. The RL-enhanced model achieves superior physical plausibility, action adherence, and visual fidelity.

4.2 Ablation Study

Removing static regularization causes the largest performance drop (Pred loss 0.374→0.409), followed by embodiment-focused and visual fidelity rewards. All components contribute; the full unified reward is best.

4.3 VLA Evaluation: World Model as Policy Evaluator

Testing $\pi_0$ and $\pi_{0.5}$ on 5 DROID tasks vs Ctrl-World. ~300 expert trajectories per task. 5K steps SFT + 1K steps RL from DROID checkpoint. Results show PAVXploreRL reduces overestimation of success rates, providing more accurate real-world performance estimates. Ctrl-World often predicts success even when the policy fails in the real world, while this model faithfully reflects real-world dynamics (object collisions, loose grasps).

5. Limitations and Future Work

• RL design focus: Base world model doesn't explicitly incorporate components like multi-view consistency. The RL framework is orthogonal to these design choices and extensible to future architectures.
• Long training time: Despite using Diffusion-NFT, long rollouts and reward computation incur substantial cost, making it primarily suitable as a post-training method.
• Future: Extend to advanced pretrained world models; integrate multi-view consistency.

6. Conclusion

PAVXploreRL presents a model-agnostic RL pipeline with a unified reward function jointly optimizing the PAV targets — Physical Plausibility, Action Adherence, and Visual Fidelity. The pipeline trains on both ID and OOD actions without paired video supervision. The core innovation is leveraging VJEPA-2's latent prediction capability to design GT-free rewards, enabling OOD action exploration — perturbing ID actions to generate OOD actions and evaluating physical plausibility via VJEPA predictor (not GT video). On Agibot and DROID benchmarks, the method improves pretrained baselines by an average of 5.6%. As a policy evaluator, it reduces success rate overestimation and provides more accurate real-world performance estimates. The RL framework is orthogonal to the base world model architecture and extensible to future models.