Robot-Factored World Models via Robot Rendering

1. Overview

Action-conditioned video world models predict future observations from an initial observation and an action signal. In robotics, actions influence future observations through two distinct processes: first realized into robot motion by the body and controller, then the scene responds through contact and object motion. Conditioning directly on action commands asks the world model to learn the realization process, while conditioning on logged future states leaks interaction outcomes it should predict.

This paper proposes Robot-Factored World Models, moving two robot-specific factors outside the world model: (1) action realization—each command is rolled through the robot's own controller and kinematics into a deployment-available nominal trajectory, avoiding both action-realization learning and future-state leakage; (2) robot rendering—the nominal trajectory is rendered through the robot URDF, factoring geometry/kinematics/appearance out of the model. Paired with camera-aware static RGB/depth context, this forms a consistent visual world-model interface across viewpoints and embodiments. The model sees the action only as visible robot geometry and learns how objects respond.

2. Method

2.1 Robot-Factored Visual World-Model Interface

The realization operator $\mathcal{R}$ maps actions to nominal trajectories: $$\tau_{\mathrm{nom}} = \mathcal{R}(\mathbf{a})$$ The rendering operator $\mathcal{V}$ renders the nominal trajectory along the target camera trajectory: $$\mathbf{V}_{\mathrm{robot}} = \mathcal{V}(\tau_{\mathrm{nom}}, \pi_{\mathrm{cam}})$$ The initial scene state is rendered through a static context stream: $$\mathbf{V}_{\mathrm{ctx}} = (I_{\mathrm{rgb}}, I_{\mathrm{depth}})$$ The world model learns: $$\hat{\mathbf{V}}_{\mathrm{future}} = f_\theta(\mathbf{V}_{\mathrm{ctx}}, \mathbf{V}_{\mathrm{robot}}, \mathbf{p})$$ where $\mathbf{p}$ is a text prompt (scene context only). Realization and rendering operators are fixed preprocessing, leaving the model to predict scene response around rendered robot motion.

2.2 Nominal Trajectory Conditioning

The nominal trajectory is a middle signal between action and interaction: robot-only motion realized by the controller/kinematics before scene interaction. Available at deployment and independent of future interaction outcomes. The action-realization gap (raw action→nominal motion) is assigned to the robot-specific realization process; the nominal-realized gap (nominal trajectory→actual observed state in contact rollout) is left to the world model.

2.3 Camera-Aware Static Context and Depth

The static context stream provides scene appearance and geometric context. End-effector depth paired with scene depth resolves depth ambiguity, giving geometric cues for contact and occlusion.

3. Overall Architecture

flowchart LR
    ACTION[Action Command a] --> REALIZE[Realization Operator R
Controller + Kinematics]
    REALIZE --> NOM[Nominal Trajectory τ_nom]
    NOM --> RENDER[Rendering Operator V
URDF + Camera Trajectory]
    RENDER --> ROBOTVIS[Rendered Robot Geometry
RGB + Depth]
    SCENE[Initial Scene] --> CTX[Static Context Stream
RGB + Depth]
    ROBOTVIS --> WM[World Model
Diffusion Video Model]
    CTX --> WM
    PROMPT[Scene Text Prompt] --> WM
    WM --> PRED[Predicted Future Scene Response]
    style REALIZE fill:#4f9eff,color:#fff
    style RENDER fill:#ff6b6b,color:#fff
    style PRED fill:#51cf66,color:#fff

4. Experimental Results

Evaluated on DROID and RoboCasa-GR1. The rendered interface outperforms vector-conditioned baselines (AdaLN state-vector). The same interface extends to unseen robot embodiments and retargeted human demonstrations. Counterfactual trajectory edits and unseen-embodiment composition validate the interface's generalization. A human-demonstration-to-robot-video application converts human hand demonstrations into robot-view videos.

5. Conclusion

Robot-factored world models factor action realization and robot rendering out of the video world model. Actions are realized into deployment-available nominal trajectories and rendered as camera-aligned robot geometry, so the model receives the action as visible robot geometry and learns scene response. On DROID and RoboCasa-GR1, the rendered interface outperforms vector-conditioned baselines and extends to unseen embodiments and retargeted human demos.

Robot forward kinematics

$$ \bm{q}_{1:F}=\Phi_{R}(\bm{a}_{1:F};\bm{q}_{0}) $$