PAPER DEEP DIVE
DeVA: Decoupled Video-Action Model with physical guidance for robot policy learning
Generalizable robot manipulation requires policies that can anticipate how visual scenes evolve while executing language instructions. While recent Vision-Language-Action models benefit from large-scale pretraining, their predominantly static pretraining objectives provide limited supervision for physical dynamics and temporal causality, leaving control-relevant knowledge to be learned from downstream robot demonstrations. Video generative models offer a promising foundation by encoding rich spatiotemporal priors through future predictions. However, existing Video-Action Models either couple video and action prediction in a shared backbone, making policy adaptation harder to optimize, or under-utilize video information when guiding the action branch. In this work, we introduce DeVA, a Decoupled Video-Action model with specialized video and action experts, multi-level feature transfer, and physically salient guidance. DeVA transfers representations from multiple video layers to the action expert, enabling rich information exchange while making policy learning more tractable. It further supervises intermediate video features and the action stream with physically salient guidance (affordance/depth). Experiments on both simulation benchmarks and real-world deployment demonstrate strong performance with limited data, faster convergence than a unified architecture, and clear performance gains from physical guidance.
1. Overview: Decoupled Video-Action Model with Physical Guidance
DeVA (Decoupled Video-Action Model), proposed by Mengqi Zhang, Sahil Khose et al. at UC Irvine and Georgia Tech (July 2026), is a decoupled video-action model with specialized video and action experts, multi-level feature transfer, and physically salient guidance. VLA models benefit from large-scale pretraining but their static pretraining objectives provide limited supervision for physical dynamics and temporal causality, leaving control-relevant knowledge to be learned from downstream robot demonstrations. Video generative models encode rich spatiotemporal priors through future predictions, offering a promising foundation. However, existing video-action models either couple video and action prediction in a shared backbone making policy adaptation harder to optimize, or under-utilize video information when guiding the action branch. DeVA transfers representations from multiple video layers to the action expert, enabling rich information exchange while making policy learning more tractable, and supervises intermediate video features and the action stream with physically salient guidance (affordance/depth). It achieves 72.0% on RoboCasa (with only 50 demos/task, beating methods trained with 300-3000 demos), 99.0% on LIBERO, 80.8% on LIBERO-Plus, and 74% on real bimanual tasks.
2. Core Problem: The Optimization Dilemma of Video-Action Coupling
Generalizable robot manipulation requires policies that anticipate how visual scenes evolve while executing language instructions. Video generative models encode rich spatiotemporal priors through future predictions, but coupling video and action prediction in a shared backbone forces visual dynamics and robot actions to share a single feature space, making policy adaptation harder to optimize; loosely using video information to guide the action branch under-utilizes video priors. DeVA asks: can video and action experts be decoupled to preserve modality-specific capacity while structured interaction lets the action expert access predictive dynamics learned by the video backbone? Let video and action expert parameters be $\theta_v, \theta_a$; the joint optimization is:
$$\min_{\theta_v, \theta_a} \mathbb{E}\left[\mathcal{L}_v(\theta_v) + \mathcal{L}_a(\theta_a, \phi(\theta_v))\right], \quad \theta_v \perp \theta_a$$where $\phi(\cdot)$ denotes multi-level feature transfer from video to action, and $\theta_v \perp \theta_a$ indicates parameter decoupling.
3. Method: Decoupled Architecture with Multi-Level Feature Interaction
3.1 Decoupled Video-Action Architecture
DeVA consists of a video expert (future observation modeling) and an action expert (robot action prediction). The video expert is initialized from Cosmos-Predict2 (a latent video diffusion transformer); its spatiotemporal VAE encodes video observations into compact latent tokens, and the video transformer models evolution conditioned on the current frame and language instruction. The action expert follows a similar DiT-style architecture in the action domain, predicting action trajectories aligned with the corresponding predicted future. The two experts maintain separate parameters and token representations, avoiding forcing visual dynamics and robot actions to share a single feature space, while being jointly optimized via separate generative objectives. The action expert receives multi-layer video features $\{F_v^{(l)}\}$ via cross-attention:
$$F_a^{(l)} = \text{CrossAttn}\left(Q_a^{(l)},\, K_v^{(l)},\, V_v^{(l)}\right) + F_a^{(l-1)}$$where $Q_a^{(l)}$ is the action query, $(K_v^{(l)}, V_v^{(l)})$ come from the $l$-th video layer. Bridge tokens $B$ aggregate video context and enter the action stream via self-attention:
$$B^{(l)} = \text{SelfAttn}\left([B^{(l-1)}; \text{Agg}(F_v^{(1:l)})]\right), \quad F_a^{(l)} \mathrel{+}= \text{Proj}(B^{(l)})$$3.2 Multi-Level Feature Interaction
Fully isolating experts would prevent the action expert from accessing predictive dynamics learned by the video backbone. Diffusion models encode complementary representations across network layers and denoising stages; DeVA transfers intermediate video representations from multiple levels to corresponding action blocks via layer-wise cross-attention. Learnable bridge tokens provide an additional compact interface by aggregating video context and introducing it into the action stream via self-attention.
3.3 Physically Salient Guidance
The multi-level interaction provides the action expert with predictive video representations, but these are optimized primarily for future prediction and may not explicitly emphasize interaction regions and geometric structure needed for control. DeVA applies task-conditioned affordance and relative-depth supervision to intermediate video features; decoded features are also injected into the action expert as physical guidance. Affordance & Depth Decoding: features sampled uniformly from video backbone layers feed into lightweight DPT-style decoders with interleaved temporal-attention layers, predicting two complementary physical signals. The affordance map is defined as:
$$\mathcal{A}_t(u, v) = P\left(p_t^{ee} = (u, v) \mid O_t, T\right)$$where $p_t^{ee}$ is the end-effector image-plane location; each pixel represents its likelihood as an interaction location under instruction $T$. The relative depth map $D_t$ complements affordance:
$$D_t(u, v) \approx \frac{z(u, v) - z_{\min}}{z_{\max} - z_{\min}}, \quad D_t \in [0, 1]$$The two capture manipulation geometry from different dimensions: affordance marks "where to interact," depth marks "how far." Physical Guidance Injection: features from the decoders' final spatiotemporal blocks are projected to video feature spatial resolution, flattened into tokens, concatenated with multi-level video representations along channels, and supplied as additional keys/values to action cross-attention.
3.4 Training and Inference
Two-stage training: Stage 1 video and decoder warmup; Stage 2 joint video-action training. The joint loss combines video denoising $\mathcal{L}_v$ and action denoising $\mathcal{L}_a$:
$$\mathcal{L}_{\text{total}} = \mathcal{L}_v + \lambda_a \mathcal{L}_a + \lambda_{\text{aff}} \mathcal{L}_{\text{aff}} + \lambda_{\text{dep}} \mathcal{L}_{\text{dep}}$$where $\mathcal{L}_{\text{aff}}, \mathcal{L}_{\text{dep}}$ are affordance and depth supervision losses. At inference, the action expert generates action chunks.
4. Experiments
4.1 RoboCasa Simulation
Evaluated on 24 kitchen manipulation tasks, 50 rollouts × 3 seeds = 3600 rollouts. DeVA achieves 72.0% success rate with only 50 demos/task, improving up to 22.0 points under the same data budget and outperforming several methods trained with 300-3000 demos.
| Benchmark | DeVA | Highlight |
|---|---|---|
| RoboCasa (50 demos/task) | 72.0% | Beats 300-3000 demo methods |
| LIBERO (4 suites avg) | 99.0% | Highest overall avg |
| LIBERO-Plus (robustness) | 80.8% | +11.2pp over OpenVLA-OFT |
| Real bimanual (3 tasks avg) | 74% | GR00T-N1.6 48%, Cosmos 34% |
4.2 LIBERO and LIBERO-Plus
Four standard suites (Spatial/Object/Goal/Long) totaling 40 tasks, 50 rollouts × 3 seeds each. DeVA achieves the highest overall average of 99.0%. LIBERO-Plus contains 10030 perturbation variants, 1 rollout each; DeVA achieves 80.8%, exceeding the strongest baseline OpenVLA-OFT by 11.2 percentage points. The 18.2-point drop from 99.0% to 80.8% reveals a remaining gap between in-distribution and robustness.
4.3 Real-World Deployment
On the I2RT platform, three bimanual tasks (Handover Marker, Lift Pot, Pick Up Bottles), 30 demos each, multi-task fine-tuning of DeVA, Cosmos Policy, GR00T-N1.6, 10 trials each. DeVA averages 74%, vs. GR00T-N1.6 48% and Cosmos Policy 34%.
5. Ablation Studies
Ablating predictive visual modeling and physical guidance on RoboCasa (matched budget): goal-image prediction improves success from 19.8% to 25.8%, future-video prediction further to 36.8%; decoupled architecture + multi-level transfer reaches 66.0%. Affordance and relative depth each independently improve the base model; combining both achieves the highest 72.0%. Training efficiency: decoupled variants converge in ~40-50K steps, with base and physically guided variants reaching ~66% and ~71% vs. 34% for the unified counterpart; DeVA processes up to 20× fewer training examples than Cosmos-Policy.
| Config | Success Rate | Increment |
|---|---|---|
| Action-only model | 19.8% | baseline |
| +goal-image prediction | 25.8% | +6.0 |
| +future-video prediction | 36.8% | +11.0 |
| +decoupled+multi-level transfer | 66.0% | +29.2 |
| +affordance+depth guidance | 72.0% | +6.0 |
6. Limitations
- Computational cost inherited: DeVA inherits the computational cost of its pretrained video backbone; training spatiotemporal attention over future observations is more expensive than action-only policy learning.
- Inference overhead: Jointly denoising video and action streams introduces additional inference overhead vs. models predicting only compact latent states and actions.
- Robustness gap: The 18.2-point drop from LIBERO 99.0% to LIBERO-Plus 80.8% reveals a remaining gap between in-distribution and robustness.
- Real-data pseudo-label dependency: Affordance pseudo-labels depend on an off-the-shelf model and depth on a prediction model; their errors may propagate.
- Efficiency improvements not realized: Latent-space prediction and accelerated sampling are future directions; current efficiency remains to be improved.
7. Conclusion
DeVA is a decoupled video-action model that transfers predictive video representations to robot control through specialized experts, multi-level feature transfer, and physically salient affordance/depth guidance. Across simulation benchmarks and real bimanual manipulation, DeVA achieves strong performance with limited demonstrations, converges faster than a matched unified architecture, and benefits from physical guidance, demonstrating the promise of video generative priors for robot policy learning. The core insight: video and action need not share a backbone—decoupling lets each specialize, while multi-level cross-attention and physical guidance let video priors' predictive dynamics and geometric structure flow effectively into the action policy, preserving capacity while promoting transfer.
flowchart LR
A["Current frame+language"] --> B["Video Expert Cosmos-Predict2
future obs modeling"]
B --> C["Multi-layer video features F_v^(l)"]
C --> D["DPT decoder
affordance+depth"]
D --> E["Physical guidance features"]
C --> F["Layer-wise cross-attn
+bridge tokens"]
E --> F
F --> G["Action Expert DiT
action chunk prediction"]
G --> H["Robot execution"]
H --> A
