RL$^2$-VLA: Adaptive RL Latent Compositional Steering with Test-Time Scaling for Vision-Language-Action Models

Background and Motivation

Vision-language-action (VLA) models have demonstrated increasingly robust and generalizable performance across diverse manipulation tasks through large-scale robot demonstrations and strong priors from vision-language models. However, performance often degrades significantly on challenging and out-of-domain (OOD) tasks—e.g., from 70.2% to 36.0%. Prior efforts improved models during pretraining via data scaling and architectural design, and during post-training via preference alignment and chain-of-thought reasoning. More recently, attention has shifted toward improving VLAs during deployment—leveraging additional computation at test time without extensive data collection and retraining.

Existing methods follow two parallel directions. Inference-time steering divides into two categories: discrete action selection samples multiple candidate actions and selects among them using external verifiers—preserving the policy's learned distribution but limited by sample diversity; differentiable steering guides action generation through VLM-derived scoring functions or differentiable metrics from world models—capable of steering beyond the original distribution but limited by imperfect physical grounding in VLMs and world model accuracy. Test-time scaling repeatedly samples actions or generates diverse samples from rephrased instructions, but all samples ultimately originate from the same policy and may inherit correlated biases and failure modes.

More fundamentally, existing steering and scaling approaches apply a fixed intervention strategy without distinguishing situations where the base VLA is already likely to succeed from those where intervention may be necessary. Humans, however, exhibit markedly different behavior: when faced with uncertainty, they actively broaden the diversity of alternatives considered. When reaching for a mug on an uncluttered table, we grasp it directly; but if partially occluded, we may approach from another side or first remove the obstacle. Diverse alternatives are most useful when the default behavior is likely to fail, but may introduce unnecessary perturbations when the base solution is likely to succeed.

This leads to the core question: How and when should we steer pretrained VLAs toward diverse action candidates to enhance robustness in challenging/OOD scenarios? RL²-VLA answers this with "Reinforcement Learning on VLA Latents."

Preliminaries

Understanding RL²-VLA requires several key concepts. First, flow-matching policies: generating action sequences by learning a vector field (flow velocity) from noise to actions, used by VLAs like $\pi_0$. Given noised action $\mathbf{a}_t$, the flow velocity $v_\theta(\mathbf{a}_t, t, \mathbf{c})$ defines the denoising direction. Second, offline reinforcement learning: learning policies from fixed datasets without online interaction, evaluating action value via Q-functions. Third, classifier guidance: steering diffusion/flow-matching sampling toward high-value regions via gradients of a differentiable scoring function $g(o, a, \ell)$.

Another key concept is compositional steering: combining the velocity fields of two flow-matching policies via weighted average, producing a new distribution that fuses both behavioral priors—retaining imitation learning's strong priors while introducing RL-discovered diverse behaviors.

Method: The RL² Framework

RL² answers two questions: how to steer (compositional steering) and when to steer (adaptive failure detection). The pipeline: extract action-expert latents from the frozen VLA → RL flow-matching policy provides guidance velocity → compose with VLA velocity → failure detector decides whether to activate → verifier selects best action.

Framework Overview

flowchart TB
    subgraph VLA["Frozen VLA (pi_0/pi_0.5)"]
        OBS[Observation + Language] --> AE[Action Expert]
        AE --> EL[Latents e_t]
        AE --> VV[VLA Flow Velocity V_VLA]
    end
    subgraph RLP["RL Steering Policy"]
        EL --> RLP2[Flow-Matching Policy pi_RL]
        RLP2 --> VRL[RL Flow Velocity V_RL]
    end
    subgraph FD["Failure Detection"]
        EL --> SAFE[SAFE LSTM Detector]
        SAFE --> SC[Failure Score s_t]
        SC --> CP[Conformal Prediction CP]
        CP --> DEC{Failing?}
    end
    VV --> COMP[Velocity Composition]
    VRL --> COMP
    DEC -->|Yes| COMP
    COMP --> AS[Steered Action Candidates]
    DEC -->|No| AS2[VLA Original Candidates]
    AS --> VER[Verifier Scoring]
    AS2 --> VER
    VER --> BEST[Best Action Executed]

The diagram above shows RL²'s complete framework. The VLA action expert's latents $e_t$ feed both the RL steering policy and the failure detection module. When failure is detected, RL velocity is composed with VLA velocity to generate steered candidates; otherwise, the system falls back to the VLA's original distribution. A verifier selects the best action from candidates.

RL Latent Policy Training

To generate a differentiable guidance function $g(o, a, \ell)$, RL² first trains a flow-matching RL policy $\pi_{\text{RL}}(a_{t:t+H} \mid e_t)$ conditioned on latents extracted from the VLA action expert. Training data is constructed by augmenting BridgeV2 and DROID datasets with policy latents: $\mathcal{D} = \{o_i, a_i, \ell_i, e_i\}_{i=1}^N$, where $e_i$ is the embedding obtained by passing observations through the VLA.

The challenge of training flow-matching actor-critic networks is that direct backpropagation from the critic $Q(s,a)$ through the policy's multi-step flow-matching process is numerically unstable. RL² adopts QAM (Q-Adjusted Matching), which replaces unstable backpropagation with a stable, step-wise matching objective. QAM uses a lean adjoint state $\tilde{g}_t$ as a time-dependent guidance signal, computed backwards from the terminal action $a_1$ via a reverse ODE:

$$d\tilde{g}_t = -\nabla_{a_t}[2f_\beta(s, a_t, t) - a_t/t]\,\tilde{g}_t\,dt$$

with terminal condition $\tilde{g}_1 = -\tau\nabla_{a_1}Q(s, a_1)$, where $f_\beta$ is a fixed behavior prior and $\tau$ is the inverse temperature. The policy's target velocity field $f_\theta$ is optimized to align with this signal by minimizing the matching loss:

$$\mathcal{L}_{\text{AM}}(\theta) = \mathbb{E}_{s,\{a_t\}}\int_0^1 \left\|\frac{2(f_\theta(s, a_t, t) - f_\beta(s, a_t, t))}{\sigma_t} + \sigma_t \tilde{g}_t\right\|_2^2\,dt$$

where $\sigma_t = \sqrt{2(1-t)/t}$ is a fixed noise schedule. This formulation ensures convergence to the optimal behavior-regularized distribution:

$$\pi(a|s) \propto \pi_\beta(a|s)\exp(\tau Q(s,a))$$

while maintaining full expressivity through a stable, first-order optimization path. In practice, simply adjusting the inverse temperature $\tau$ suffices to outperform behavior cloning (BC) baselines. The RL² framework is compatible with different RL methods, including non-flow-based methods like V-GPS trained via CQL for autoregressive VLAs like OpenVLA.

Compositional Steering

At inference, RL² composes the RL policy's flow velocity with the frozen VLA's flow velocity. At each flow-matching step, the composed velocity is:

$$V_{\text{comp}} = W \cdot V_{\text{VLA}} + (1-W) \cdot V_{\text{RL}}$$

where $W$ is the composition weight. This compositional steering fuses the behavioral priors of large-scale imitation learning with the action diversity induced by offline RL. The VLA contributes strong behavioral priors learned from large-scale but inherently imperfect demonstrations, while offline RL encourages the discovery of alternative high-value behaviors and induces greater action diversity beyond dominant demonstration modes.

For autoregressive VLAs (e.g., OpenVLA), RL² uses Gaussian perturbation: fitting a Gaussian distribution over 9 action samples (VLA and/or composed), then re-sampling 32 actions. For flow-matching VLAs (e.g., $\pi_0$), 8 rephrases with 5 action samples each are used.

Failure Detection for Adaptive Steering

RL²'s core insight is that compositional steering's diversity is most beneficial when the base VLA is likely to fail, but can unnecessarily perturb already-accurate actions when success is likely. Therefore, a failure-detection trigger enables steering only at timesteps when the VLA is detected as failing, otherwise falling back to the original VLA distribution.

The failure detector uses SAFE—a lightweight multitask detector conditioned on informative internal VLA features. An LSTM detector $f_\psi$ takes VLA features up to the current timestep and outputs a failure score:

$$s_t = f_\psi(e_{0:t}) \in [0, 1]$$

Steering is triggered by a simple threshold rule: intervene when the predicted failure score exceeds a time-varying threshold $\delta_t$. This is calibrated using conformal prediction (CP) with a one-sided time-varying formulation. Given a rollout-level score sequence $s_{1:T}$ and significance level $\alpha \in (0,1)$, the CP band is $C_\alpha = \{[\text{lower}_t, \text{upper}_t]\}$ with $\text{upper}_t = \mu_t + h_t$, where $\mu_t$ and $h_t$ are the time-varying mean failure score and conformal bandwidth. Calibration uses successful rollouts so that any new successful rollout satisfies $s_t < \mu_t + h_t$ with probability at least $1-\alpha$.

In practice, the best CP significance level $\alpha$ varies across tasks—$\alpha$ directly controls steering trigger frequency. RL² introduces a test-time alpha selection heuristic: evaluating candidate $\alpha$ values on the validation set by balanced accuracy and selecting the best.

Test-Time Scaling Laws

RL² is the first to establish distinct test-time scaling laws comparing steering approaches separately for success and failure states. Assuming an oracle verifier, the relationship between normalized action error and the number of generated action samples follows an exponential power law. The analysis reveals a key insight: compositional steering's diversity is most useful when the base VLA is likely to fail—it can avoid dominant failure modes, but may unnecessarily degrade already-accurate actions when success is likely. This finding directly motivates the adaptive steering design.

Experimental Results

Experimental Setup

RL² is evaluated across multiple simulation benchmarks, VLAs, and verifiers. SIMPLER (in-domain): OpenVLA on 4 standard BridgeV2 tasks, using V-GPS's CQL policy as RL steering and RoboMonkey as verifier. SIMPLER (OOD): $\pi_0$ on the same 4 tasks with red-teaming instructions for OOD language prompts, plus 4 OOD task environments, using QAM policy and CoVer verifier. PolaRiS (OOD): $\pi_{0.5}$ on Franka Panda with 3 DROID-adapted tasks under OOD red-teaming prompts. All experiments: 50 runs × 3 random seeds.

Simulation Results

Table 1: PolaRiS OOD prompt evaluation. Adaptive RL² achieves up to +17.3% success rate over Rephrase baseline.

On SIMPLER in-domain tasks, OpenVLA's adaptive RL² achieves +7.5% average (task-wise +19.4%) over the strongest Repeated baseline. On SIMPLER OOD language prompts, $\pi_0$'s adaptive RL² achieves +10.1% average (up to +14.7%) over Rephrase. On SIMPLER OOD task environments, +8.5% average (up to +14.6%). On PolaRiS OOD prompts, $\pi_{0.5}$'s adaptive RL² achieves up to +17.3% success rate and +12.4% progress rate over Rephrase.

Ablation Studies

Adaptiveness is crucial. Removing the failure detection module (Compose-Always) drops performance from 42.7% to 33.8% on $\pi_{0.5}$, and from 53.8% to 46.5% on OOD tasks. Non-adaptive steering underperforms adaptive steering across all settings, validating the core insight of "steer only during failure."

Table 2: RL training and VLA latents enhance compositional steering. Without latents, performance collapses to 0.5%.

Using latents and RL training is effective. RL training consistently outperforms behavior cloning (BC): $\pi_0$ RL 57.8% vs BC 55.8%, $\pi_{0.5}$ RL 33.8% vs BC 29.7%. A critical finding: removing VLA latents collapses OpenVLA performance to 0.5%—latents provide the RL policy with crucial information about the current state; without them, the RL policy cannot effectively steer.

SAFE failure detection outperforms other triggers. SAFE trigger achieves 60.3% (OOD prompts) and 53.8% (OOD tasks) on $\pi_0$, outperforming CoVer trigger (57.8%/50.3%) and Always trigger (57.8%/46.5%), and on $\pi_{0.5}$: 42.7% vs 39.8% vs 33.8%.

Real-World Evaluation

Validated on a PiperX manipulator with Realsense D405 camera, using the same $\pi_0$ model weights as simulation, QAM as steering policy, and CoVer as verifier. Four OOD tasks (2 red-teaming prompts + 2 OOD environments), 10 runs × 3 seeds. Adaptive RL² achieves +17.5% average over Rephrase baseline and +14.2% over non-adaptive RL². Real-world results align closely with simulation, validating the framework's practicality and modularity.

Limitations

The paper acknowledges three limitations. First, steering function: while many lightweight steering functions were evaluated, comparison with differentiable steering from large models like VLMs or other VLAs is left to future work, as the focus is on minimizing additional overhead. Second, failure detection: the alpha selection heuristic requires additional test-time evaluation to determine the optimal CP significance level $\alpha$, and could be improved with more effective metrics than balanced accuracy (e.g., detection timeliness); current reliance on online rollout collection for training could be replaced by large-scale offline datasets. Third, verifier: the work assumes a robust verifier that effectively selects among action candidates; future work will co-train the RL steering policy with the verifier to enhance selection robustness.

From an independent assessment, the generalization of the failure detector is a key bottleneck. The paper acknowledges that the SAFE detector trained on simulation cannot directly generalize to real hardware—requiring additional real-world rollout collection for retraining. This means each deployment to a new platform requires re-collecting failure data, increasing deployment cost. Additionally, the selection of composition weight $W$ is insufficiently discussed—a fixed weight may be suboptimal across different tasks and states, and adaptive weight adjustment could further improve performance.

Conclusion and Outlook

RL²-VLA presents an inference-time steering framework that improves pretrained VLAs through adaptive RL compositional steering conditioned on VLA latents. A lightweight RL policy trained on VLA latents and the same offline VLA finetuning dataset improves downstream performance by steering imitation-learned samples toward diverse actions beyond demonstration modes. The paper discovers distinct test-time scaling laws for VLA steering under success and failure states, showing RL compositional steering is most beneficial when the base policy is likely to fail. Experiments across VLAs, verifiers, and manipulation benchmarks validate RL² as a modular framework for in-domain and OOD VLA deployment.

This work positions adaptive test-time steering as a practical pathway toward robust, general-purpose robotics foundation models. The core insight—that "diversity is most useful during failure but can be harmful during success"—applies not only to VLAs but also provides a design principle for inference-time strategies of other foundation models.