VT-Bridge: Bridging Pretrained Foundation VLAs to VTLAs via Lightweight Residual Adaptation

Paper Metadata

Title: VT-Bridge: Bridging Pretrained Foundation VLAs to VTLAs via Lightweight Residual Adaptation

Authors: Yansong Wu, Tuo Yang, Rongping Zhao, Lingyun Chen, Xiao Chen, Junnan Li, Fan Wu, Alois Knoll (Technical University of Munich, Mohamed bin Zayed University of Artificial Intelligence, Shanghai University)

Links: arXiv 2609.22606v1 (cs.RO, submitted 2026-09-18, CC BY-SA 4.0); IEEE RA-L version DOI 10.1109/LRA.2026.3692345; project page

Code status: the acknowledgements say the codebase is being prepared for open-source release. As of v1 no public repository exists, and the Code button on the project page does not resolve to one.

One-Sentence Summary

VT-Bridge leaves a pretrained VLA architecturally untouched and bolts a 0.98M-parameter tactile residual adapter onto it at the robot's execution rate. With at most 50 visuotactile demonstrations per task it turns the open foundation VLAs π0, π0.5 and SmolVLA into tactile-aware VTLAs, and lifts the mean strict success rate across four contact-rich manipulation tasks from 11.7% to 62.9%.

Background and Motivation

Vision-language-action models are the most practical route we have to general robot intelligence: one pretrained policy reads a natural-language instruction and produces actions across a wide range of manipulation settings. Large-scale pretraining has made zero-shot and few-demonstration fine-tuning viable for a growing list of tasks. The paper starts from an uncomfortable observation that keeps recurring in this literature: even after task-level fine-tuning, existing VLAs still fail most of the time on real-world contact-rich manipulation. Predicting actions from vision, language and proprioception alone, that is from non-contact modalities only, does not carry enough information to close a physical interaction reliably.

The natural response is to feed tactile signals into the policy, which produces vision-tactile-language-action (VTLA) models. Recent work in this direction differs mainly in where and how the tactile stream enters the network, and reports clear gains on contact-rich tasks; several ablations isolate the tactile contribution on its own and show that touch is not a nice-to-have but a load-bearing modality for reliable physical interaction.

Those gains come with a visible bill. On the data side, paired visuotactile trajectories are far scarcer in public robot datasets than vision-only ones, and collecting good tactile demonstrations demands fine force modulation from the human operator, which makes bimanual teleoperation substantially harder. On the model side, a tactile encoder plus a cross-modal fusion stage adds parameters and architectural complexity, and with them training cost. Constrained by both, existing VTLAs cover a much narrower task range than a general VLA such as π0.

That tension frames the paper's central question: can an existing open-source VLA be adapted into a VTLA at a controlled cost? VT-Bridge answers by bridging instead of rebuilding. The pretrained VLA keeps its original architecture and the general knowledge stored in its weights, and a lightweight residual adapter sits on top of the action chunk it predicts, correcting each action with the most recent tactile feedback just before execution, which is what lets the system regulate contact dynamics at run time.

Four properties follow, and the paper tests each of them. Reusability: the same adapter architecture pairs with different backbones, and only the adapter weights are backbone-specific, demonstrated here on π0, π0.5 and SmolVLA. Low entry cost: at most 50 visuotactile demonstrations per task suffice to fine-tune the backbone and train a 0.98M-parameter adapter, so no training budget beyond ordinary VLA fine-tuning is needed. Runtime reactivity: semantic reasoning and tactile adaptation run at different frequencies instead of competing. Consistent gains: mean task completion rises from 11.7% under task-level fine-tuning to 62.9%, and it rises on all four tasks and all three backbones.

Preliminaries: the VTLA Fusion Taxonomy and Slow-Fast Inference

The paper sorts existing VTLAs by where the tactile information is fused. The first family is unified VTL fusion: vision, touch and language are encoded separately and then handed to a single multimodal transformer. The second is sequential VL-T fusion: vision and language are encoded jointly first, and touch enters later, either through a mixture-of-experts stage before action prediction, by injection during action decoding or inside an action expert, or as a post-hoc correction of an already predicted action. The third family collects the remaining routes: multi-stage fusion, converting touch into continuous soft prompts that participate in VLA inference, and predicting stiffness with a VLM so the robot can execute compliantly. Figure 1 lays these strategies out side by side and marks VT-Bridge as the only one that achieves high-rate tactile adaptation without modifying the pretrained VLA architecture.

Figure 1: representative multimodal fusion strategies in existing VTLA models. VT-Bridge keeps the pretrained VLA backbone with no architectural modification while still supporting high-rate tactile adaptation.

Vision and force are complementary precisely because they behave so differently. Vision is high-dimensional, sampled relatively slowly, and carries global scene semantics; force is low-dimensional, sampled fast, and tightly coupled to local contact dynamics. RDP was the first to exploit that asymmetry with slow-fast inference: a slow policy predicts a latent action chunk from vision, and a fast decoder turns the newest tactile feedback into an executable action. Few VTLAs have adopted the pattern; T-Rex, FAVLA and UniTacVLA are the main examples. VT-Bridge combines slow-fast scheduling with a frozen backbone plus residual correction, and that pairing is what separates it from fusion-style VTLAs.

One more piece of background is needed to read the ablation later: pose deviation under impedance control. Every experiment in the paper runs under impedance control for interaction safety, and a compliant controller means the desired pose and the executed pose do not coincide once contact is established. Pressing a pump dispenser needs a peak interaction force of about 40 N, and the corresponding desired-to-executed offset reaches 4 to 5 cm. This physical fact dictates how the residual supervision signal must be built.

Method

VT-Bridge can be described in one line as a low-rate backbone plus a high-rate residual. The pretrained VLA backbone keeps its original structure and handles semantic understanding and coarse motion generation; the residual adapter runs at the robot's execution rate and corrects the predicted action chunk step by step. The two are time-sliced on the same GPU so that neither preempts the other.

The low-rate backbone. Given the current observation $\bm{o}$ and the language instruction $\bm{l}$, the pretrained VLA policy samples an action chunk:

$$\bm{A} \sim \pi^{\mathrm{VLA}}(\cdot \mid \bm{o}, \bm{l}) \tag{1}$$

where $\bm{A} = [\bm{a}_1, \bm{a}_2, \dots, \bm{a}_H]$ holds the next $H$ actions, with $H = 50$ in the experiments. This chunk encodes task intent and the coarse motion trend, and it is the nominal action sequence that the adapter later revises.

Structure of the high-rate residual adapter. The adapter is two parallel transformer encoders followed by an MLP decoding head. The action encoder consumes the embedded action chunk and emits an action feature $\bm{z}^a$ that represents the VLA's coarse motor intent and the overall trend of the whole chunk. That feature is computed once at the start of a chunk and reused for its entire execution, which is the reason the adapter can afford 0.7 ms per step. The force encoder extracts a temporal contact feature $\bm{z}^f_t$ from the most recent force window:

$$\bm{F}_t = [\bm{f}_{t-n+1}, \bm{f}_{t-n+2}, \dots, \bm{f}_t] \tag{2}$$

with window length $n$ (set to $n = 50$) and current execution step $t$. The two features are concatenated and decoded by the MLP into a residual action:

$$\bm{a}^{\mathrm{res}}_t = \mathrm{MLP}\left([\bm{z}^a; \bm{z}^f_t]\right) \tag{3}$$

Inference and execution. At deployment each action chunk is one scheduling unit. At the beginning of a cycle the backbone predicts the chunk $\bm{A}$, then goes idle, and the system enters an adapt-execute loop. At every execution step $t$ the adapter predicts the residual conditioned on the full chunk and the newest force window:

$$\bm{a}^{\mathrm{res}}_t = \pi^{\mathrm{res}}(\cdot \mid \bm{A}, \bm{F}_t) \tag{4}$$

The residual is added to the corresponding nominal action, and the sum is what actually reaches the robot:

$$\bm{a}^{\mathrm{exe}}_t = \bm{a}_t + \bm{a}^{\mathrm{res}}_t \tag{5}$$

The loop continues until the chunk is exhausted; the adapter then pauses and the backbone is called again for the next chunk. Because the two inference processes are staggered in time they never contend for the GPU. One chunk inference costs 153.1 ms on π0, 163.8 ms on π0.5 and 155.4 ms on SmolVLA, while the adapter needs 0.4 ms to encode the chunk and 0.7 ms to predict a single residual. That gap of two orders of magnitude is exactly what makes the slow-fast split affordable. Figure 2 draws the three interleaved timelines: observation, inference and execution.

Figure 2: temporal scheduling across the VLA backbone, the residual adapter and the robot executor. The backbone only infers at chunk boundaries, and the adapter fills the gaps at execution rate.

Training step 1, teleoperated demonstration collection. Demonstrations for the contact-rich tasks are collected with the bimanual teleoperation strategy from SharedAssembly: the interaction load on the follower arm is reflected back to the leader, so the operator feels contact transitions while teaching and can modulate applied force in real time. Each demonstration records camera streams, robot motion states with their corresponding desired commands, and tactile signals, namely the estimated external end-effector force and the external joint torques.

Step 2, backbone fine-tuning. The collected demonstrations adapt the pretrained backbone to the task. Camera observations, language instructions and robot states are converted into each backbone's own observation preprocessing and action representation, then the backbone is fine-tuned with LoRA for 30,000 steps at batch size 32, about 8 hours on four A100-SXM4 GPUs. The fine-tuned backbone produces the nominal action chunks used in the next step.

Step 3, building the residual dataset. The fine-tuned backbone is replayed on the recorded observations to generate nominal action chunks. After temporal alignment, the residual target at each execution step $k$ is

$$\bm{a}^{\mathrm{res}}_k = \bm{a}^{\mathrm{desired}}_k - \bm{a}_k \tag{6}$$

where $\bm{a}^{\mathrm{desired}}_k$ is the controller command recorded during teleoperation and $\bm{a}_k$ is the backbone's prediction. Supervising with the desired command rather than the executed pose preserves the operator's true motor intent when contact is present; the ablation quantifies what that choice buys. Every residual target is paired with its nominal chunk $\bm{A}$ and contact feedback window $\bm{F}_k$ as adapter input. The action representation follows π0 in supporting both Cartesian and joint space: Cartesian actions pair with the estimated end-effector force, joint actions with the external joint torques, so action and tactile representations stay consistent. All non-contact samples additionally have their residual target set to zero, which pushes the adapter to leave the backbone's decision intact in free space and spend its capacity after contact is established.

Step 4, adapter training. The fine-tuned VLA backbone is frozen and only the 0.98M-parameter adapter is optimized on the residual dataset. This decoupling preserves the nominal task behaviour of the backbone while letting the contact-conditioned correction be learned on its own terms.

flowchart TD
  O["Observation o and instruction l"] --> VLA["VLA backbone, low-rate inference, 153-164 ms"]
  VLA --> CHUNK["Action chunk A with H=50 actions"]
  CHUNK --> AE["Action encoder emits z^a, computed once per chunk"]
  AE --> MLP["MLP decodes residual a^res_t in 0.7 ms"]
  FW["Newest force window F_t of length n=50"] --> FE["Force encoder emits z^f_t"]
  FE --> MLP
  MLP --> ADD["a^exe_t = a_t + a^res_t"]
  ADD --> ROBOT["Robot executes at 30 Hz"]
  ROBOT --> DONE{"Chunk exhausted?"}
  DONE -- no --> FW
  DONE -- yes --> VLA

Figure A: the slow-fast interleaved inference loop of VT-Bridge, drawn from Section III of the paper.

flowchart LR
  S1["Bimanual teleoperation: 50 demos per task, recording cameras, states, desired commands, forces and joint torques"] --> S2["LoRA fine-tune the VLA backbone: 30k steps, batch 32, about 8 h on 4 A100"]
  S2 --> S3["Replay the backbone on recorded observations to get nominal chunks"]
  S3 --> S4["Residual target a^res = a^desired - a^pred, zeroed for non-contact samples"]
  S4 --> S5["Freeze the backbone, train only the 0.98M residual adapter"]

Figure B: the four-step training pipeline of VT-Bridge, drawn from Section IV of the paper.

Experimental Results

The platform is a Franka Emika Panda arm with two Intel RealSense D435i cameras. The robot controller runs on a NUC with an i7-10700, while the VLA and the residual policy run on a separate workstation with an RTX 3090. The four evaluation tasks cover four distinct physical interaction regimes: Pump Bottle (pressing), Clean Whiteboard (sustained surface contact), Insert Plug (high-precision insertion) and Flip Rocker Switch (a discrete force-triggered event). Each task has 50 demonstrations, each backbone-method combination is run for 20 independent trials per task, and a trial ends when the strict success criterion is met or the 90 s time limit expires. Table 1 collects the key configuration.

Table 1: experiment configuration summary (paper Table I).

Evaluation uses a two-tier success criterion. The strict criterion requires the task to be completed end to end; the loose criterion only requires the policy to reach the task-relevant interaction state and establish the expected contact. Separating the two distinguishes failures that never reached contact from failures that reached contact and then did not finish. The paper reports strict success rate $\mathrm{SR}_{\mathrm{strict}} = N_{\mathrm{strict}} / N$, loose success rate $\mathrm{SR}_{\mathrm{loose}} = N_{\mathrm{loose}} / N$, and the conditional completion rate

$$\rho_{\mathrm{sl}} = N_{\mathrm{strict}} / N_{\mathrm{loose}} \tag{7}$$

$\rho_{\mathrm{sl}}$ measures the probability of finishing the task once a near-success state has been reached. It therefore aims the evaluation at the contact-rich phase itself and suppresses the differences between backbones in task understanding and coarse motion generation.

Q1, overall performance. Table 2 gives strict success rates. VT-Bridge is the best method in all 12 task-backbone cells without exception. Against fine-tuning the same backbone on the same demonstrations, the mean strict success rate goes from 11.7% to 62.9%, an absolute gain of 51.2 percentage points. The informative comparison is VLA-Touch, which also follows a fine-tuned-backbone-plus-tactile-correction route yet averages only 7.5% to 20.0%. That gap says the gain comes from how the residual target is constructed and from the high-rate schedule, not from the mere presence of a tactile signal.

Table 2: strict success rate in percent (paper Table III).

Figure 3: mean strict success rate across the three backbones. The advantage of VT-Bridge over both the fine-tuned baseline and VLA-Touch holds on every backbone.

Q2, the last-inch bottleneck. A large share of failures is not a wrong task intent. The policy often gets close to finishing and then misses the final contact-sensitive step: the pump is pressed but the stroke is too short, the plug enters the socket but is not fully seated. Such executions are geometrically near success and functionally a failure. The conditional completion rates in Table 3 quantify exactly this ability. VT-Bridge is again highest in every cell, with a macro average 58.8 percentage points above the fine-tuned backbone. Pump Bottle is the sharpest case: every baseline-backbone combination scores 0%, meaning not one trial ever converted reaching the pump into dispensing liquid, while VT-Bridge averages 90.6%. On Insert Plug the fine-tuned backbones score 0% on all three and VLA-Touch reaches 6.3% in a single cell, against 85.9% for VT-Bridge. The z-axis interaction profiles in Figure 4 show the mechanism: demonstrations and successful VT-Bridge trials both exhibit a force peak near 40 N during the contact phase, whereas the force trace of the ablated policy stays almost flat there, so it never builds the contact force the task requires.

Table 3: conditional completion rate ρ_sl in percent (paper Table IV).

Figure 4: interaction profiles along the z axis on Pump Bottle for demonstration, VT-Bridge and the ablated policy. The top row shows desired versus actual pose, the bottom row contact force; the contact phase is shaded.

Figure 5: initial states of the four contact-rich manipulation tasks, with examples that satisfy the loose and the strict criteria.

Q3, desired or executed poses as the residual target. The default setting builds the residual target from the difference between the desired pose recorded in the demonstration and the backbone prediction; the ablation swaps the desired pose for the actually executed pose and changes nothing else. Figure 6 shows the desired pose winning consistently across every task-backbone combination, with the mean strict success rate rising from 12.1% to 62.9%. The explanation ties straight back to impedance control. Pressing the pump needs a peak force of about 40 N, and under a compliant controller the desired-to-executed offset reaches 4 to 5 cm. Supervising with the desired pose writes that force-dependent offset explicitly into the training data; supervising with the executed pose throws it away. Even if the ablated policy reproduces the recorded executed pose exactly, sending that pose back as a new desired command produces a fresh tracking gap under contact, so the robot reproduces neither the demonstrated pose nor the demonstrated contact force. Insert Plug has a comparable peak force near 40 N and shows the same pattern. Flip Rocker Switch needs only about 7 N, the compliant offset is small, and the ablation does relatively better there, yet the desired pose still delivers an 82% relative improvement.

Figure 6: ablation on residual target construction. Hatched bars are the executed pose (ablation), solid bars the desired pose (this paper).

Limitations

First, stated by the authors in the conclusion: future work will focus on strengthening the representation and adaptation capacity of the residual adapter for larger and more consistent gains. That is an admission that the 0.98M adapter is still the bottleneck on some tasks, and the numbers agree. On Clean Whiteboard VT-Bridge reaches only 20% to 45% strict success, which is the best result in the table but far from deployable.

Second, also author-stated: a safety-versus-precision tradeoff. Raising controller stiffness would shrink the desired-to-executed offset, but it would amplify the risk of excessive interaction force. The paper therefore compensates the offset through the supervision signal under compliant control rather than switching to a stiffer controller, which moves the burden onto the quality of the residual dataset.

Third, from the experimental design: every result comes from a single Franka Panda platform and four tabletop tasks, and the tactile signal is limited to estimated external force and external joint torques. No tactile array or tactile image is used, so the extensibility of the method to richer tactile modalities is untested. A related cost is that adapter weights are trained per backbone, so switching backbones means retraining.

Fourth, statistical power. Each combination gets 20 trials, and several $\rho_{\mathrm{sl}}$ entries in Table 3 rest on denominators as small as four or five, for instance VLA-Touch on Pump Bottle with π0 at 0/5. Single-cell readings at that scale carry wide confidence intervals, and the paper does not report variance across seeds.

Summary and Outlook

The contribution of VT-Bridge is not a new model but an affordable engineering path: tactile adaptation stops being "retrain a VTLA" and becomes "train a sub-million-parameter residual head on top of a frozen or LoRA-tuned VLA". Slow-fast interleaved scheduling resolves the coexistence of high-rate touch and low-rate semantic inference on one GPU; the desired-pose residual resolves the distortion of supervision under impedance control; and keeping the architecture fixed while swapping weights per backbone establishes generality across π0, π0.5 and SmolVLA.

The budget is concrete and reusable for any team that wants touch inside an existing VLA stack: 50 demonstrations per task, about 8 hours of backbone fine-tuning, a 0.98M-parameter adapter, in exchange for a mean strict success rate going from 11.7% to 62.9% and a 58.8 percentage-point gain in conditional completion. If follow-up work strengthens the adapter's representation and extends the framework to tactile arrays and other robot platforms, this bridging route is a serious candidate to become the default way of building VTLAs.

Golden Quote

"Being geometrically close to success is not the same as finishing: the last inch between pressing the pump and dispensing the liquid is exactly where tactile feedback pays for itself."