τ: Learning Touch-Augmented Vision-Language-Action Models from Future Visual Supervision

1. Overview: τ Touch-Augmented Vision-Language-Action Models

τ (tau), proposed by Ning Cheng, Jinan Xu, Wanlin Li et al. in July 2026, is a touch-augmented vision-language-action (VLA) framework that learns an action-conditioned spatiotemporal tactile representation from future visual supervision inspired by JEPA, fusing it with vision-language features for action generation. Existing VLA models rely on vision as the primary modality, limiting their capability in contact-rich manipulation where success depends on capturing physical interaction dynamics. Tactile sensing naturally provides contact states, force distribution, and deformation information, but learning informative tactile representations and effectively adapting them to pretrained VLA models is challenging at both data and modeling levels: limited task-specific demonstrations constrain representation quality while large-scale pretraining is costly; existing methods focus on instantaneous contact states or model temporal dynamics using 6D wrench sequences, leaving high-dimensional tactile signals underexplored. τ operates future visual supervision in latent space, used only during training with no deployment overhead. The TacAura dataset of synchronized vision, proprioception, and vision-based tactile signals across four contact-rich tasks is introduced. Experiments show τ achieves an average success rate of 71.25%, surpassing the strongest baseline by 41.25 percentage points.

2. Core Problem: The Dual Challenge of Tactile Representation Learning and VLA Adaptation

Vision-Language-Action models have established a new paradigm for general-purpose robot manipulation through large-scale multimodal pretraining, but their reliance on vision as the primary modality limits contact-rich manipulation—successful execution depends on physical interaction dynamics that tactile sensing naturally provides (contact states, force distribution, deformation patterns). Existing methods incorporate touch through specialized fusion architectures, focusing on instantaneous contact states; even when modeling temporal tactile dynamics, they use 6D wrench sequences without fully exploring high-dimensional visual tactile signals. More critically, adapting tactile representations to the pretrained VLA's multimodal embedding space faces data and modeling challenges: limited demonstrations constrain representation quality while large-scale pretraining is costly. τ's core insight: leverage correspondences between actions and future visual changes to learn interaction-aware tactile representations through JEPA-style self-supervision, enabling the policy to more effectively integrate tactile cues with vision-language features and generate contact-aware actions.

3. Method: τ Model Architecture

3.1 Foundational VLA Backbone

τ is built upon the pretrained VLA model $\pi_{0.5}$. At each time step $t$, the model receives observation $o_t = \{\mathcal{I}_t, \tilde{\ell}_t\}$, where $\mathcal{I}_t = \{\mathcal{I}_t^1, \ldots, \mathcal{I}_t^N\}$ is $N$ RGB images and $\tilde{\ell}_t = [\ell_t; q_t]$ is the language instruction formed by concatenating task description $\ell_t$ with proprioceptive state $q_t$. RGB images are encoded by a vision encoder into visual tokens, $\tilde{\ell}_t$ is tokenized, multimodal tokens are fused by the LLM, and an action expert predicts future action chunks via conditional flow matching:

$$\hat{a}_{t:t+H} = [\hat{a}_t, \hat{a}_{t+1}, \ldots, \hat{a}_{t+H-1}] \tag{1}$$

3.2 Tactile Encoding and Adaptation Module

The observation is extended to $\tilde{o}_t = \{\mathcal{I}_t, \tilde{\ell}_t, \mathcal{T}_t\}$, where $\mathcal{T}_t = \{\mathcal{T}_t^L, \mathcal{T}_t^R\}$ are left and right tactile sensor signals. Tactile signals are first processed by a touch encoder $E_{\text{tou}}$ to extract features:

$$z_t^{\text{touch}} = E_{\text{tou}}(\mathcal{T}_t) \tag{2}$$

encoding normal and shear deformations. Since the pretrained VLA backbone operates in a multimodal embedding space learned from visual and textual data, tactile features cannot be directly integrated. A learnable touch adapter $A_{\text{tou}}$ projects them into the same latent space:

$$\mathcal{Z}_t^{\text{touch}} = A_{\text{tou}}(z_t^{\text{touch}}) \tag{3}$$

The adapted tactile tokens are concatenated with visual and textual tokens forming $\mathcal{Z}_t = [\mathcal{Z}_t^{\text{vision}}; \mathcal{Z}_t^{\text{language}}; \mathcal{Z}_t^{\text{touch}}]$, processed by the LLM, then used by the action expert to predict action chunks.

3.3 JEPA-style Predictive Self-Supervised Branch

The adapted tactile tokens $\mathcal{Z}_t^{\text{touch}}$ and action tokens $\mathcal{Z}_{t:t+H}^{\text{action}}$ are concatenated and fed into a predictor network $P(\cdot)$ to predict future visual latent representations:

$$\{\hat{z}_{t+\Delta_k}^{\text{vision}}\}_{k=1}^K = P(\mathcal{Z}_t^{\text{touch}}, \mathcal{Z}_{t:t+H}^{\text{action}}) \tag{4}$$

where $\Delta_k$ are predefined temporal offsets. Action tokens are obtained by encoding vectorized action sequences: $\mathcal{Z}_{t:t+H}^{\text{action}} = E_{\text{act}}(\text{vec}(a_{t:t+H}))$. The prediction target is the future visual observation's latent representation extracted by the pretrained VLA's vision encoder $E_{\text{vis}}$: $\{z_{t+\Delta_k}^{\text{vision}}\}_{k=1}^K = E_{\text{vis}}(\{\mathcal{I}_{t+\Delta_k}\}_{k=1}^K)$. The predictor is a 3-layer MLP with GELU activations. This supervision is used only during training and removed at inference, adding no deployment overhead.

3.4 Training Strategy

τ is trained end-to-end by jointly optimizing a supervised imitation learning objective and a predictive self-supervised objective. The total training loss is:

$$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{cfm}} + \lambda \mathcal{L}_{\text{jepa}} \tag{5}$$

where $\mathcal{L}_{\text{cfm}}$ is the conditional flow matching imitation loss, $\mathcal{L}_{\text{jepa}}$ is the JEPA predictive self-supervised loss, and $\lambda$ is a balancing weight. Imitation learning uses conditional flow matching: random Gaussian noise $\epsilon \sim \mathcal{N}(0, I)$ is sampled, noisy actions constructed as $a_{t:t+H}^s = s \cdot a_{t:t+H} + (1-s) \cdot \epsilon$, $s \in [0,1]$. Trained with 100 expert demos per task, batch size 32, action horizon 32, cosine learning rate schedule (10000 warmup, peak 5e-5), 30000 steps, 2x A800 80GB GPUs.

4. Experiments

4.1 Main Results

Evaluated on plug insertion, USB insertion, stamp press, and whiteboard erasing—four contact-rich tasks, 20 trials per model. Among baselines, ForceVLA achieves the highest average success rate (30.00%), $\pi_{0.5}$ at 28.75%, ForceFlow at 23.75%, $\pi_0$ at 20.00%. All three τ variants substantially outperform: τ-Wrist achieves 71.25% (surpassing the strongest baseline by 41.25pp), τ-Front at 68.75%, τ-DualView at 57.50%.

Method	Plug Insert	USB Insert	Stamp Press	Whiteboard Erase	Avg
π₀	—	—	—	—	20.00%
π₀.₅	—	—	—	—	28.75%
ForceVLA	0%	—	—	—	30.00%
τ-Wrist	65%	50%	90%	95%	71.25%

Baselines are primarily limited in final task completion rather than initial grasping or contact: ForceVLA achieves 85% alignment on plug insertion but 0% insertion completion; ForceFlow reaches 95% contact on whiteboard erasing but only 50% wiping completion. τ variants maintain high success rates from intermediate stages to final completion, exceeding the strongest baseline by 45, 30, 20, and 45 percentage points respectively.

4.2 Ablation Study

Ablation Configuration	Avg Success Rate	Drop
Full τ-Wrist	71.25%	—
Remove action sequence conditioning	58.75%	-12.50pp
Remove predictive self-supervision	51.25%	-20.00pp
Remove tactile encoding & adaptation	28.75%	-42.50pp

The tactile encoding and adaptation module is most critical (-42.50pp), with removal also substantially reducing intermediate-stage performance; predictive self-supervision is second (-20.00pp), mainly affecting final execution; action sequence conditioning has larger impact on tasks relying on precise force control. Whiteboard erasing drops 35pp when removing predictive self-supervision, indicating this objective helps capture temporal evolution of tactile observations during sustained contact.

5. Limitations

• Limited data scale: trained with only 100 demonstrations per task; whether larger-scale data brings further improvements remains to be verified.
• Hardware dependency: relies on left and right vision-based tactile sensors mounted on the gripper; robots without tactile sensors cannot directly use this approach.
• Task scope: only four contact-rich tasks are evaluated; generalization to more scenarios (e.g., deformable manipulation, assembly) needs expansion.

6. Conclusion

τ extends pretrained VLA models into a touch-augmented VTLA framework through a tactile encoding and adaptation module and a JEPA-style predictive self-supervised branch. The touch encoder extracts high-dimensional tactile features projected by an adapter into the VLA embedding space for efficient multimodal fusion; the predictive branch uses action-conditioned future visual latent prediction as a training regularizer, removed at inference with no deployment cost. The TacAura dataset covers four contact-rich tasks with synchronized vision, proprioception, and vision-based tactile signals. τ-Wrist achieves a 71.25% average success rate, surpassing the strongest baseline by 41.25 percentage points. Ablation reveals component importance ranking: tactile encoding & adaptation > predictive self-supervision > action sequence conditioning. The core insight: the value of touch lies not in instantaneous contact detection but in understanding "how actions change future visuals"—the JEPA predictive branch lets tactile representations learn to foresee the consequences of contact, upgrading the policy from passive perception to proactive anticipatory control.

flowchart TD
    A["Multi-view RGB images I_t"] --> B["Vision encoder -> visual tokens Z_vision"]
    C["Language instruction + proprioception"] --> D["Word embedding -> language tokens Z_language"]
    E["Left/right tactile signals T_t"] --> F["Touch encoder E_tou -> z_touch"]
    F --> G["Touch adapter A_tou -> Z_touch"]
    B --> H["Unified tokens Z = [vision; language; touch]"]
    D --> H
    G --> H
    H --> I["LLM fusion"]
    I --> J["Action expert: conditional flow matching -> action chunk a_hat"]
    G --> K["JEPA predictive branch"]
    L["Action tokens Z_action"] --> K
    K --> M["Predictor P -> future visual latent z_hat_vision"]
    N["Future images I_{t+delta}"] --> O["Vision encoder E_vis -> target z_vision"]
    M --> P["Loss: ||z_hat - z_vision||"]
    O --> P
    P --> Q["Training regularizer, removed at inference"]