ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation

ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation

Paper: ViTacWorld: Scaling Visuo-Tactile World Models for Contact-Rich Robot Manipulation

Authors: Yunao Huang, Shiyu Sang, Haotao Lu, Suting Ni, Shijie Wu, Ziyang Guo, Ye Shi, Jingya Wang

Affiliation: ShanghaiTech University

Links: arXiv:2607.22530 · Project Page

---

One-Sentence Summary

ViTacWorld is the first action-conditioned visuo-tactile world model for robot visuo-tactile-action trajectory generation and policy evaluation — leveraging public real tactile datasets and constructed simulation to scale visuo-tactile-action data, generating temporally aligned visual and tactile observations via stream-aware DiT with cross-stream exchange, pretrained at scale then finetuned with real policy rollouts, improving tactile policy average success rate from 42.5% to 67.5% across four contact-rich tasks.

---

Background and Motivation

Contact-rich manipulation tasks (insertion, peeling, plugging) are central to real-world robot manipulation but difficult with vision alone due to occlusion, constrained interfaces, and subtle force-dependent dynamics. Tactile sensing directly captures physical interaction at contact, making it a key modality rather than supplementary. However, scaling visuo-tactile robot learning remains difficult: task-relevant visuo-tactile-action trajectories are hard to obtain at scale. Real-robot teleoperation provides faithful data but is expensive and slow — GelSight-style sensors are costly, have limited lifetime, and contact-rich demonstrations are hard to collect reliably.

Existing visuo-tactile learning methods mainly rely on real-robot teleoperation, limited by sensor cost, lifetime, and task diversity. Simulation systems like TacSL, TacEx, and UniVTAC can generate tactile data, but how to effectively combine real and simulated data to scale visuo-tactile world models remains open. ViTacWorld's core insight: tactile signals are directly grounded in physical contact and may exhibit a smaller sim-to-real gap than purely visual observations. Simulation offers a scalable alternative.

Figure 1: Overview of ViTacWorld. Right: scaling training pipeline; left: visuo-tactile rollout generation and policy evaluation.

---

Method

1. Problem Formulation

Given current observations $o_t$ (main camera, wrist camera, tactile images) and action sequence $u_{t:t+H-1}$, the world model predicts future visuo-tactile rollouts. Each observation stream is encoded by a VAE encoder into latent tokens, assembled as a unified visuo-tactile sequence.

2. Stream-Aware DiT with Cross-Stream Exchange

ViTacWorld extends a pretrained action-conditioned robot video world model to visuo-tactile generation. Stream identity embeddings $e^v$ distinguish visual and tactile streams, projected into the AdaLN modulation path. Each block $b$ first applies stream-aware self-attention within each stream:

$$ \tilde{Z}_{b}^{v}=\mathrm{SelfAttn}_{v}\left(\mathrm{AdaLN}(Z_{b-1}^{v};c_{b}+P(e^{v}))\right),\quad v\in\mathcal{V} $$

Then an explicit cross-view attention module exchanges information across streams:

$$ Z_{b}^{v}=\mathrm{CrossViewAttn}_{v}\left(\tilde{Z}_{b}^{v},\{\tilde{Z}_{b}^{v'}\}_{v'\neq v}\right),\quad v\in\mathcal{V} $$

This avoids uncontrolled mixing between camera and tactile tokens during ordinary self-attention while allowing the tactile stream to exchange contact information with visual streams.

3. Training Objective

ViTacWorld is trained with the latent denoising objective applied to future visual and tactile latents:

$$ \mathcal{L}_{\mathrm{wm}}=\mathrm{E}_{z_{0},\sigma}\left[\left\|D_{\theta}(z_{\sigma},\sigma,o_{t},u_{t:t+H-1},m)-z_{0}\right\|_{2}^{2}\right] $$

where $z_0$ is the target latent rollout, $z_\sigma$ its noisy version, $m$ the view-presence mask. Action conditioning is embedded and injected via AdaLN, repeated across visual and tactile streams:

$$ c_b = \text{Embed}(u_{t:t+H-1}) + \text{PosEmb}(t), \quad c_b^v = c_b + P(e^v) $$

4. Pretraining and Finetuning

First pretrained with large-scale real and simulated visuo-tactile trajectories, then finetuned with real policy rollouts. Generated rollouts are filtered by task success and visuo-tactile plausibility; 200 high-quality successful rollouts are selected and merged with expert demonstrations:

$$ \mathcal{D}_{\text{aug}} = \mathcal{D}_{\text{expert}} \cup \{\tau_i \in \mathcal{D}_{\text{gen}} \mid \text{success}(\tau_i) \land \text{plausible}(\tau_i)\}_{i=1}^{200} $$

flowchart TD
    A["Public real tactile datasets"] --> C["Large-scale pretraining
real + simulated trajectories"]
    B["Task-aligned simulation"] --> C
    C --> D["ViTacWorld world model
stream-aware DiT + cross-stream"]
    E["Real policy rollouts"] --> F["Finetune
match downstream behavior"]
    D --> F
    F --> G["Generate visuo-tactile rollouts"]
    G --> H["Filter: success + plausibility"]
    H --> I["200 high-quality rollouts"]
    I --> J["Merge with expert demos"]
    J --> K["Augment downstream policy training"]
    K --> L["Policy evaluation
action-conditioned prediction"]
    style D fill:#e1f5fe
    style F fill:#fff3e0
    style K fill:#e8f5e9

---

Experiments

Policy Improvement

Figure 2: Real-robot success rates table. ViTacWorld rollout augmentation consistently improves performance.

Table 1: Real-robot success rates (%) — ViTacWorld rollout augmentation consistently improves performance

Data Source	Method	Charger	Peeling	U-Block	Cuboid	Avg.
Expert only	ACT+tactile	0	0	30	30	15.0
Expert only	π0.5	10	30	60	40	35.0
Expert only	π0.5+tactile	20	40	70	40	42.5
Expert+ViTacWorld	ACT+tactile	10	20	40	40	27.5
Expert+ViTacWorld	π0.5	30	60	60	40	47.5
Expert+ViTacWorld	π0.5+tactile	40	80	80	70	67.5

ViTacWorld rollout augmentation improves tactile π0.5 policy average success from 42.5% to 67.5% (+25pp). Gains appear across both vision-only and tactile policies. Stronger tactile improvement indicates generated tactile observations contain contact-relevant information. Qualitatively, augmented policies show better grasp localization and robustness to initial pose variations, with tactile policies performing corrective motions using contact feedback.

Figure 3: Qualitative comparison of real-robot rollouts. Top: expert only; bottom: expert+ViTacWorld.

World Model Quality Analysis

Table 2: World-model prediction quality on held-out validation clips — pretraining and task-aligned simulation both improve generation

View	Variant	PSNR↑	SSIM↑	LPIPS↓
Main	w/o pretraining	22.718	0.7859	0.0781
Main	w/o task-aligned sim	23.128	0.8011	0.0687
Main	Full ViTacWorld	24.258	0.8286	0.0513
Tactile	w/o pretraining	34.967	0.9204	0.0211
Tactile	Full ViTacWorld	35.225	0.9318	0.0157

Figure 4: Qualitative world-model predictions on held-out validation clips. Full model produces clearer rollouts with better preserved geometry.

Figure 5: Policy evaluation with ViTacWorld table.

---

Limitations

1. Sim-to-real gap: While tactile signals have a smaller sim-to-real gap, simulated tactile may not fully match real sensor noise and deformation characteristics, affecting generated data fidelity.
2. Task-aligned simulation dependency: Task-aligned simulation environments require manual construction per real scene, limiting fully automated scaling to new tasks.
3. Generation filtering cost: Filtering generated rollouts (task success and visuo-tactile plausibility) requires additional evaluation pipelines, and filtering criteria may introduce bias.

---

Conclusion and Outlook

ViTacWorld is the first action-conditioned visuo-tactile world model for robot visuo-tactile-action trajectory generation and policy evaluation. Through stream-aware DiT with cross-stream exchange, it generates temporally aligned visual and tactile observations, leveraging public real data and task-aligned simulation to scale training data. It improves tactile policy average success from 42.5% to 67.5% across four contact-rich tasks and supports action-conditioned policy evaluation, demonstrating that visuo-tactile world models can generate policy-improvement contact-rich data.

Golden quote: "Tactile signals are directly grounded in physical contact and may exhibit a smaller sim-to-real gap than purely visual observations — making simulated tactile data a viable path for scaling visuo-tactile robot learning."