PAPER DEEP DIVE
$N_0$-TWAM: Scaling Tactile-Native World-Action Model for Contact-Rich Manipulation
We present $N_0$-TWAM, a tactile-native world-action model for contact-rich manipulation that predicts both future vision and future contact. To our knowledge, it is the first tactile world-action model trained at large scale, and it shows strong capability on contact-rich tasks. We pre-train $N_0$-TWAM at large scale with visuo-tactile joint training over tactile-rich demonstrations spanning six embodiments and 450 tasks. We use NeoForce, a unified force-based tactile representation, to form a physically grounded contact signal that conditions action generation. To improve long-horizon and multi-stage manipulation, we introduce tactile contact events for task staging and advance through them during execution. For real-time efficiency, we adopt an asymmetric Mixture-of-Transformers architecture that pairs a full-width expert for video prediction with slim experts for downstream action and tactile prediction. Evaluations on both real and simulated benchmarks justify the capabilities of $N_0$-TWAM across a range of contact-rich tasks, and demonstrate the benefit of data scaling for precise tactile and action prediction. In summary, $N_0$-TWAM endows a world-action model with predictive capabilities to foresee vision, touch and action, building a solid foundation for fine-grained manipulation on open contact-rich tasks. The codebase and model checkpoints will be made publicly available to foster further research and development in tactile-enabled robotic manipulation.
1. Overview
Contact-rich manipulation requires tactile sensing, but existing VLA models lack tactile-native world modeling. This paper presents N₀-TWAM (Tactile-Native World-Action Model), a scaling tactile-native world-action model using a unified diffusion Transformer for simultaneous visual world modeling, tactile prediction, and action generation.
N₀-TWAM is pretrained on large-scale self-collected data using a Mixture-of-Transformers (MoT) backbone processing multi-modal streams (multi-view RGB + tactile), trained with conditional flow-matching objectives. Tactile modeling has two aspects: (1) predicting future tactile—from current observation+action to future tactile images; (2) conditioning on observed tactile. A tactile punctuation mechanism enables long-horizon execution: using tactile events to segment demonstrations and advance sub-tasks at inference. Evaluated on UniVTAC and NeoSim simulation plus real robots, N₀-TWAM significantly outperforms baselines on contact-rich tasks.
2. Method
2.1 Model Architecture
Conditional flow-matching objective training a Mixture-of-Transformers backbone with parameter-efficient experts and diffusion-forcing cascade. Multi-view RGB and tactile streams are encoded separately, interacting through joint self-attention masks (V/T blocks processed independently then cross-stream exchange).
2.2 Modeling Touch
Future touch prediction: from current observation+action to future tactile images. Conditioning on observed touch: using observed tactile as conditioning input. Staged training: pretrain NeoForce force-space representation first, then integrate tactile modeling.
2.3 Tactile-Aware Execution
Streaming deployment and real-time inference. Tactile punctuation mechanism: segment demonstrations by tactile contact events—advance to next sub-task when key contact events are detected, enabling long-horizon autonomous execution.
3. Overall Architecture
flowchart TB
RGB[Multi-view RGB] --> MoT[Mixture-of-Transformers]
TAC[Tactile Stream] --> MoT
ACT[Action] --> MoT
MoT --> PRED_FUTURE[Predict Future Visual+Tactile]
MoT --> GEN_ACTION[Generate Action]
PRED_FUTURE --> WORLD[World Model]
GEN_ACTION --> EXEC[Execution]
EXEC --> CONTACT[Contact Event Detection
Tactile Punctuation]
CONTACT --> |Advance Sub-task| GEN_ACTION
style MoT fill:#4f9eff,color:#fff
style CONTACT fill:#ff6b6b,color:#fff
style EXEC fill:#51cf66,color:#fff
4. Experimental Results
Evaluated on UniVTAC simulation, NeoSim simulation, and real robots. N₀-TWAM significantly outperforms baselines on contact-rich tasks. Task-level gain analysis shows tactile prediction and tactile punctuation are key. Ablation studies validate component contributions. Generalization experiments verify cross-task and cross-sensor transfer.
5.
Autoregressive factorization
$$ p_{\theta}(x^{v}_{1:K},x^{t}_{1:K},x^{a}_{1:K}|c)=\prod_{k=1}^{K}p_{\theta}(x^{v}_{k},x^{t}_{k}|X_{<k},c)\cdot p_{\theta}(x^{a}_{k}|x^{v}_{k},x^{t}_{k},X_{<k},c) $$
Noise interpolation
$$ \hat{x}^{m}_{k,j}=(1-\sigma^{m}_{k,j})\,x^{m}_{k,j}+\sigma^{m}_{k,j}\,\epsilon^{m}_{k,j} $$
History context
$$ X_{<k}=(x^{v}_{<k},x^{t}_{<k},x^{a}_{<k}) $$
Noised sample set
$$ \hat{X}=\{\hat{x}^{m}_{k,j}\} $$
Jacobian dimensions
$$ J_{v}=J_{t}=J_{a}=2 $$
| Parameter | Value | Description |
|---|---|---|
| Learning rate | 1e-4 | Adam optimizer |
| Batch size | 256 | Training batch |
| Discount $\gamma$ | 0.99 | Reward discount |
| Epochs | 5000 | Training iterations |
| Parameter | Value |
|---|---|
| Optimizer | Adam |
| Learning rate | 1e-4 |
| Batch size | 256 |
| Training steps | 5000 |
N₀-TWAM is a tactile-native world-action model using a unified MoT diffusion Transformer for simultaneous visual world modeling, tactile prediction, and action generation. Tactile punctuation naturally segments and advances long-horizon tasks via contact events. Significantly outperforms baselines on contact-rich manipulation, demonstrating the value of tactile as a native stream in world modeling.
SOURCE LINKS
