FeelWorld: Visuo-Tactile World Model for Hierarchical Contact Prediction and Planning

1. Overview: A Hierarchical Visuo-Tactile World Model with Layered Contact Prediction

FeelWorld, proposed by Wenxuan Ma, Chaofan Zhang et al. from the Institute of Automation CAS, Imprintx Robotics, and BAAI in July 2026, is a hierarchical visuo-tactile world model that jointly predicts future visual latents and three tactile states. Humans plan physical interactions by imagining the outcomes of candidate actions, yet existing visual world models mainly capture appearance dynamics while ignoring the tactile states that govern contact-rich interactions, potentially producing visually plausible but physically implausible imagined futures. FeelWorld organizes tactile states hierarchically into: contact state, a 3D tactile latent encoding force-related information, and slip state, all jointly predicted and explicitly supervised by a shared latent dynamics model. To prevent irrelevant tactile signals during free-space motion from degrading visual prediction, it introduces contact-gated asymmetric attention—maintaining a pure-visual prediction path before contact and enabling joint visuo-tactile dynamics during contact. The model is also trained with autoregressive rollout and context noise injection to improve robustness against compounding errors. The predicted contact and slip states support contact-aware CEM planning. On chip grasping, fruit grasping, and USB insertion, FeelWorld reduces 10-step LPIPS from 0.084 to 0.058, with 80-step autoregressive rollout LPIPS 61% lower than the visual baseline, achieving a mean zero-shot planning success rate of 81.7%.

2. Core Problem: Visual World Models Ignore Tactile Dynamics

Contact-rich manipulation requires robots to reason about physical interaction. Vision provides global scene context but only indirectly reflects contact physics; tactile sensing directly measures force distribution, local deformation, and incipient slip, often capturing events that vision misses. Existing visual world models primarily capture appearance dynamics and ignore tactile states, which can produce visually plausible but physically implausible imagined futures. The challenge is twofold: how to incorporate tactile sensing into the world model without letting irrelevant tactile signals during free-space motion degrade visual prediction, and how to use tactile predictions to guide planning without prematurely optimizing unreachable tactile goals before contact occurs.

3. Method: FeelWorld Architecture

3.1 Problem Formulation

FeelWorld learns an action-conditioned dynamics model that jointly predicts future visual latents and three tactile states. Visual input consists of $V$ viewpoint images $I_t$; a frozen visual encoder produces $o_t = E_v(I_t)$, and a frozen tactile encoder produces $h_t = E_\tau(P_t)$. The tactile state tuple $\tau_t = (c_t, h_t, s_t)$ contains contact state, 3D tactile latent, and slip state. Fused visuo-tactile latents $z_t = [z_t^o; z_t^\tau]$ are modulated by a contact gate $g_t$. Given proprioception $q_t$ and a 7-dimensional action $a_t$, the dynamics model predicts:

$$\hat{z}_{t+1} = [\hat{z}_{t+1}^o; \hat{z}_{t+1}^\tau] = \mathcal{W}_\theta(z_t, g_t, q_t, a_t) \tag{1}$$

The visual component decodes to $\hat{o}_{t+1}$ and the tactile component to $\hat{\tau}_{t+1} = (\hat{c}_{t+1}, \hat{h}_{t+1}, \hat{s}_{t+1})$, where $\hat{c}, \hat{s} \in [0,1]$.

3.2 Hierarchical Tactile Modeling

The 3D tactile point cloud $P_t$ is encoded by a pretrained encoder: $h_t = E_\tau(P_t) \in \mathbb{R}^{d_\tau}$ (Eq. 2), encoding contact area, pressure distribution, and principal deformation directions (LayerNorm is omitted to preserve contact-intensity magnitude). The tactile prediction head decodes: $\hat{h}_{t+1} = H_\tau(\hat{z}_{t+1}^\tau)$ (Eq. 3).

Level 1 Contact State: a two-layer MLP contact head predicts the binary contact probability $\hat{c}_{t+1} = H_c(\hat{z}_{t+1}^\tau) \in [0,1]$ (Eq. 4), supervised with BCE loss:

$$\mathcal{L}_{\text{contact}} = \mathrm{BCE}(\hat{c}_{t+1}, c_{t+1}) \tag{5}$$

The predicted contact probability controls visuo-tactile fusion and activates tactile goals during planning. Level 2 3D Tactile Latent: $\hat{h}_{t+1}$ encodes local contact geometry and surface deformation, supervised directly against a stop-gradient target from the tactile encoder: $\mathcal{L}_{\text{tis}} = \|\hat{h}_{t+1} - \text{sg}(h_{t+1})\|$. Level 3 Slip State: a slip head predicts $\hat{s}_{t+1} \in [0,1]$.

3.3 Contact-Gated Visuo-Tactile Attention

Naive concatenation would let noisy tactile tokens interfere with visual tokens during free-space motion. FeelWorld introduces contact-gated asymmetric attention: tactile tokens first attend to visual tokens (anchoring local tactile observations to the global scene):

$$\hat{z}_t^\tau = z_t^\tau + \mathrm{CrossAttn}(z_t^\tau, z_t^o, z_t^o) \tag{11}$$

Whether visual tokens attend to tactile features is governed by the binary contact gate $g_t$:

$$\hat{z}_t^o = z_t^o + g_t\, \mathrm{CrossAttn}(z_t^o, \hat{z}_t^\tau, \hat{z}_t^\tau) \tag{12}$$

The visual component retains a pure-visual prediction path, incorporating tactile evidence only during contact. During training $g_t = c_t$ (ground truth); at inference $g_t = \mathbb{I}[\hat{c}_t \geq 0.5]$.

3.4 Rollout Training and Context Noise Injection

After teacher-forcing training, autoregressive rollout training is used to improve robustness against compounding errors, with context noise injection applied. The total training loss is a weighted sum of all components:

$$\mathcal{L}_{\text{total}} = \lambda_o \mathcal{L}_{\text{vis}} + \lambda_c \mathcal{L}_{\text{contact}} + \lambda_h \mathcal{L}_{\text{tis}} + \lambda_s \mathcal{L}_{\text{slip}} \tag{13}$$

where $\mathcal{L}_{\text{vis}}$ is the LPIPS perceptual loss for visual latent prediction, $\mathcal{L}_{\text{slip}} = \mathrm{BCE}(\hat{s}_{t+1}, s_{t+1})$ is the slip classification loss, and $\lambda_o, \lambda_c, \lambda_h, \lambda_s$ are balancing weights. During autoregressive rollout, the model's own predictions progressively replace ground-truth inputs, and context noise injection applies Gaussian perturbations to encoder outputs to simulate distribution shift.

3.5 Contact-Aware CEM Planning

The predicted contact and slip states support contact-aware CEM. At each replanning cycle, CEM samples 400 candidate action sequences over $H_p=6$ imagination steps, executes the first two actions of the optimal sequence, then replans. Before contact, only the visual goal is optimized; after contact, joint visuo-tactile optimization is activated.

4. Experiments

4.1 Robot Platform and Datasets

The Imeta-Y1 robot is equipped with three RGB cameras and tactile sensors, evaluated on three contact-rich tasks: chip grasping, fruit grasping, and USB insertion.

4.2 Visual Prediction Evaluation

FeelWorld reduces 10-step LPIPS from 0.084 to 0.058, with 80-step autoregressive rollout LPIPS 61% lower than the visual baseline.

Model	10-step LPIPS ↓	80-step LPIPS ↓	Improvement vs. visual baseline
V-JEPA 2 (visual baseline)	0.084	0.149	—
Naive visuo-tactile fusion	0.072	0.094	−37%
FeelWorld (ours)	0.058	0.058	−61%

FeelWorld maintains nearly identical LPIPS (0.058) after 80-step long-horizon rollout as at 10 steps, while the visual baseline degrades from 0.084 to 0.149, showing that contact gating protects visual prediction from tactile-noise erosion during long-horizon rollout.

4.3 Zero-Shot Planning Results

Task	V-JEPA 2 (visual baseline)	Naive visuo-tactile CEM	FeelWorld (ours)
Chip grasping	40.0%	47.5%	82.5%
Fruit grasping	70.0%	75.0%	87.5%
USB insertion	37.5%	50.0%	75.0%
Mean	49.2%	57.5%	81.7%

Visual CEM can guide toward visually plausible goals but cannot guarantee stable contact or successful insertion; naive visuo-tactile CEM improves contact-period optimization but disrupts free-space approach by optimizing unreachable tactile goals before contact; contact-gated CEM relies on vision before contact and activates joint visuo-tactile optimization only after contact. USB insertion improves from 37.5% to 75.0%.

5. Limitations

• Tactile encoder dependency: the 3D tactile latent relies on a pretrained tactile encoder whose representation quality affects downstream prediction.
• Fixed contact-gate threshold: the $g_t = \mathbb{I}[\hat{c}_t \geq 0.5]$ threshold is hardcoded; different tasks may require adaptive thresholds.
• Limited planning horizon: CEM plans with $H_p=6$ steps; planning ability on longer-horizon tasks remains to be validated.
• Three-task evaluation: only chip/fruit/USB tasks are evaluated; generalization to more contact-rich scenarios needs expansion.
• Hardware dependency: requires tactile sensors; robots without tactile sensing cannot use this approach.

6. Conclusion

FeelWorld is a hierarchical visuo-tactile world model that hierarchizes tactile states into contact, 3D tactile latent, and slip, jointly predicted and explicitly supervised by a shared dynamics model. Contact-gated asymmetric attention protects visual prediction before contact and enables joint visuo-tactile dynamics during contact, solving the problem of irrelevant tactile signals degrading visual prediction in free space. Autoregressive rollout and context noise injection improve robustness against compounding errors. The predicted contact and slip states support contact-aware CEM planning—relying on visual guidance before contact and activating joint visuo-tactile optimization only after contact. 10-step LPIPS is reduced from 0.084 to 0.058, with a mean zero-shot planning rate of 81.7%. The core insight: tactile evidence should be "listened to" only upon contact—incorporating it prematurely is like chasing phantoms in free space; contact gating lets the world model switch between vision and touch according to physical timing.

flowchart TD
    A["Multi-view images I_t"] --> B["Frozen visual encoder -> o_t"]
    C["3D tactile point cloud P_t"] --> D["Frozen tactile encoder -> h_t"]
    B --> E["Visual tokens z_t^o"]
    D --> F["Tactile tokens z_t^tau"]
    F --> G["Tactile attend visual: z_t^tau hat"]
    E --> H{"Contact gate g_t?"}
    G --> H
    H -->|No free space| I["Pure-visual prediction path"]
    H -->|Yes contact| J["Joint visuo-tactile dynamics"]
    I --> K["Fused latent z_t hat"]
    J --> K
    L["Proprioception q_t + action a_t"] --> M["Dynamics model W_theta"]
    K --> M
    M --> N["z_{t+1} hat = [z^o; z^tau]"]
    N --> O["Decode: o_{t+1} + (c, h, s)"]
    O --> P["Contact-aware CEM planning"]