PAPER DEEP DIVE
DexTacWAM: A Visuo-Tactile World-Action Model for Dexterous Manipulation
Dexterous manipulation depends on contact dynamics that are often only partially observable from vision. Recent World-Action Models (WAMs) couple predictive video world modeling with action generation, but remain largely vision-centric and therefore cannot directly model these contact dynamics. We present DexTacWAM, a visuo-tactile WAM that encodes each fingertip independently, aggregates the resulting features through a finger- and pose-aware tactile compressor, and injects the tactile latent into a video diffusion world model for joint visuo-tactile world modeling. Across six contact-rich dexterous manipulation tasks on a 22-DoF bimanual platform, DexTacWAM achieves the highest score on every task, averaging 70.6 versus 38.0 for the strongest baseline. Ablations attribute the gain to modeling contact evolution as part of the predicted world state rather than tactile conditioning alone: removing tactile world modeling reduces the four-task mean from 74.7 to 26.6 while keeping the same tactile features and action expert. After four hours of tactile-encoder adaptation with a frozen pretrained vision VAE, our continual vision-to-touch learning extends the pretrained video model to touch using roughly 100 demonstrations per task without tactile midtraining, while retaining visual prediction quality within 0.5 dB of vision-only counterparts. The compressor retains 89.4% of pre-fusion contact recall while enabling 2.26x faster training and 1.29x faster inference. Together, these results show that pretrained video priors can be extended to distributed multi-finger contact dynamics in a data- and compute-efficient manner.
One-line Summary
DexTacWAM turns ten fingertip tactile streams from an extra policy input into part of the predicted world state: a frozen visual VAE encodes each fingertip map, a finger- and pose-aware compressor pools them 5:1 into one latent per hand, those latents join the visual views inside a video diffusion world model, and an action expert reads actions off the predicted visuo-tactile latents, scoring 70.6 on average across six real dexterous tasks versus 38.0 for the strongest baseline.
Figure 1: Paper overview. Left: the three-stage training recipe (tactile-encoder adaptation, visuo-tactile world modeling, action-expert training). Middle: model overview with headline numbers for compression retention and training/inference speedups. Right: real-robot snapshots of the six tasks and the baseline comparison.
Background and Motivation
World-Action Models (WAMs) have become one of the main lines of research in robot learning: a model first predicts the action-conditioned visual future, then an action head reads control commands out of the predictive latents. Together with the success of vision-language-action policies on large-scale visual representations, this recipe already works well for pick-and-place style manipulation. But it has a structural blind spot: it models how the visible scene evolves, not how hidden physical interaction states evolve. Contact, slip, and grasp stability, the physical variables that decide success in contact-rich tasks, are only partially observable from cameras, and sometimes not observable at all.
Dexterous hands multiply this blind spot. A parallel gripper concentrates contact on two pads, and a wrist-mounted six-axis force sensor roughly describes it. A dexterous hand distributes contact over ten fingertip surfaces, and that distribution changes continuously with hand pose, object geometry, and task phase. Judging from vision alone whether the thumb and index finger have actually pinched the bowl rim with enough friction is hard; it becomes impossible when the arm occludes the object. Extending visuo-tactile world modeling from grippers to dexterous hands is therefore not a matter of adding sensor channels: the model must preserve finger identity, contact locality, temporal structure, and pose-dependent contact dynamics at the same time.
Existing visuo-tactile world modeling has not answered this. VTAM augments a pretrained video-action model with tactile streams to correct visual estimation errors; VT-WM predicts future visual and tactile observations to improve the physical fidelity of imagined rollouts under occlusion; OmniVTA predicts short-horizon contact evolution on a larger contact-rich dataset. All of them experiment on parallel grippers or simple end effectors, where tactile observations are localized and structurally simple. The closest relative, ViTacFormer, adds an auxiliary tactile-forecasting head inside an ACT/CVAE policy and autoregressively feeds the forecast back, but it never predicts future vision, and tactile forecasting remains an internal auxiliary process rather than part of the world state.
A second constraint is data asymmetry. Human video is nearly free; tactile data require physical interaction, sensor contact, and often heterogeneous hardware, which makes large-scale tactile pretraining prohibitively expensive. This rules out training a tactile foundation model from scratch and points to the pragmatic route taken here: reuse the spatial representations and temporal dynamics already learned from abundant visual data, extend them to touch with as little interaction data as possible, and do so without sacrificing the original visual prediction quality.
DexTacWAM answers with a single reframing: treat tactile sensing as part of the modeled world state rather than as an auxiliary policy input. Three mutually supporting designs follow. A per-finger tactile encoder that reuses a frozen visual VAE, plus a compressor that preserves finger identity and hand pose. A video diffusion world model in which left- and right-hand tactile latents join the visual views and are denoised jointly. And an action expert that reads the mixed latents stably through per-modality K/V normalization. On a 22-DoF bimanual Sharpa tactile platform across six contact-rich tasks, it scores highest on every task.
Preliminaries: World-Action Models and the Visuo-Tactile World State
At each time step $t$ the robot observes RGB images $o_{t}^{v}$, tactile observations $o_{t}^{\tau}$ distributed across fingers and hands, proprioceptive hand state $q_{t}$, and executes action $a_{t}$. A WAM consists of a world model that predicts future observations in latent space and an action expert that generates actions from the world-model representation. DexTacWAM uses a pretrained video diffusion transformer (an LTX-style DiT) as the world-model backbone, where multi-view latents are coupled by cross-view self-attention and denoised jointly under a flow-matching velocity objective.
The defining move of the paper is the joint visuo-tactile world state. With visual latents $z_{t}^{v}$ and compressed left/right hand tactile latents $\hat{z}_{t}^{\tau,L}$, $\hat{z}_{t}^{\tau,R}$:
$$s_{t}=\{z_{t}^{v},\,\hat{z}_{t}^{\tau,L},\,\hat{z}_{t}^{\tau,R}\}$$The difference from vision-centric WAMs is that tactile latents are not conditioning inputs but constituents of the state, and the world model is held accountable for their future. The action expert then conditions on this joint representation, so it receives predicted contact evolution rather than a tactile snapshot of the current instant.
This framing also buys a deployment advantage: the action expert needs only the predictive latent features from a single world-model forward pass, without waiting for fully denoised visual or tactile futures. The paper calls this the prediction-depth versus latency trade-off; forecast-then-act designs such as ViTacFormer must finish the tactile forecast before any action can be produced.
Method in Detail
1. Per-finger tactile encoding with a frozen visual VAE
The first design choice is not to train a tactile VAE but to reuse the pretrained visual VAE as a per-finger tactile feature extractor. The justification comes from the morphology of the tactile map itself: each Sharpa fingertip returns a grayscale image of a deformable marker grid, and contact appears as localized compression and shear of that grid rather than as a scalar force reading. Such spatially organized local structure, contact regions, deformation patterns, edges, pressure-sensitive texture changes, is compatible with image-style latent encoding, so the spatial prior of the visual VAE transfers directly.
The engineering seam is thin: each per-finger map $o_{t,i}^{\tau}$ is single-channel while the visual VAE expects three channels. A lightweight $1\times 1$ grayscale-to-RGB adapter $\phi_{g\rightarrow rgb}$, initialized to replicate the grayscale map across channels, bridges the gap before the frozen VAE encoder:
$$z_{t,i}^{\tau}=E_{\mathrm{VAE}}\!\left(\phi_{g\rightarrow rgb}(o_{t,i}^{\tau})\right)$$where $i$ indexes fingers. Each hand yields five per-finger latents $Z_{t}^{\tau}=\{z_{t,1}^{\tau},\ldots,z_{t,5}^{\tau}\}$, ten streams bimanually. Keeping the VAE frozen means tactile adaptation happens only in two lightweight components, the grayscale-to-RGB projection and the downstream compressor, leaving the pretrained visual latent space untouched. The probe experiments in Appendix A.11 support this choice: with only the frozen VAE and the learned projection, pooled 128-D right-thumb features reach 95.6% 5-nearest-neighbour accuracy and 99.0% linear-probe accuracy on the six tasks, against a 16.7% uniform chance level.
Figure 2: Multi-finger tactile encoder with the finger- and pose-aware compressor. Five fingertip maps pass through the 1x1 grayscale-to-RGB adapter into a shared frozen visual VAE; per-finger latents receive finger identity embeddings and enter self-attention pooling with a hand query and pose embeddings, performing the 5:1 compression into one hand latent, followed by temporal refinement; pre- and post-fusion reconstruction losses keep localized multi-finger contact information in the compressed latent.
2. Finger- and pose-aware tactile compression
Feeding ten fingertip latent streams into the world model as separate views would explode the view count and token budget, so the compressor aggregates each hand's five per-finger latents into one hand-level latent while preserving finger identity, hand pose, and temporal contact evolution. Two bookkeeping steps precede aggregation: learnable finger identity embeddings $\tilde{z}_{t,i}^{\tau}=z_{t,i}^{\tau}+e_{i}$ keep every token attributable to a specific finger, and a pose encoder $b_{t}^{q}=E_{q}(q_{t})$ injects the current hand configuration into the aggregation query to disambiguate contacts under different hand shapes.
Compression proceeds per spatio-temporal cell $(t,h,w)$ as set encoding: one hand query and five finger tokens form a six-token set, a small self-attention set encoder processes it, and only the hand-query output is retained, completing the 5:1 reduction. For training stability the set-encoder output is added as a residual correction on a softmax-weighted finger average, with logits and residual gain initialized so that the compressor starts from a uniform finger average:
$$\tilde{z}_{t,h,w}=\sum_{i=1}^{5}\mathrm{softmax}(\ell)_{i}\,z_{t,i,h,w}^{\tau}+\alpha\,\tilde{z}_{t,h,w}^{\mathrm{set}}$$The hand query itself carries a learnable spatial bias $p_{h,w}$ and a zero-initialized gated pose term $q_{\mathrm{hand}}^{t,h,w}\leftarrow q_{\mathrm{hand}}+\alpha_{q}E_{q}(q_{t})+p_{h,w}$. The fused hand-level grid is then refined by lightweight divided space-time attention, temporal attention modeling contact onset, slip, regrasp, and handover, spatial attention smoothing neighbourhoods, again added through a zero-initialized residual gate. The compressor outputs
$$\hat{z}_{t}^{\tau,L},\hat{z}_{t}^{\tau,R}=A_{\theta}(Z_{t}^{\tau,L},Z_{t}^{\tau,R},q_{t}),\quad \hat{z}_{t}^{\tau,\cdot}\in\mathbb{R}^{C\times T_{\mathrm{lat}}\times H^{\prime}\times W^{\prime}}$$with $C=128$ and spatial grid $H^{\prime}\times W^{\prime}=6\times 8$ in the implementation, and temporal length aligned to the visual latent track: $T_{\mathrm{lat}}=\mathrm{mem\_size}+\lfloor \mathrm{chunk}/8\rfloor+1$. Each compressed hand representation can therefore enter the world model exactly like one additional camera view.
3. Visuo-bimanual tactile world modeling in a video diffusion backbone
The world-model side changes little on purpose: left- and right-hand tactile latents are appended to the visual latents along the view axis, so the diffusion transformer receives $V=V_{v}+V_{\tau}$ latent views. The DiT architecture is unchanged; cross-view self-attention couples visual and tactile tokens and denoises both modalities jointly, making scene evolution and contact evolution two faces of the same denoising problem.
Training uses a modality-split flow-matching objective. With clean joint state $s^{\mathrm{clean}}=[z^{v}\|\hat{z}^{\tau}]$, noise level $\sigma\in(0,1]$ and Gaussian corruption $\epsilon$, the network predicts the velocity $v_{\theta}(s_{\sigma},\sigma)\approx\epsilon-s^{\mathrm{clean}}$ and is trained with
$$\mathcal{L}_{\mathrm{wm}}=\lambda_{v}\,\mathbb{E}\left[\left\|v_{\theta}(s_{\sigma},\sigma)_{[:V_{v}]}-(\epsilon-z^{v})\right\|_{2}^{2}\right]+\lambda_{\tau}\,\mathbb{E}\left[\left\|v_{\theta}(s_{\sigma},\sigma)_{[V_{v}:]}-(\epsilon-\hat{z}^{\tau})\right\|_{2}^{2}\right]$$with $\lambda_{v}=\lambda_{\tau}=1.0$. The split keeps the visual term responsible for global scene evolution and the tactile term for local contact evolution, allows independent weighting, and makes it trivial to switch tactile prediction off in ablations.
Figure 3: Overall architecture. Left: the World-Model Transformer fuses vision and bimanual tactile tokens through per-modality self-attention and a shared cross-modality self-attention, producing predictive latents that form the joint visuo-tactile world state. Right: the Action Model cross-attends to those latents with per-modality K/V RMS normalization, conditioned on a proprioceptive state token, and predicts contact force, action, and next state.
4. Per-modality K/V normalized action expert
The central difficulty for the action expert is distribution mismatch: visual latents are dense and carry global scene information, tactile latents are sparse and contact-sensitive, and their token scales differ substantially. Concatenating both into a shared cross-attention K/V space lets the larger-magnitude modality dominate the attention softmax and makes optimization ill-conditioned. The empirical verdict is blunt: without per-modality normalization the action expert fails to converge in the authors' experiments, with open-loop predictions unable to track ground-truth actions.
The fix splits the K/V sequence into visual and tactile partitions before cross-attention, normalizes each partition with parameter-free RMSNorm, and concatenates them back:
$$\widetilde{x}=\left[\mathrm{RMS}_{v}\!\left(x_{[:V_{v}L]}\right)\;\|\;\mathrm{RMS}_{\tau}\!\left(x_{[V_{v}L:]}\right)\right],\quad h^{a}=\mathrm{Attn}^{a}(h,\widetilde{x})$$The attention module itself is shared across modalities; there are no learnable modality gates and no separate visual/tactile action pathways, only per-partition statistics equalized before attention. The design costs almost no parameters, stabilizes training, and reduces visual dominance.
The action target is a continuous vector containing per-finger force, arm pose, hand target, and proprioceptive state prediction:
$$a_{t}=[f_{t}^{1:N},\,a_{t}^{\mathrm{arm}},\,a_{t}^{\mathrm{hand}},\,s_{t}^{\mathrm{prop}}]$$The force block holds 60 dimensions, six force-related targets for each of $N=10$ fingers. The arm block is parameterized as an end-effector pose relative to the measured pose at the start of the chunk rather than as absolute joint targets, keeping the prediction centered and scale-stable across the workspace, which matters most for fine relative alignment in bimanual handover. Training uses a single flow-matching MSE over the whole vector:
$$\mathcal{L}_{\mathrm{act}}=\mathbb{E}_{\sigma}\left[\left\|v_{\theta}^{a}(h,\sigma)-(\epsilon_{a}-a_{t})\right\|_{2}^{2}\right]$$Where the force supervision sits matters: per-finger force is part of the unified action target, not an extra weighted force loss, and at deployment the action model takes no force input at all. Touch enters the policy only as predictive latent representations; the auxiliary force supervision exists to keep the action model from ignoring the tactile channel during training.
5. Continual vision-to-touch learning in three stages
Stage 1 adapts the trainable parts of the tactile encoder, the grayscale-to-RGB projection, the compressor, the pose encoder, and the temporal refinement, with the visual VAE frozen. Pretraining uses the diverse-488 corpus of roughly four hours and 488 bimanual dexterous episodes, training on episodes 0 to 463 and holding out 464 to 487 for representation analyses; the loss includes pre- and post-fusion per-finger reconstruction terms ($\lambda_{\mathrm{loc}}=1.0$, $\lambda_{\mathrm{loc\_pre}}=0.3$), which is what writes "compression must still recover per-finger contact" into the objective.
Stage 2 performs no tactile midtraining at all: with Stage-1 tactile modules frozen, the pretrained video model is finetuned per downstream task on roughly 100 teleoperated demonstrations using the modality-split objective above. Stage 3 initializes a fresh 160M action expert and trains it on the same demonstrations, with no separate action pretraining. The philosophy of the pipeline is that a diverse interaction corpus learns a reusable tactile representation, while task-scale data only adapt the world model and the action expert.
flowchart TD
M1[10 fingertip grayscale tactile maps] --> M2[1x1 GrayToRGB adapter]
M2 --> M3[Frozen LTX visual VAE]
M3 --> M4[Per-finger latents plus finger identity embeddings]
M4 --> M5[Per-cell set encoder: 1 hand query plus 5 finger tokens]
Q1[Hand pose encoder] --> M5
M5 --> M6[Softmax finger average plus residual set correction]
M6 --> M7[Divided space-time refinement]
M7 --> M8[Left hand tactile latent]
M7 --> M9[Right hand tactile latent]
R1[Head-view RGB] --> R2[Visual latent views]
M8 --> C1[Concat along view axis: 3 latent views]
M9 --> C1
R2 --> C1
C1 --> W1[Video DiT world model with cross-view self-attention]
W1 --> W2[Modality-split flow matching over vision and touch]
W2 --> W3[Predictive visuo-tactile latents]
W3 --> N1[Per-modality K/V RMSNorm]
N1 --> A1[Action expert cross-attention]
P1[Proprioception state token via MLP] --> A1
A1 --> A2[Action chunk: 60-D finger force, arm, hand, proprio]
Figure 4: Forward data flow of DexTacWAM. Ten fingertip tactile streams become two hand-level views through the frozen VAE and the 5:1 compressor, join the head visual view inside the video DiT for joint denoising, and the action expert reads the predictive latents through per-modality K/V RMSNorm to emit an action chunk that includes per-finger force.
6. Why visual priors transfer to touch
Appendix A.11 answers why a frozen visual VAE can supply useful tactile representations with a battery of probes. On 1,350 right-hand tactile clips sampled from held-out episodes, pooled 128-D right-thumb features reach 95.6% 5-NN and 99.0% linear-probe task accuracy; after 5:1 compression the hand-level features do not degrade but improve, reaching 96.4% and 99.6%, with the silhouette score rising from 0.060 to 0.081. Compression does not erase task-relevant structure.
The temporal analysis is more interesting. Within each task, k-means clusters of right-thumb features align with normalized clip phase: ANOVA $R^{2}$ is 0.75 for Tongs, 0.61 for Two-Hand Wipe, 0.48 for Cube Place, 0.40 for Cube Handover, and 0.17 for Bowl, while adjusted mutual information with episode identity never exceeds 0.07. The latents track interaction progress, not recording-identity. Bottle Cap is the exception at $R^{2}=0.03$ because cyclic unscrewing is not described by a single start-to-end phase. At the same time, visual and tactile latent spaces remain clearly separated, with a between- to within-modality distance ratio of about 89: the benefit does not come from mapping both modalities into one invariant representation, but from the spatial prior of the visual encoder itself.
Experiments
Main results: leading on all six contact-rich tasks
The platform is a bimanual rig with 22-DoF Sharpa tactile hands providing dense contact observations over the finger surfaces. The six tasks probe complementary contact capabilities: Cube Place w/ Occlusion enables only the head camera so arm self-occlusion cannot be resolved by a close-range view; Cube Handover requires coordinated bimanual transfer; Two-Hand Wipe demands sustained contact and stable pressure over a long horizon; Tongs transfers a cherry tomato through a tool, testing force control mediated by the tongs; Bowl requires thumb-index friction to separate stacked bowls; Bottle Cap requires friction plus torque for rotational contact. Camera access is identical across methods within each task, with 20 real-robot trials per method per task and roughly 100 demonstrations per task.
| Method | Cube Place (H) | Handover (H+L+R) | Wipe (H+L+R) | Tongs (H+R) | Bowl (H+R) | Bottle Cap (H+L+R) | Avg. |
|---|---|---|---|---|---|---|---|
| pi0.5 (no tactile) | 10.0 | 46.0 | 24.0 | 0.0 | 40.0 | 45.0 | 27.5 |
| ViTacFormer | 20.0 | 33.3 | 26.0 | 0.0 | 20.0 | 10.0 | 18.2 |
| RDP | 50.0 | 60.0 | 48.0 | 10.0 | 30.0 | 30.0 | 38.0 |
| Genie Envisioner (no tactile) | 10.0 | 30.0 | 44.0 | 5.0 | 35.0 | 40.0 | 27.3 |
| DexTacWAM (ours) | 90.0 | 71.6 | 77.0 | 60.0 | 65.0 | 60.0 | 70.6 |
Table 1: Task scores (%) over 20 real-robot trials per method per task; parenthesized letters are the camera configuration, identical across methods within a task.
DexTacWAM takes first place on all six tasks, averaging 70.6 against 38.0 for the strongest baseline, RDP. The largest margin is Tongs, 60 versus 10: the tomato is held through a tool, cameras cannot see the contact between tongs and tomato, and the only reliable evidence of grasp state is force transmitted through the tongs, so every camera-driven baseline scores at or below 10. The pattern matches the motivating hypothesis: gains are largest exactly where instantaneous visual observations least suffice to infer task-relevant physical state.
The controlled comparisons matter as much as the headline. RDP receives the same ten fingertip 6-D wrenches as DexTacWAM but consumes them as downstream policy inputs without modeling contact evolution, and averages 38.0; the two vision-only entries, pi0.5 and Genie Envisioner, score 27.5 and 27.3. Tactile conditioning alone is not sufficient; putting contact evolution into the predicted world state is what produces the gap. Notably, these results come with no tactile midtraining of the video model, an action expert trained from scratch, and no reinforcement-learning post-training.
Figure 5: Keyframes of autonomous closed-loop rollouts on the six tasks, ordered left to right by execution progress, showing occluded placement, bimanual handover, two-hand wiping, tool-mediated grasping, bowl separation, and cap unscrewing.
Statistical significance and unseen configurations
To compensate for 20 trials per task, the paper pools the four binary tasks (Cube Place, Tongs, Bowl, Bottle Cap) into 80 trials per method: DexTacWAM succeeds on 55 (68.8%) versus 24 (30.0%) for RDP, with 95% Wilson confidence intervals of [57.9, 77.8] and [21.1, 40.8] and a Fisher exact test at $p=1.59\times 10^{-6}$. The overall success-rate advantage is statistically significant.
Generalization is tested on Bowl: training sees only three-bowl stacks in pink, blue, and orange; at test time unseen green, purple, and yellow bowls yield 13/20 successes, matching in-distribution performance, and unseen stack heights of two or four bowls yield 12/20. Appearance and stack geometry change together without a performance drop, indicating the policy relies on contact and geometric relations rather than color memory of the training bowls.
Figure 6: Rollouts on held-out Bowl configurations. The top three rows use unseen bowl colors and the bottom two unseen stack heights; in every case the policy reaches the rim, establishes thumb-index contact, and separates the top bowl.
Ablations: tactile world modeling and per-modality normalization
The policy-level ablations subtract only. The "w/o tactile world modeling" variant keeps the tactile encoder, observations, action expert, action space, and training data, and removes only tactile prediction from the world model so the action expert conditions directly on encoded tactile features: the four-task mean collapses from 74.7 to 26.6. The tactile information content is unchanged; what changes is whether contact evolution belongs to the predicted world state, which attributes the gain to predictive modeling rather than tactile conditioning. The "w/o per-modality K/V RMS normalization" variant fails to converge, with open-loop predictions unable to track ground truth, so it never reaches closed-loop evaluation.
| Variant | Cube Place | Handover | Wipe | Tongs | Avg. |
|---|---|---|---|---|---|
| DexTacWAM (full) | 90.0 | 71.6 | 77.0 | 60.0 | 74.7 |
| w/o tactile world modeling | 20.0 | 33.3 | 48.0 | 5.0 | 26.6 |
| w/o per-modality K/V RMS normalization | Did not converge | ||||
Table 2: Policy-level ablations, best task score (%).
Compressor ablations: contact recall and retention
Encoder-level ablations are measured by contact recall on the 24 held-out episodes of diverse-488, a more direct measure than downstream task scores. Retention is defined as the ratio of post-fusion to pre-fusion recall:
$$\mathrm{Retention}=\frac{\mathrm{Post\text{-}Fusion\ Recall}}{\mathrm{Pre\text{-}Fusion\ Recall}}$$Starting from single-layer cross-attention pooling (Recall@9 of 0.492, retention 62.9%), six-token self-attention set encoding raises Recall@9 to 0.575; adding pose injection or temporal refinement gives 0.645 or 0.710; the full compressor with both reaches Recall@1 0.748, Recall@9 0.725, and retention 89.4%. Removing finger-identity embeddings drops Recall@9 to 0.671, confirming that tokens must stay finger-attributable instead of degenerating into an unordered contact bag. All variants share the same frozen VAE and per-finger inputs, so the differences come purely from the aggregation design.
| Variant | Recall@1 | Recall@9 | Retention@9 |
|---|---|---|---|
| Cross-attention pooling | 0.561 | 0.492 | 62.9% |
| Self-attention pooling | 0.656 | 0.575 | 71.7% |
| Self-attention + pose injection | 0.750 | 0.645 | 79.6% |
| Self-attention + temporal refinement | 0.748 | 0.710 | 87.3% |
| Full compressor (pose + temporal) | 0.748 | 0.725 | 89.4% |
| w/o finger-identity embeddings | 0.702 | 0.671 | 84.9% |
Table 3: Tactile-encoder component ablations on held-out episodes, measured by contact recall and retention.
Is the world model really predicting contact evolution
Tactile signals are temporally smooth, so a forecaster that merely repeats the last observation can score well; the paper therefore compares against a copy-last-frame baseline. On held-out trajectories the predicted tactile futures reach 0.967 latent cosine similarity and 0.063 NMSE against ground truth, with contact precision/recall/F1 of 0.749/0.725/0.737 in flow space, and an error ratio of 0.561x relative to copy-last-frame. The model predicts contact evolution rather than exploiting smoothness.
| Metric | DexTacWAM |
|---|---|
| Latent cosine similarity (higher better) | 0.967 |
| Latent NMSE (lower better) | 0.063 |
| Contact precision / recall / F1 | 0.749 / 0.725 / 0.737 |
| Error ratio vs copy-last-frame (lower better) | 0.561x |
Table 4: Tactile world-model prediction quality on held-out trajectories; contact metrics are computed in flow space.
Figure 7: Joint visual and tactile prediction on the wiping task. Tactile rows show right-index shear magnitude on a shared color scale; across approach, contact, wiping, and release the predicted contact patch emerges, moves, and fades with the interaction instead of persisting as a static imprint.
Figure 8: Per-finger tactile-flow prediction at peak wiping shear (t = 24.2 s) for all ten fingers across dx, dy, and divergence channels; predicted fields reproduce the dominant deformation on contacting fingers, stay near zero on idle fingers, and follow the sign flip of the left-thumb dx field between the two instants.
Does acquiring touch degrade visual prediction
The other side of continual learning is catastrophic forgetting. For each task the paper trains a vision-only counterpart identical in every respect except the tactile-view switch, and reports paired per-episode differences rather than unpaired means: delta PSNR is +0.06 dB on Cube Handover and -0.07 dB on Two-Hand Wipe, with paired 95% confidence intervals of [-0.16, +0.28] and [-0.49, +0.36] dB, and similarly negligible SSIM and LPIPS differences. Given that episode difficulty alone shifts PSNR by about 2 dB, these data bound any visual prediction loss at half a decibel.
Efficiency gains from compression
Compressing ten fingertip streams into two hand-level latents cuts the views attended by the DiT from 11 (head plus ten fingertips) to 3 (head plus two hands), reducing view tokens from 3168 to 864 (3.67x). On a single RTX 4090, end-to-end latency drops from 363.0 to 281.6 ms per chunk (1.29x), staying below the 1.8 s action-chunk execution time so the control loop closes; under the same training setup, iteration time falls from 3.87 to 1.71 s per iteration (2.26x), directly lowering the cost of scaling visuo-tactile world-model training.
Failure modes
The per-task failure analysis in Appendix A.8 provides an honest counterpoint. On Two-Hand Wipe, DexTacWAM failures often amount to wiping part of the board without finishing it, exposing a weakness in instruction following and long-horizon progress monitoring, precisely where a large pretrained VLA such as pi0.5 is strong; conversely pi0.5 fails on the same task because its left hand cannot detect that the whiteboard is slipping. On Cube Handover the dominant failure is sensitivity to the cube's initial pose outside the training distribution, with the error propagating from the first grasp into the transfer. The authors propose tactile-informed reinforcement learning for closed-loop correction: once contact is established, fingertip observations describe object pose relative to the fingers and grasp stability more directly than wrist cameras do.
Limitations
First, sensor and embodiment coverage is narrow, a limitation the authors state themselves: every experiment uses the Sharpa dexterous hand with vision-based tactile sensing, and transfer to capacitive, piezoresistive, piezoelectric, or magnetic sensing principles is entirely untested. Whether reusing a visual VAE works for non-image-like tactile signals, such as sparse electrode arrays or one-dimensional force traces, remains open.
Second, the training distribution is success-only, also author-stated: both the world model and the action expert see only successful demonstrations, leaving missed grasps, slips, and dropped objects out of distribution with little supervision for recovery. The qualitative observation in Appendix A.9 agrees: visuo-tactile predictions stay plausible until the first execution error, after which both prediction and action quality deteriorate once the policy enters unseen failure states. Recovery behavior will require failure and recovery data from perturbation, intervention, or reinforcement learning.
Third, evaluation scale and policy form leave room for caution: 20 trials per task and a separately finetuned model per task mean six world-model and action-expert weight sets rather than one multi-task policy; the action expert is initialized from scratch on roughly 100 task demonstrations, and the authors themselves name action pretraining as a clear direction for further gains. All results also come from a single laboratory platform, with no cross-institution reproduction yet.
Conclusion and Outlook
DexTacWAM's contribution reduces to a change of perspective: touch is no longer a signal consumed by the policy but a future the world model must account for. Around that reframing the paper supplies three independently verifiable components, per-finger encoding with a frozen visual VAE, 5:1 compression that preserves finger identity and hand pose, and modality-split joint denoising with per-modality K/V normalization, and uses ablations to attribute the gain to predicting contact evolution rather than conditioning on touch. On the data side, four hours of tactile-encoder adaptation plus about a hundred demonstrations per task graft tactile prediction onto a pretrained video model while bounding visual prediction loss at half a decibel; on the compute side, compression makes training 2.26x and inference 1.29x faster.
For robot learners the paper offers a replicable recipe: do not build a separate tactile stack, translate touch into a language the visual prior already understands, and let the prediction objective itself carry the fusion burden. The natural extensions are the two the authors name, cross-sensor and cross-embodiment tactile pretraining, and failure/recovery data for self-correction; and the choice to supervise per-finger force inside the unified action target while deploying without any force input is a template worth borrowing for distilling touch into policies.
