PAPER DEEP DIVE
$N_0$-VTLA: Scaling Vision-Tactile-Language-Action Model with Latent Tactile Tokens
We present $N_0$-VTLA, a vision-tactile-language-action (VTLA) foundation model capable of (1) fine-grained contact-rich manipulation with tactile perception and tactile-feedback control, and (2) offline policy improvement from stored deployment data. Building on current vision-based backbones, we propose a training recipe for tactile integration consisting of visuo-tactile pre-training, staged tactile-pathway integration, and advantage-conditioned offline policy improvement. During pre-training, the policy learns broad contact priors from NeoData, our large-scale visuo-tactile robot dataset; to our knowledge, $N_0$-VTLA is the first VTLA model pretrained on tactile data at scale. During post-training, we augment the policy with a predictive tactile pathway that distills the contact patterns learned at scale into the fine motion adjustments required by downstream tactile-centric manipulation. For offline policy improvement, we introduce ALTER, an advantage-conditioned offline reinforcement learning method that converts relative progress and trajectory-event comparisons into binary advantage labels for policy training on a fixed deployment corpus, further improving task-specific learning on contact-rich skills such as deformable object manipulation. Across contact-rich benchmarks, $N_0$-VTLA outperforms strong baselines by wide margins: it wins all nine real-robot NeoReal tasks and reaches 63.8% mean success on a twenty-task simulation suite, against 44.0% for the strongest baseline. $N_0$-VTLA policies trained with ALTER reach 75-95% success on three long-horizon real-robot tasks. These results lay a foundation for versatile tactile-driven manipulation policies.
𝒩₀-VTLA: Scaling Vision–Tactile–Language–Action Model with Latent Tactile Tokens
Authors: NeoteAI Team & Fudan TEAI Team | arXiv: 2607.23782v1 | Code: github.com/neoteai/N0-VTLA | Website: research.neoteai.com/n0-vtla/
One-Sentence Summary
𝒩₀-VTLA is the first vision–tactile–language–action (VTLA) foundation model pretrained on tactile data at scale, using latent tactile tokens that treat touch as a prediction target (not observation context) to condition the action expert, and ALTER offline RL to continuously improve policies from deployment data, winning all nine real-robot tasks and outperforming baselines across twenty simulation tasks.
Background and Motivation
VLA models have made manipulation policies general, but touch has remained largely absent, leaving a persistent weakness in contact-rich manipulation. Prior tactile extensions either concatenate tactile tokens into the vision–language prefix (treating touch as another camera) or inject current tactile readings into the action pathway. The former wastes capacity on sparse signals in an information-dense prefix; the latter keeps the policy one step behind — a tactile frame records contact that actions already produced, not contact the next actions must anticipate.
Figure 1: 𝒩₀-VTLA encodes vision, instruction, and tactile difference images, predicts latent tactile tokens $z$, and conditions a flow-matching action expert on them. ALTER converts deployment experience into stage-relative advantage labels for offline policy learning.
𝒩₀-VTLA takes a third path: a predictor reads the vision–language context and current tactile tokens, emitting latent tactile tokens $z$ that estimate the net tactile change over the coming action chunk, so the policy acts on the contact its own actions are about to cause.
Base Architecture
The base policy pairs a PaliGemma vision–language backbone with a flow-matching action expert. Camera views, instruction, and robot state form the model prefix (state enters discretized). The action expert denoises an action chunk over $H=50$ steps in a 32-dimensional canonical cross-embodiment container (inherited from $\pi_{0.5}$). Single- and dual-arm data coexist in one fixed-width model.
Latent Tactile Tokens
Tactile Encoder
The policy never sees raw tactile frames. For view $k$, the episode-start baseline $\mathrm{tac}_0^k$ is subtracted from the current frame $\mathrm{tac}_\tau^k$, encoded by a frozen self-supervised visual encoder with a trainable projection:
$$g_{k}=f_{\mathrm{enc}}\!\big(\mathrm{tac}_{\tau}^{k}-\mathrm{tac}_{0}^{k}\big)\in\mathbb{R}^{10\times d}$$Each tactile image yields 10 tokens (1 class + 9 spatial from 3×3 adaptive pooling). Differencing removes static gel appearance and mount-specific imprints. The frozen encoder preserves the self-supervised representation, enables onboarding new sensors via lightweight projection, and saves memory.
Predictor and Future-Tactile Target
The predictor reads current tactile difference tokens $g$ and the contextualized VL prefix, distilling them into latent tactile tokens $z$. The key is $z$'s predictive target — applying the same encoder to each view's tactile change over the next $H=50$ steps and averaging:
$$z^{*}=\frac{1}{n}\sum_{k=1}^{n}f_{\mathrm{enc}}\!\big(\mathrm{tac}^{k}_{\tau+H}-\mathrm{tac}^{k}_{\tau}\big)\in\mathbb{R}^{10\times d},\qquad H=50$$The predictor output $z$ is trained to match $z^*$ via symmetric InfoNCE contrastive loss and $L_1$ reconstruction.
Why Predict, Not React?
The behaviors where touch matters most are anticipatory — millimeter-scale pre-load before grasp closure, catching incipient slip. These live in the pre-contact blind spot where reactive signals arrive too late. The predictor predicts in latent space rather than reconstructing raw signals, following the joint-embedding predictive principle. Current tactile encoding alone retrieves future targets at 57% top-1 vs. the predictor's 92.3% (chance: 3.2%).
flowchart TD
A["Tactile difference image
tac_τ - tac_0"] --> B["Frozen encoder + trainable projection
g ∈ R^(10×d)"]
B --> C["Predictor
reads g + VL prefix"]
C --> D["Latent tactile tokens z
predicts future H-step tactile change"]
D --> E["Action expert
conditioned on z, generates action chunk"]
F["Vision-language prefix
excludes tactile"] --> E
D -.->|"Stage1 supervision:
InfoNCE + L1 reconstruction"| G["Future tactile target z*"]
Three-Stage Training Recipe
Stage 1 (Grounding the predictor): Freeze the entire base policy; train only the predictor, tactile projection, and reconstruction head. Contrastive loss:
$$\mathcal{L}_{\mathrm{NCE}}=-\frac{1}{2B}\sum_{i=1}^{B}\left[\log\frac{e^{s_{ii}}}{\sum_{j=1}^{B}e^{s_{ij}}}+\log\frac{e^{s_{ii}}}{\sum_{j=1}^{B}e^{s_{ji}}}\right]$$where $s_{ij}=\langle h(z_i), h(z_j^*)\rangle$. Reconstruction: $\mathcal{L}_{\mathrm{rec}}=\|r_\psi(z)-\bar{D}_{\tau\to\tau+H}\|_1$. Stage 1 total:
$$\mathcal{L}_{1}=\mathcal{L}_{\mathrm{NCE}}+\lambda_{\mathrm{rec}}\,\mathcal{L}_{\mathrm{rec}}$$Stage 2 (Aligning the action expert): Freeze the tactile perception stack; train the latent-to-expert projection and action expert. Mask the VL prefix keys/values in attention, forcing action prediction through $z$.
Stage 3 (End-to-end joint training): Unfreeze everything except the tactile encoder backbone; train jointly under the action objective. Stage 1 targets are no longer applied; the predictor retains grounding through the action gradient alone.
ALTER: Offline RL from Deployment Data
ALTER (Advantage Labeling from Trajectory Events and Relative Progress) performs advantage-conditioned offline RL from a fixed deployment corpus. A pairwise progress model $A_\theta(x_a, x_b) \in [-1,1]$ is trained from clean demonstrations (dense stage-progress), tactile-detected object-drop events, and logged human corrections (sparse preferences).
Duration-calibrated stage progress: $w_k = \bar{d}_k / \sum_j \bar{d}_j$, and frame $t$'s global progress:
$$\phi_{t}=\sum_{j<k}w_{j}+w_{k}u_{t}$$The progress model optimizes:
$$\mathcal{L}_{\mathrm{prog}}=\mathbb{E}_{\mathcal{D}_{\mathrm{stage}}}[A_\theta(x_a,x_b)-(\phi(x_a)-\phi(x_b))]^2+\lambda_{\mathrm{event}}\,\mathbb{E}_{\mathcal{D}_{\mathrm{event}}}[\max(0,m-yA_\theta(x_a,x_b))]^2$$Binary advantage labels: top $\rho=0.3$ within each stage are positive ($c_t = \mathbf{1}[\hat{r}_t \geq q_{s_t}]$), appended as text to the task prompt. Deployment always uses "Advantage: positive".
Experimental Results
Real-World: NeoReal (9 tasks)
| Method | Avg Success Rate | Avg Progress Score | Notes |
|---|---|---|---|
| ACT | 0% | 10.2 | Completes no task |
| π₀.₅ | 29.4% | 42.3 | Strongest baseline |
| 𝒩₀-VTLA | 47.2% | 56.8 | Beats strongest baseline on every task |
Simulation: UniVTAC + NeoSim (20 tasks)
| Method | UniVTAC avg (8 tasks) | NeoSim avg (12 tasks) | 20-task overall |
|---|---|---|---|
| π₀.₅ | 41.4% | 45.8% | 44.0% |
| InternVLA-A1 | 67.1% | 8.6% | — |
| StarVLA-α | 56.1% | 23.2% | — |
| 𝒩₀-VTLA | 83.1% | 50.8% | 63.8% |
ALTER Offline Policy Improvement
| Variant | Towel Folding | Bag Packing | Cardboard Box Folding |
|---|---|---|---|
| π₀.₅-SFT | 40% | 20% | 5% |
| 𝒩₀-VTLA-SFT | 50% | 50% | 20% |
| π₀.₅+ALTER | 90% | 75% | 60% |
| 𝒩₀-VTLA+ALTER | 95% | 80% | 75% |
Representation Analysis
After Stage 1, latent tokens $z$ retrieve matching future-tactile targets at 92.3% top-1 accuracy (chance: 3.2%). Current tactile encoding alone achieves only 57%, confirming that predictive modeling outperforms reactive encoding.
Key Findings
- Touch turns insertion from a one-shot visual commitment into a contact-aware retry loop.
- Touch enables fine control of gripper force through aperture adjustment.
- Touch changes the action exactly when contact decides the outcome.
- Stronger tactile pretraining remains advantageous under ALTER.
Limitations
- Dual-arm success rate (39.4%) is far below single-arm (73.8%); bimanual coordination remains challenging.
- Tactile sensors require custom hardware, increasing deployment cost.
- ALTER depends on deployment corpus quality; mislabeled annotations may propagate.
Conclusion and Outlook
𝒩₀-VTLA is the first VTLA foundation model pretrained on tactile data at scale. The core innovation is treating touch as a prediction target rather than observation context — latent tactile tokens estimate upcoming contact changes, enabling anticipation rather than reaction. The three-stage recipe stably grafts the tactile pathway onto a pretrained VLA. ALTER offline RL further extracts advantage signals from deployment data for continuous improvement.
Key insight: "Touch turns insertion from a one-shot visual commitment into a contact-aware retry loop." — Through predictive latent tactile modeling, 𝒩₀-VTLA changes actions at the critical moment when contact decides the outcome, something purely visual policies cannot achieve.
Deep analysis generated by RobotWorld paper-detail-generator based on full-text reading | arXiv:2607.23782v1
SOURCE LINKS



