Skip to content
RobotWorld
Back to Papers

PAPER DEEP DIVE

触觉感知视触表示学习灵巧操作

$\mathcal{N}_0$-Foundation: Towards the Age of Tactile Intelligence

We present $\mathcal{N}_0$-Foundation, a paradigm for tactile-enabled embodied manipulation, which integrates tactile sensing hardware, large-scale multimodal data, tactile representation learning, and standardized evaluation. First, we engineer the infrastructure for scalable data collection, including a vision-based tactile sensor, a tactile Universal Manipulation Interface (UMI), and a synchronized visuo-tactile data collection system supporting both robot embodiments and UMI-based demonstrations. Leveraging this infrastructure, we construct NeoData, which contains more than 30000 hours of synchronized visual and tactile demonstrations, spanning six embodiments, 450 tasks, and billions of paired RGB and tactile frames collected through a mixture of real-robot teleoperation and UMI-based demonstrations. To facilitate open research, we further release OpenNeoData, a 5000-hour open-source subset of NeoData. The dataset addresses a central limitation of existing manipulation corpora, critical for deformable-object manipulation, precise assembly, delicate force control, and sustained surface interaction. Capitalizing on the large-scale, heterogeneous tactile measurements, we propose NeoForce, a visuo-tactile representation model that learn transferable tactile representations across different sensor designs. To enable systematic evaluation of tactile embodied models built upon our infrastructure, datasets and tactile representations, we further propose a comprehensive benchmark, which combines the real-world NeoReal suite and the simulated NeoSim suite for standardized evaluation. Experiments across both suites show that policies benefit from the physical contact state rather than from the device-specific appearance of the tactile signal. We release the dataset, the representation, and the benchmark, aiming at supporting future work on tactile-enabled embodied manipulation.

NeoteAI Team, Fudan TEAI TeamAugust 30, 202621 min read
中文

Paper Information

  • Title: $\mathcal{N}_0$-Foundation: Towards the Age of Tactile Intelligence
  • Teams: NeoteAI Team & Fudan TEAI (Touch Embodied AI) Team
  • arXiv: 2608.29601 (submitted Aug 30, 2026; cs.RO / cs.CV / cs.LG)
  • Project page: research.neoteai.com/n0-foundation
  • Open data: HuggingFace NeoteAIEmbodied/OpenNeoData (5,000-hour open-source subset, CC BY-NC-SA 4.0, gated access)
  • Code status: no code repository public at reading time; the paper releases the dataset, the representation and the benchmark, and OpenNeoData is already online on HuggingFace.

One-Sentence Summary

This report closes the loop "tactile hardware → 30,000-hour visuo-tactile corpus NeoData → hardware-agnostic unified force-field representation NeoForce → NeoReal/NeoSim dual benchmarks" as one coherent foundation for tactile-enabled embodied manipulation, and shows experimentally that policies benefit from the physical contact state itself rather than from the device-specific appearance of tactile signals.

Figure 1: Overview of $\mathcal{N}_0$-Foundation, a tactile-centric foundation for embodied manipulation that unifies hardware, data, representation and evaluation (paper Figure 1).

1. Background and Motivation

Over the past three years, progress in general manipulation policies has been built almost entirely on visual corpora. BridgeData V2, Open X-Embodiment and DROID established data scale and embodiment diversity as the two central ingredients of robot learning, but their observation channels remain dominated by vision and proprioception. A structural gap keeps resurfacing: contact states, local forces and incipient slip are barely observable from vision alone, yet they are decisive for deformable-object manipulation, precise assembly, delicate force control and sustained surface interaction. Vision can tell a policy that the hand is approaching a cup; it cannot tell the policy whether the grasp is already secure enough to lift.

A second problem is the fragmentation of tactile hardware itself. Optical gel sensors, capacitive taxel arrays and piezoresistive skins differ in units, resolution and sensing geometry, so any representation or model built on one signal form is bound to the device that produced it and cannot be carried elsewhere. Even where tactile data accumulate, they fail to produce the compounding scale effects vision enjoys — every new sensor effectively restarts the modeling effort from scratch.

The paper's Table 1 maps this gap systematically. BridgeData V2 (60k trajectories / 13 tasks), Open X-Embodiment (1M+ / 22 embodiment types) and DROID (76k / 350 hours) have no tactile channel at all; RH20T provides synchronized vision, force, audio and action, but fingertip tactile signals appear in only one of its seven configurations; AgiBot World collects more than one million trajectories (2,976.4 hours) with over 100 homogeneous mobile dual-arm robots, but on a single platform; portable visuo-tactile collection devices (Touch in the Wild, FreeTacMan, ViTaMIn-B, TacUMI, exUMI) attach contact sensors to handheld collectors but stay at the scale of $10^2$–$10^4$ demonstrations with mutually incompatible tactile encodings; TacVerse and Daimon-Infinity scale robot-free collection but contain no physical-robot demonstrations; the closest prior corpus, OmniVTA (21,879 visuo-tactile-action trajectories over 86 tasks), is the only one pairing robot-based and handheld collection, still without verified per-episode quality control.

The authors therefore distill the bottleneck of tactile embodied intelligence into four interlocking gaps: (i) no scalable collection infrastructure that combines real-robot teleoperation's executability with handheld collection's scale; (ii) no large-scale, cross-embodiment visuo-tactile corpus with verified quality control; (iii) no hardware-agnostic tactile representation that arbitrary downstream policies can consume; (iv) no standardized real-plus-simulated evaluation for contact-rich manipulation. $\mathcal{N}_0$-Foundation answers each gap in turn. It deliberately proposes no stronger policy model; it defines the complete pipeline — from collection to evaluation — on which tactile policies can be built.

This positioning matters. In visual manipulation, the division of labor between data infrastructure (Open X-Embodiment) and policy models (the $\pi_0$ family) is already clear; in tactile manipulation the infrastructure itself has not yet taken shape because of hardware fragmentation. The contribution is closer to paving the first highway for the field — its value will be redeemed by the policies later trained on top of it.

2. Collection Infrastructure: Sensor and $\mathcal{N}_0$-TacUMI

The foundation starts with a camera-based visuo-tactile sensor in the GelSight/DIGIT lineage: contact information is recovered from the optical distortion of a deformable sensing surface. The sensor is a layered assembly — a thin wear-resistant glass panel protects the contact surface; beneath it an elastic sensing layer carries a textured pattern whose displacement encodes the applied force; an RGB camera embedded in the body observes the underside of the sensing layer while a compact controller board handles illumination, acquisition and transmission.

When a finger contacts an object, local pressure and shear deform the sensing layer and the camera captures the resulting texture displacement as a tactile image $T_t \in \mathbb{R}^{H \times W \times 3}$. The paper writes this imaging process as a composition of two operators:

$$T_t = \mathcal{C}\big(\Phi(F_t)\big)$$

where $\Phi$ is the elastomer force-to-deformation transfer function mapping a spatial force distribution $F$ to a deformation field, and $\mathcal{C}$ is the internal camera's imaging process. The implication is precise: recovering the force field $F_t$ from $T_t$ means inverting the composed operator $\mathcal{C} \circ \Phi$ — the theoretical motivation for the learned force-field conversion model $g_\theta$, and the seed of the argument that "tactile images are device appearance; force fields are the physical essence".

Figure 2: The $\mathcal{N}_0$-TacUMI handheld collector: a fisheye camera captures the wide-angle wrist view, two camera-based tactile sensors record contact-rich deformation at the fingertips, an infrared tracker provides a low-drift six-DOF pose and a magnetic encoder reads gripper aperture through a rack-and-pinion transmission; inset shows the layered sensor structure (paper Figure 2).

On top of the sensor, the team builds $\mathcal{N}_0$-TacUMI for robot-free data collection: a parallel gripper equipped with two tactile sensors, a $160^\circ$-FOV fisheye camera for the wide-angle wrist view, an infrared tracker providing the six-DOF gripper pose in a global tracking frame, and a magnetic encoder reading instantaneous finger separation through a rack-and-pinion transmission. It records synchronized wrist images, left and right tactile images, gripper aperture and relative end-effector motion.

Actions follow the UMI convention — end-effector pose plus gripper width rather than robot-specific joint commands — so demonstrations are not bound to any particular robot base frame, and handheld trajectories are directly compatible with teleoperated robot data. This single design choice is what later allows two collection modes to be fused into one corpus.

3. NeoData: A 30,000-Hour Visuo-Tactile Corpus

NeoData aggregates more than 30,000 hours of interaction into 1.4M episodes and 3.3B timesteps, yielding 8B RGB frames paired with 10B tactile frames. The corpus spans six embodiments — the handheld $\mathcal{N}_0$-TacUMI plus five robot platforms (ARX X5, UR5e, Flexiv Rizon 4s, Piper, Franka Research 3) — collected by nearly 100 human operators. Every episode carries synchronized visual and tactile streams, so the contact signal is available throughout the manipulation process rather than only at discrete grasp events. For open research the team releases OpenNeoData, a 5,000-hour open-source subset spanning six embodiments, more than 250 tasks and over 200 skills.

Figure 3: NeoData statistics — trajectories per embodiment, with $\mathcal{N}_0$-TacUMI contributing 57% of all trajectories (paper Figure 4).

The composition is deliberately tactile-relevant. $\mathcal{N}_0$-TacUMI contributes 57% of all trajectories, reflecting the scalability of handheld collection for dexterous and repetitive tasks, while the five robot embodiments anchor the corpus in physically executable behavior. High-volume task categories — folding, pick-and-place, wiping/cleaning, stacking, assembling, inserting, pouring, opening, closing, classification — cover deformable-object manipulation, precise mating, delicate grasping, sustained surface contact and manipulation under ambiguous visual states. Scenes span tabletop, workbench, laundry, kitchen, lab bench and storage environments; episode durations peak near 80 seconds with a long tail of multi-step interactions, supporting both short contact primitives and extended task sequences.

4. Data Governance: Five Quality Gates and Four-Level Annotation

Fusing real-robot teleoperation with handheld UMI demonstrations into one trainable corpus requires serious engineering. The curation pipeline (Figure 5 in the paper) runs five complementary checks. Data Completeness: episodes missing any required visual, tactile, action or metadata stream are removed. Motion Quality: trajectory smoothness is estimated along each episode; abrupt tracking pulses and tracking-lost segments are flagged and episodes dominated by such artifacts are dropped. Episode Quality: for each task the median duration $m$ and the interquartile range $r = q_{75} - q_{25}$ are computed, and only episodes with duration in $[m-3r, m+3r]$ are kept; episodes terminating before task completion are also removed, normalizing demonstration pace. Video Quality: every view is inspected for corruption, severe blur, dropped frames and abnormal illumination; asynchronously recorded modalities are aligned to a unified timeline with strict one-to-one correspondence between video, tactile and action frames; swapped camera mappings, mislabeled arms and abnormal end-effector poses are repaired where possible (camera remapping, pose recovery via forward kinematics) and discarded otherwise; near-static segments are trimmed so that models do not pause on inconsistent actions at inference. Transfer Feasibility: since most data are collected robot-free, each trajectory is checked with inverse kinematics and poses that fail to solve or exceed joint and workspace limits are rejected — preserving handheld scalability while guaranteeing every retained demonstration is executable on the target embodiment.

Figure 4: The NeoData curation pipeline — raw demonstrations must pass completeness, motion quality, episode quality, video quality and transfer feasibility before entering the released corpus (paper Figure 5).

Annotation is equally engineered. Each episode receives a four-level hierarchy — task (L3), subtask (L2), action (L1), atomic segment (L0) — separating semantic structure from temporal localization: templates define what should occur in a task, while signal-based segmentation determines when each event occurs in a specific episode. The pipeline has two stages: a vision-language model first summarizes representative episodes into a time-independent task template (objective, ordered semantic phases, action-level substeps), which a human expert verifies before it is applied to the whole task category; then each episode is segmented by fusing action-based and visual event boundaries, the model labels the fixed intervals against the verified template, and deterministic code assembles the L0–L3 temporal extents. Temporal endpoints stay grounded in measured signals while language models handle only semantics, reducing temporal drift on long manipulation videos and supplying the temporal structure long-horizon, multi-stage policies need.

Figure 5: The hierarchical annotation pipeline — VLM-proposed task template, human verification, signal-based segmentation, then deterministic assembly of the four-level hierarchy (paper Figure 6).

5. A Unified Force-Based Tactile Representation

The representation design starts from three principles. Physical grounding: anchor the representation to what touch physically delivers rather than to an appearance-level proxy — what an end-effector delivers to an object is fully described by its effect at the contact interface. Cross-sensor unification: the target space must be one that any calibrated sensor can map into, making heterogeneous tactile data directly comparable and jointly trainable. Spatially rich semantics: the location, shape and extent of the contact patch, together with the intensity variation inside it, distinguish a full grasp from an edge contact and a stable hold from incipient slip — the representation must preserve this distribution, not summarize it away.

Force is chosen as the physical attribute, since it describes the effect of contact independently of the measuring sensor. At every point of the sensing surface the local force is recorded as two in-plane tangential components (shear, driving sliding and grasping) plus one normal component (pressure during pressing). Sampling on a high-resolution lattice yields the dense force field

$$F_t \in \mathbb{R}^{H \times W \times 3}, \qquad F_t(u,v) = \big(f_x(u,v),\; f_y(u,v),\; f_z(u,v)\big)$$

where each location $(u,v)$ carries tangential components $f_x, f_y$ encoding shear and a normal component $f_z$ encoding pressure. For a parallel gripper the left and right fields $F_t^l$, $F_t^r$ may be concatenated into a six-channel tensor — a physically interpretable, gel-appearance-free tactile input for downstream policies.

The force fields are produced directly from raw visuo-tactile images by learning the inverse of the sensing process, a dense mapping

$$\hat{F}_t = g_\theta(T_t), \qquad g_\theta: \mathbb{R}^{H \times W \times 3} \rightarrow \mathbb{R}^{H \times W \times 3}$$

trained on paired calibration data with simulation augmentation (Appendix D). Its output doubles as the supervision target on which the representation model below is built.

NeoForce: the representation model

NeoForce learns a temporally structured visuo-tactile representation on top of the force fields. It receives synchronized chunks of RGB observations and tactile force fields; each modality is patchified into tokens, fused by a shared transformer backbone, and temporal context is aggregated across the chunk, so the representation encodes not only the current contact state but also the recent evolution of contact. Supervision is joint: a reconstruction objective in force space plus a latent-space predictive objective.

flowchart TB
    subgraph INPUT["Input chunk"]
        A1["RGB frames"]
        A2["Tactile force fields F_t"]
    end
    A1 --> P1["visual patch tokens"]
    A2 --> P2["tactile patch tokens"]
    P1 --> BB["shared transformer backbone"]
    P2 --> BB
    BB --> H1["reconstruction head"]
    BB --> H2["latent prediction head"]
    H1 --> R1["predicted force field F-hat"]
    H1 --> R2["contact mask M-hat"]
    H2 --> L1["student predicts teacher latents
visual, tactile, cross-modal, patch"] R1 --> LOSS["L = L_recons + L_pred + lambda_c * L_con"] R2 --> LOSS L1 --> LOSS

Figure 6: NeoForce architecture reconstructed from the paper — RGB frames and force fields are patchified, fused by a shared transformer, and supervised by a reconstruction head (force field + contact mask) and a latent prediction head against teacher latents.

The reconstruction head predicts the force field $\hat{F}$ and contact mask $\hat{M}$ from fused tokens:

$$\mathcal{L}_{\mathrm{recons}} = \lambda_f \left\|(\hat{F} - F) \odot M\right\|_{\mathrm{H}} + \lambda_g \left\|\bar{\hat{F}} - \bar{F}\right\|_{\mathrm{H}} + \lambda_m \mathcal{L}_{\mathrm{mask}}(\hat{M}, M)$$

where $\|\cdot\|_{\mathrm{H}}$ is the Huber norm, $\bar{\cdot}$ the spatial mean (a global force-magnitude term), and $\mathcal{L}_{\mathrm{mask}}$ supervises contact segmentation. The latent prediction head uses a teacher that sees the complete input to produce latent targets, while the student receives masked inputs and predicts them:

$$\mathcal{L}_{\mathrm{pred}} = H(Z_\tau^v, \hat{Z}_s^v) + H(Z_\tau^t, \hat{Z}_s^t) + \lambda_x H(Z_\tau^{v\leftrightarrow t}, \hat{Z}_s^{v\leftrightarrow t}) + \lambda_p H(Z_{\tau,\mathrm{patch}}, \hat{Z}_{s,\mathrm{patch}})$$

with $Z^v$, $Z^t$ the visual and tactile latents, $Z^{v\leftrightarrow t}$ the cross-modal latent (predicting the tactile latent from visual context and vice versa, forcing mutual recoverability), and $Z_{\mathrm{patch}}$ masked patch-level latents encouraging local contact prediction under masking. The complete objective adds a contrastive alignment term between visual and tactile embeddings:

$$\mathcal{L} = \mathcal{L}_{\mathrm{recons}} + \mathcal{L}_{\mathrm{pred}} + \lambda_c \mathcal{L}_{\mathrm{con}}$$

6. Experiment I: Representation Ablation and Cross-Skill Consistency

The evaluation asks two questions: does the force-based representation encode contact intensity over space and time, and does the learned force-space distribution stay consistent when the manipulation task changes? Against a reconstruction-only baseline, adding latent prediction gives:

ObjectiveMAE ↓RMSE ↓mIoU ↑
Reconstruction0.0700.0950.968
+ latent prediction0.0660.0890.966

Table 1: Ablation of the NeoForce training objective on force-field reconstruction (paper Table 2).

Two readings follow. First, the gain concentrates in force magnitude rather than contact localization — mIoU is essentially unchanged while MAE drops from 0.070 to 0.066. Predicting masked patch-level and cross-modal latents forces the model to infer contact at unobserved locations; the surrounding contact area, neighboring frames in the chunk and the visual context over the same region jointly constrain that missing location, driving the representation to convey how contact intensity is distributed in space and time. Second, large-scale training on NeoData gives the shared backbone a strong visual prior over object geometry and material, and cross-modal prediction carries this prior into the tactile tokens, letting the tactile branch estimate intensity more precisely once the contact region is known. Figure 8 of the paper visualizes reconstruction on five contact behaviors — press, twist, grasp, lift, hold: press, twist and grasp yield localized peaks with sharp boundaries and near-zero surroundings, while lift and hold yield broader sustained distributions, and the reconstructed field tracks the ground truth's fine-grained intensity across a wide dynamic range.

Figure 7: NeoForce force-field reconstruction on five contact behaviors (paper Figure 8). Each column, top to bottom: RGB frame, ground-truth force field, NeoForce reconstruction; the colorbar encodes normalized force magnitude.

The paper attributes this cross-skill consistency to the choice of representation space: anchoring on force places the contact effects of different skills into the same space defined by physics rather than by device or behavior — consistency comes from the design, not from skill-specific fitting.

7. Experiment II: NeoReal Real-Robot Evaluation and Tactile Integration Ablation

NeoReal contains 10 real-world contact-rich tasks selected from frequent, tactile-relevant skills in NeoData — Cardboard Box Folding, Bag Packing, Cable Winding, Board Wiping, Cup Stacking, Socket Plugging, Board Insertion, Bottle Standing, Fruit Collection, Towel Folding — each with a standardized initial-state distribution, reset protocol and binary success criterion, executed on robots equipped with the tactile fingers. Baselines span imitation learning (ACT, Diffusion Policy), VLA families ($\pi_{0.5}$, Xiaomi-Robotics-0, InternVLA-A1, StarVLA-$\alpha$), world-action models (Fast-WAM, GigaWorld Policy) and a VA baseline (LingBot-VA); all are pretrained on NeoData then post-trained per task, with 20 randomized rollouts per task. The three strongest policies average: $\pi_{0.5}$ at 26.5% success / 38.1% progressive score, LingBot-VA trailing, Fast-WAM close behind.

Figure 8: The NeoReal suite — 10 contact-rich tasks executed on robots equipped with tactile fingers (paper Figure 9).

The progressive score, which credits partial milestone completion, consistently exceeds the binary success rate — on Socket Plugging $\pi_{0.5}$ moves from 60% success to a 73.5% progressive score, on Bag Packing from 20% to 43.0%. Policies frequently reach intermediate milestones (grasping, alignment) even when the task fails, and the shortfall is most pronounced exactly where touch must act as staged feedback: verifying a secure grasp before lifting, or a seated fit before releasing. Neither vision nor proprioception exposes the contact state to perceive, nor the signal needed to correct an action while it is still in progress.

The integration ablation fixes $\pi_{0.5}$ as backbone and compares four ways of consuming touch:

Variant ($\pi_{0.5}$ backbone)Avg success rateAvg progressive score
No tactile26.538.1
Tactile image concatenation27.541.4
Tactile image action-expert conditioning30.044.3
NeoForce representation action-expert conditioning32.547.5

Table 2: Effect of tactile integration strategy on NeoReal (paper Table 3, average columns).

Every tactile variant beats vision-only. Conditioning the action expert on tactile images beats simple concatenation because the contact signal reaches the module that actually generates actions. The NeoForce force representation improves further over raw tactile images, because it expresses contact as a compact force field rather than as device appearance. Individual tasks fluctuate — raw tactile variants drop on Bag Packing and Cup Stacking before the force representation recovers them — but the aggregate ordering confirms that the gains also appear as more complete milestone progress, not merely as extra binary successes.

8. Experiment III: NeoSim Exposes the Bimanual Contact Bottleneck

NeoSim, built following UniVTAC, renders per-contact force fields in the same $x$, $y$, $z$ format as the unified representation, so policies consuming force-based tactile inputs are evaluated in simulation with the very representation used for real data. The suite has 12 contact-rich tasks — 4 single-arm (Pour Ball, Unplug and Plug Charger, Plug USB, Grasp Chip) and 8 dual-arm (Insert Screw, Place Gears, Unstack Bowl, Cup Handover, Stack Cups, Stack Plates, Stack Bowls, Unstack Cup) — with 100 randomized rollouts per task.

PolicySingle-arm meanDual-arm meanOverall mean
$\pi_{0.5}$68.834.345.8
LingBot-VA36.330.032.1
Xiaomi-Robotics-042.014.123.4
StarVLA-$\alpha$49.310.123.2
GigaWorld Policy27.02.610.8
InternVLA-A123.81.08.6
Fast-WAM0.00.00.0

Table 3: NeoSim single-arm / dual-arm mean success rates, grouped from the 12 per-task values in paper Table 4 (4 single-arm, 8 dual-arm tasks).

Several patterns stand out. First, dual-arm tasks are uniformly harder for every policy, and the drops are severe: StarVLA-$\alpha$ falls from 49.3% to 10.1%, Xiaomi-Robotics-0 from 42.0% to 14.1%, InternVLA-A1 from 23.8% to 1.0%; only $\pi_{0.5}$ and LingBot-VA retain substantial bimanual competence (34.3% / 30.0%), and almost entirely on Plate Stack and Bowl Stack where the two arms can act loosely coupled. Second, tasks demanding sustained mutual contact remain nearly unsolved — the best score is 18% on Place Gears and 25% on Cup Handover, and on each of these two tasks at most two of the seven policies succeed at all. Third, Fast-WAM, a NeoReal top-three policy, fails all twelve NeoSim tasks, indicating that its competence does not transfer to simulated contact dynamics. The paper's reading: vision-only policies cannot see whether a grasp is secure or whether contact is stably maintained — exactly the information bimanual contact-rich manipulation fails on — which strengthens the case for equipping robots with tactile hardware and policies that consume a tactile representation.

NeoSim demonstrations

Figure 9: Demonstrations from NeoSim's 12 tasks (paper Figure 11); left panels show simulated RGB scenes, right panels the particle gel images rendered from the tactile sensors.

The per-task breakdown in Table 4 exposes where current policies actually fail. On single-arm tasks, pi0.5 reaches 92 percent on Pour Ball and 86 percent on Grasp Chip, tasks where force regulation is moderate and visual cues remain informative. The picture changes sharply on dual-arm tasks requiring sustained mutual contact: only Plate Stack (96 percent for pi0.5) and Bowl Stack (93 percent) are solved, precisely because the two arms can act in a loosely coupled manner. Insert Screw, Place Gears, Cup Handover, and Cup Unstack remain near-zero for almost every policy. LingBot-VA is the only model that attempts the dual-arm contact tasks seriously, scoring 20 percent on Cup Stack and 18 percent on Bowl Unstack, suggesting its tactile-aware architecture provides some transfer benefit. The aggregate pattern is unambiguous: whenever success requires verifying that contact is maintained between two effectors, vision-only policies collapse.

9. Limitations

The authors themselves flag three open directions: the unified representation currently targets parallel-jaw visuo-tactile fingers, and extending it to dexterous hands and non-camera-based transducers is needed to broaden applicability; policy learning has not yet been scaled to the full multimodal corpus; and the connection between NeoSim and real deployment needs tightening. Beyond the self-reported limits, three more deserve attention. First, absolute real-robot success is still low — the strongest policy $\pi_{0.5}$ averages only 26.5% on NeoReal, and on Board Insertion and Bottle Standing every tactile integration variant stays in single digits, showing that even with a tactile representation, precision assembly remains hard. Second, the tactile-integration ablation is run on a single $\pi_{0.5}$ backbone; whether the ordering transfers to other policy families is unverified. Third, force-field conversion depends on calibration data plus simulation augmentation (Appendix D), and cross-sensor unification is so far validated within a camera-based sensor family — empirical unification against piezoresistive or capacitive arrays is still missing.

10. Conclusion and Outlook

$\mathcal{N}_0$-Foundation assembles the four pieces of the tactile-intelligence puzzle — sensing hardware, data, representation, evaluation — into a self-consistent loop. Its three empirical claims are: tactile feedback genuinely improves contact-rich manipulation; latent supervision improves temporal tactile representation learning; and the force representation offers a compact physical alternative to raw tactile images. Together with the 5,000-hour open-source OpenNeoData, the work gives the field a reproducible starting point: anyone can download the data, train against the unified force-field format, and compare on NeoReal/NeoSim. The natural next steps are extending the unified representation to dexterous hands and additional sensor types, scaling policy training to the full corpus, and closing the simulation-to-reality gap.

11. Golden Quotes

Policies benefit from the physical contact state rather than from the device-specific appearance of the tactile signal; a sensor-agnostic force field is precisely the universal interface that delivers such a representation.

Contact states, local forces and incipient slip are rarely observable from vision alone, yet they are critical for deformable-object manipulation, precise assembly, delicate force control and sustained surface interaction.

Sources: arXiv:2608.29601 · project page research.neoteai.com/n0-foundation · dataset huggingface.co/datasets/NeoteAIEmbodied/OpenNeoData

Related Papers

Tac4Loco: Learning Spatiotemporal Plantar Pressure Representations for Humanoid Locomotion

Tac4Loco: Learning Spatiotemporal Plantar Pressure Representations for Humanoid Locomotion

Humanoid robots are expected to traverse complex terrains, where the plantar support may vary dramatically due to foot placement errors, ground properties, and transient dynamics. To achieve robust locomotion, the robots are required to adapt to uneven terrain and uncertain foot--ground interactions. Existing locomotion policies rely primarily on proprioception or exteroceptive terrain perception, where the former provides only indirect evidence of plantar support, while the latter predicts contact conditions before touchdown but cannot observe the actual support in real-time. Although some studies incorporate plantar contacts as an auxiliary perception, they rely mainly on summary statistics, overlooking the spatial topology of plantar pressure, which provides a more direct characterization of the realized contact state. To bridge this gap, we present Tac4Loco, a tactile-perceptive framework that incorporates multi-array plantar pressure as direct feedback for humanoid locomotion. We formulate a topology-preserving ordinal representation to map simulated and physical sensor signals into a shared observation space, with a dual-branch encoder for extracting their spatial and temporal representations. Subsequently, the learned spatiotemporal features are integrated with augmented proprioception including terrain estimation cues, and provided to an asymmetric actor-critic architecture for policy learning. Extensive simulation and real-world experiments demonstrate improved tracking performance and support adaptation on terrains with inclined, partial, asymmetric, and changing support. We further demonstrate its zero-shot deployment on unseen compliant and unstructured terrains, including a foam platform and a gravel road. All code and experimental configurations will be released as open-source to facilitate reproducibility.

人形机器人足底压力触觉感知Aug 16, 2026
Towards Artificial Nerves: Biomimetic Optical-Fiber Tactile Sensing for Robots

Towards Artificial Nerves: Biomimetic Optical-Fiber Tactile Sensing for Robots

Robotic systems increasingly demand tactile sensing that approaches the adaptability and resolution of human skin to enable dexterous manipulation and safe interaction. OptiTac is a biomimetic tactile sensor that emulates the mechanoreceptor-to-nerve architecture of human touch by pairing each mechanical pin on a soft skin with an optical fiber acting as an artificial nerve. This design demonstrates an architectural principle for routing tactile information away from the sensing surface while preserving high spatial resolution, establishing a practical route toward distributed tactile sensing in future robotic systems. By treating tactile signals as images, simple analytical methods, rather than opaque deep-learning models, are used to infer contact location, size, and shape, providing interpretable and scalable tactile intelligence. This work demonstrates how evolutionary principles from biology can guide the development of artificial nerve systems for robots, offering a pathway toward human-like tactile perception in next-generation robotic platforms. More broadly, OptiTac establishes an artificial nerve-inspired sensing framework for interpretable robotic touch and a scalable route toward future distributed tactile systems.

触觉感知tactile sensingOptiTacJul 17, 2026
BayesContact: Uncertain Pose Estimation via Visuo-Tactile Proposals and Simulation-based Inference

BayesContact: Uncertain Pose Estimation via Visuo-Tactile Proposals and Simulation-based Inference

Contact-rich manipulation requires pose estimates that are often more accurate than what depth-only sensing provides. Existing methods, relying on vision and contact, employ costly offline training procedures that need to be retrained for new environments and geometries. We propose BayesContact, a Simulation-Based Inference framework for visuo-tactile pose estimation in peg-in-hole insertion. BayesContact maintains a particle belief over object pose and fuses depth observations with force/torque-derived contact evidence. We employ simulation based forward models to approximate these observation likelihoods. For each pose hypothesis, a renderer predicts depth measurements and a physics simulator predicts contact outcomes under guarded probing actions; both are scored against real observations to update the belief. The resulting multimodal belief also enables information-gain-based probing for active disambiguation. Across simulated geometries and real-robot experiments, BayesContact improves pose observability and insertion success over vision-only inference by 30%

触觉感知tactile姿态估计Jul 17, 2026
VTLoc: Learning-based Tactile Contact Localization in Visual Point Clouds

VTLoc: Learning-based Tactile Contact Localization in Visual Point Clouds

Vision and touch are complementary modalities essential for robotic perception and manipulation. While vision provides global object context, touch offers precise local information at contact points. Integrating these modalities for contact localization, i.e., predicting the location of touch on an object's surface, poses significant challenges due to the need for accurate spatial alignment between tactile data and visual geometry. To address this challenge, we propose VTLoc, a novel visual-tactile framework that localizes contact points from tactile readings using a 3D point cloud as visual input. VTLoc introduces two key components: a geometric multi-modal alignment module, which reconstructs a pseudo-point cloud from fused visual-tactile features and aligns it with the visual point cloud to enforce spatial consistencies across modalities; and an iterative localizing updater, which iteratively refines the predicted contact location using fused visual-tactile features. Evaluated on a new benchmark of 100 real-world objects, VTLoc improves single-touch contact localization by reducing local-to-global correspondence ambiguity.

触觉感知tactile视觉Jul 17, 2026