PAPER DEEP DIVE
DexPoint uses point cloud representations for generalizable dexterous manipulation without CAD models, showing strong sim-to-real transfer.
DexPoint uses point cloud representations for generalizable dexterous manipulation without CAD models, showing strong sim-to-real transfer.
SOURCE LINKS
Most robotic grasping research picks a pose from vision and closes a parallel gripper, while humans rely on fingertip touch and almost never drop things. The LASR Lab at TU Dresden reframes the question as "is this multi-fingered grasp stable before we lift?" and answers it with data. They mount four Digit 360 sensors on the fingertips of a 16-DoF Tilburg Hand on a 7-DoF xArm7 and automatically collect 10,000 grasp trials over 200 objects (105 rigid, 95 deformable, 1.9-244 g), recording external RGB-D, proprioception and four tactile streams (camera, audio, IMU, pressure) throughout each grasp. Stability labels come from SAM 3 measuring post-lift object height and agree with manual verification on 92.34% of trials. The predictor consumes a 3-second window anchored at grasp initiation: 16-frame RGB and tactile clips go through pretrained VideoMAEv2-Base, 100-step proprioception and 224-step audio/IMU/pressure go through Transformers, fingertip features are fused across modalities, time and fingers by weight-shared factorized attention, and everything is aggregated by attention fusion into a post-lift stability probability. Under 5-fold object-disjoint cross-validation, vision+proprioception+touch reaches 83.55% and removing touch costs 3.99 points (79.56%). Tactile spatial resolution matters monotonically (79.40% at 1x1 up to 83.55% at 112x112), and freezing touch to its final frame costs 2.07 points, so the useful signal is high-resolution and dynamic. Deployed as an online lift-or-regrasp gate (threshold 0.95, up to five regrasps) on 20 objects excluded from training, the visuo-tactile gate reaches 82.0% success among executed lifts, 10.5 points above a non-tactile gate. The ~1.2 TB dataset is public on OPARA.
WB-WAM from Tsinghua IIIS, Xiong'an Institute of AI and University of Melbourne (Hang Zhao group) injects explicit whole-body action supervision into generative video pre-training: a shared 72-D physical action space (body 29 + root 3 + two hands 20+20) unifies partial annotations from 1,880.2 hours of nine-source heterogeneous video and motion data via masked flow matching, followed by PICO egocentric mid-training (22 h / 73 tasks, GMR retargeting + constrained IK + MINT) and real-robot post-training with a forward-kinematics loss on Unitree G1 with Wuji hands. It wins all seven HumanoidArena tasks (81.9% mean), reaches 84.0% on five real tasks vs OpenWAM's 80.0% (ACT 20%, Fast-WAM 6%), and PICO mid-training lets 30 real demos hit 73.8% — beating 100-demo direct training at 65.0%, a 70% cut in robot data.
Large-scale pre-training has made robot policy fine-tuning increasingly data-efficient, but this progress has largely been driven by datasets and embodiments built around simple parallel-jaw grippers. Dexterous, multi-fingered hands remain comparatively data-starved because real teleoperation is costly to scale, while human hand video is off-embodiment and requires lossy pose estimation and retargeting. We introduce Simulation Pre-training for Dexterity (SPD), a pre-training framework for dexterous manipulation that uses data entirely collected in simulation. In SPD, humans manipulate virtual objects inside a VR headset, enabling on-embodiment trajectories and robot-free collection. With the help of five operators, we collect 75 hours of multi-task dexterous manipulation over one week, and use it to pre-train a causal transformer on a sequence modeling objective. We study the benefits of simulation pre-training on real-world tasks by fine-tuning on 1-2 hours of physical demonstrations on a 56-DoF bimanual dexterous setup. We find that our approach outperforms training behavior cloning policies from scratch, showing that simulation teleoperation is a viable pre-training source for real-world dexterous manipulation. We perform ablation studies, measuring the benefits of history conditioning and short action chunks for reactive control.
This paper presents a tactile-reactive gripper that integrates a Visuo-Tactile Active Palm (VTAP) and compliant, reconfigurable fingers equipped with tactile array sensors. The design exploits structured finger-palm synergy and multi-modal perception to achieve both robust grasping and fine manipulation. The actuated bi-modal palm seamlessly combines long-range visual localization with contact-rich tactile feedback, substantially extending the system's manipulation capability. To bridge the embodiment gap between human hand motion and the heterogeneous three-finger structure, we further propose a staged, gesture-conditioned retargeting framework for dexterous teleoperation. Extensive experiments validate the system across a range of challenging tasks: reactive grasping of YCB and fragile objects, in-hand syringe reorientation and plunger actuation, singulation of clustered objects down to 3 mm in diameter, and vision-tactile peg-in-hole insertion. Results demonstrate that high manipulation performance can be achieved through coordinated finger-palm interaction and multi-modal sensing, without resorting to high degrees of freedom anthropomorphic designs. The VTAP gripper and its retargeting framework offer a practical reference architecture for dexterous gripper design, manipulation, and contact-rich data collection in support of learning-based approaches. Project webpage: https://yuhochau.github.io/vtap/.
As an Amazon Associate, we earn from qualifying purchases.