VTAP Gripper: Synergizing Fingertip Sensing and a Visuo-Tactile Active Palm for Dexterous In-Hand Manipulation

In one sentence: This paper presents VTAP — a gripper integrating a Visuo-Tactile Active Palm with compliant reconfigurable fingers equipped with tactile array sensors, achieving robust grasping and fine manipulation through finger-palm synergy and multi-modal perception without high-DOF anthropomorphic designs.

1. Background and Motivation

Robotics is transitioning from structured industrial deployments to consumer-facing services requiring operation in dynamic, semi-structured human environments. Industrial manipulation is typically predefined pick-and-place within controlled work cells, while real-world human environments involve diverse objects, clutter, and varying contact conditions requiring post-grasp actions such as in-hand reorientation and precise alignment.

Parallel grippers are mechanically simple and easy to control, but their limited reconfiguration restricts dexterous tasks. Anthropomorphic hands offer high dexterity but are bulky, expensive, and difficult to control. Multi-finger grippers balance actuation complexity and dexterity, especially with tactile feedback, but face two key challenges:

• Neglected palm interaction: Existing grippers concentrate sensing and actuation on fingers, ignoring palm-object contact that is critical in many grasping and manipulation tasks. An active palm addresses this by enabling controllable object manipulation, particularly in-hand scenarios requiring coordinated contact redistribution across fingers and palm.
• Single modality insufficiency: Relying solely on vision or tactile sensing is often insufficient for comprehensive environmental perception. Integrating tactile and visual sensing in a unified framework mitigates occlusion and enhances robustness, but compactly incorporating fingertip tactile, palm tactile, and visual perception into one gripper remains challenging.

2. Hardware Design

2.1 Finger Kinematics

VTAP uses a three-finger structure, each finger based on the Fin-Ray compliant principle with four joints $q_1$ through $q_4$. The forward kinematics is described by a series of homogeneous transformation matrices:

$$\begin{cases}{}^{1}_{\mathcal{O}}\mathbf{T}=\mathbf{D}_{\mathbf{p}_1}\mathbf{R}_Z(\phi)\mathbf{R}_Y\left(\frac{\pi}{2}\right)\mathbf{R}_X(-q_1)\\ {}^{2}_{1}\mathbf{T}=\mathbf{D}_{\mathbf{l}_1}\mathbf{R}_Z\left(-\frac{\pi}{2}\right)\mathbf{R}_X\left(\frac{\pi}{2}+q_2\right)\\ {}^{3}_{2}\mathbf{T}=\mathbf{D}_{\mathbf{l}_2}\mathbf{R}_Z\left(-\frac{\pi}{2}\right)\mathbf{R}_X(-q_3)\\ {}^{4}_{3}\mathbf{T}=\mathbf{D}_{\mathbf{l}_3}\mathbf{R}_Z\left(\frac{\pi}{2}\right)\mathbf{R}_X(q_4)\\ {}^{\mathcal{E}}_{4}\mathbf{T}=\mathbf{D}_{\mathbf{l}_4}\end{cases}$$

where $\mathbf{D}_{\mathbf{l}_i}$ represents translation along link vector $\mathbf{l}_i$, $\mathbf{R}_X$, $\mathbf{R}_Y$, $\mathbf{R}_Z$ denote rotations about corresponding axes, $\mathbf{p}_1$ is the base offset, and $\phi$ is the initial azimuthal orientation. The three fingers are arranged around the palm, each spanning a 120° sector: $[-60°, 60°]$, $[60°, 180°]$, $[180°, 300°]$.

The reachable workspace is estimated via Monte Carlo sampling (5000 random configurations per joint), spanning approximately 469mm along X, 477mm along Y, and 311mm along Z. The symmetric 120° finger distribution ensures uniform reachability in all directions, while the Fin-Ray compliant structure enables automatic enveloping adaptation to object geometry upon contact.

2.2 Multi-Modal Perception

Fingertip tactile arrays: Thin piezoresistive FlexiTac sensors provide $32 \times 12$ taxels over a $66 \times 25\,\text{mm}^2$ active area, with effective spatial resolution of approximately $2 \times 2\,\text{mm}^2$. Pressure-induced resistance changes transduce mechanical stimuli into measurable signals, enabling instantaneous pressure distribution sensing for force-controlled grasping and slip detection.

Visuo-Tactile bi-modal palm: A USB camera (GC0307, GalaxyCore) with 50° FOV is enclosed in a 50mm diameter cylindrical housing with a detachable sensing cartridge providing 40mm diameter sensing area. The cartridge consists of a 4mm acrylic substrate and 5mm silicone elastomer layer with mirror-effect coating. A multi-color LED ring (red, green, blue, white) provides controllable illumination. When LEDs are off, the module is transparent for external scene imaging; when activated, illumination reflects off the deformable surface, transforming the module into an optical tactile sensor. This enables seamless switching between vision and tactile sensing without mechanical reconfiguration.

3. Retargeting Framework for Three-Finger Gripper

Non-anthropomorphic grippers face a substantial embodiment gap preventing direct human-to-robot retargeting. VTAP proposes a staged, gesture-conditioned retargeting framework using VR (Meta Quest 3) to capture human hand motion and map it to the three-finger structure.

3.1 Subspace Reduction and Gesture Conditioning

The framework constrains radial-ulnar deviation to three representative primitives: cage, power, and pinch grasps. Left-hand palm open-close gestures switch modes by updating each finger's first joint $q_1$. The adduction/abduction DOF is fixed to prevent kinematic ambiguity. An intermediate coordinate frame $\mathcal{I}$ is defined by applying a constant $SE(3)$ transformation to the VR-captured wrist frame $\mathcal{W}$.

3.2 Optimization Objectives

Three primary objectives are defined for the remaining MCP and PIP DOFs:

Fingertip position loss — mitigating geometric discrepancies via the intermediate base frame:

$$\mathcal{L}_{\text{pos}}=\sum_{i=1}^{3}\left\|\mathbf{v}_{i}^{g}-\mathbf{v}_{i}^{I}\right\|_{2}^{2}$$

Fingertip orientation loss — using PIP-to-fingertip vectors:

$$\mathcal{L}_{\text{rot}}=\sum_{i=1}^{3}\left\|\mathbf{r}_{i}^{g}-\mathbf{r}_{i}^{I}\right\|_{2}^{2}$$

Smoothness regularization:

$$\mathcal{L}_{\text{vel}}=\sum_{j=1}^{6}w_{j}\left\|q_{j}^{t}-q_{j}^{t-1}\right\|_{2}^{2}$$

Total objective:

$$\mathcal{L}=\lambda_{1}\mathcal{L}_{\text{pos}}+\lambda_{2}\mathcal{L}_{\text{rot}}+\mathcal{L}_{\text{vel}}$$

Solved at each timestep subject to joint limits:

$$\mathbf{q}_{t}^{*}=\arg\min\mathcal{L}(\mathbf{q}_{t}),\quad\text{s.t.}\quad\mathbf{q}_{\min}\leq\mathbf{q}_{t}\leq\mathbf{q}_{\max}$$

A Huber-type robust penalty function is applied across all sub-terms for numerical stability against tracking noise. VR capture at ~25Hz is temporally upsampled to 100Hz control stream via interpolation.

3.3 Singulation Retargeting

Singulation requires isolating a single object from clustered items, demanding precise fingertip spacing and relative orientation control. The framework uses the reconfigurable pinch mode, mapping only human thumb and index to two active gripper fingers. Two geometric cues are monitored: the angle between thumbtip-MCP and index fingertip-PIP vectors, and the Euclidean distance between fingertips. When vectors approach parallel alignment and distance falls below threshold, the controller enters singulation mode. Fingertip distance linearly maps to equal-magnitude opposite-direction actuation of $q_3$ joints; MCP flexion $q_2$ is regulated via thumb-middle distance mapping, enabling coordinated control of two active DOFs per finger.

4. Tactile-Reactive Grasping

The gripper is mounted on a UR5e arm. A threshold-based tactile stopping criterion enables adaptive grasping:

$$Q=\sum_{i=1}^{3}\left[\alpha\,\Delta C_{f}+(1-\alpha)\sum_{m,n}S_{f}(m,n)\right],\quad Q>T_{\text{th}}$$

where $\Delta C_{f}$ is the change in non-zero contact pixels, $\sum_{m,n}S_{f}(m,n)$ is the cumulative signal magnitude of tactile array taxels on the $i$-th finger, and $\alpha \in [0,1]$ balances contact area and signal intensity. Each finger continues flexion until $Q$ exceeds the threshold $T_{\text{th}}$. The palm camera estimates object orientation via morphological processing and Canny edge detection, adjusting the end-effector to an orientation-aligned grasp pose before the tactile-reactive controller activates.

Overall grasping success rate is 93.3%. Tactile sensors enable damage-free grasping of fragile objects (bulbs, chips), while power mode and the actuated palm support stable handling of large, wide, and thin cylindrical objects. In-hand manipulation experiments show a 40mm sphere rotating about X, Y, Z axes with approximately ±15° (X/Y) and ±20° (Z), demonstrating multi-axis in-hand reorientation capability.

5. Syringe Re-orientation and Plunger Actuation

Inspired by a laboratory pipetting scenario: extract a syringe from a stand and depress the plunger to transfer liquid into a beaker. The radial-ulnar DOF is configured for pinch grasping; thumb-index closure executes a two-finger pinch grasp. Tactile arrays capture contact distribution and in-hand orientation. Middle finger contraction triggers dexterous in-hand reorientation, aligning the syringe outlet downward. Little finger contraction drives the actuated palm to depress the plunger. Over 20 consecutive teleoperation trials, success rate is 65% (13/20) with average completion time $33.4 \pm 5.43$ seconds, demonstrating continuous multi-stage in-hand manipulation from grasping through reorientation to actuation while maintaining stable object holding.

6. In-Hand Singulation and Peg-in-Hole Assembly

In-hand singulation: Isolating a single object from clustered items down to 3mm diameter. The pinch mode with thumb-index coordinated rolling/shearing motions continuously adjusts inter-fingertip distance and angular alignment. The $2 \times 2\,\text{mm}^2$ tactile resolution suffices to distinguish and separate millimeter-scale objects.

Visuo-tactile peg-in-hole insertion: Combines the palm's visual modality for long-range localization and pose estimation with fingertip tactile feedback for fine alignment during contact. The palm camera estimates object orientation; the tactile-reactive controller provides closed-loop force feedback ensuring precise centering and force-controlled insertion.

7. Method Flowchart

8. Limitations

• Teleoperation dependency: The system is currently VR-teleoperation-driven, not autonomous. While it provides a data collection architecture for learning-based approaches, autonomous dexterous manipulation requires additional policy learning.
• Limited syringe success rate: 65% success indicates multi-stage in-hand manipulation remains challenging, particularly continuous action sequences involving reorientation and plunger actuation.
• Drill grasping difficulty: Only 5/10 success for drills reflects insufficient grasping strategy for specific geometries (long handle + large head).
• Limited rotation range: In-hand reorientation of only ~±15°–20° falls short of large-angle reorientation, limiting continuous rotation scenarios.

9. Conclusion

VTAP achieves high dexterous manipulation performance through the synergy of a visuo-tactile bi-modal active palm and compliant reconfigurable fingers with tactile arrays, without relying on high-DOF anthropomorphic designs. The staged gesture-conditioned retargeting framework effectively bridges the embodiment gap between human hands and the heterogeneous three-finger structure. Experiments validate reactive grasping of YCB and fragile objects (93.3% success), syringe in-hand reorientation and plunger actuation, singulation of objects down to 3mm, and visuo-tactile peg-in-hole assembly. VTAP offers a practical reference architecture for dexterous gripper design, manipulation, and contact-rich data collection in support of learning-based approaches.