Towards Human-level Dexterous Teleoperation

Paper Metadata

Title: Towards Human-level Dexterous Teleoperation (TeleDexter)

Authors: Puhao Li (Tsinghua University, BIGAI), Zeyuan Chen (BIGAI, Peking University; equal contribution, corresponding), Yingying Wu (Tsinghua University, BIGAI), Pengkun Wei (BIGAI), Yuyang Li (BIGAI, Peking University; equal contribution, corresponding), Tianyu Wang (BIGAI, Peking University; equal contribution, corresponding), Jiaxiao Shi (BIGAI), Mingrui Yu (Tsinghua University), Baoxiong Jia (BIGAI), Song-Chun Zhu (Tsinghua, BIGAI, PKU), Tengyu Liu (BIGAI), Siyuan Huang (BIGAI)

Links: arXiv:2607.11481 (https://arxiv.org/abs/2607.11481 , v1, 13 Jul 2026, primary cs.RO); project blog https://bigai-dex.github.io/blog/teledexter

Code and data status: partially released. The repository https://github.com/ice4133/TeleDexter ships the two-stage retargeting chain: scripts/run_kinematic_retargeting.py runs stage-one vector-alignment retargeting, scripts/run_interaction_retargeting.py refines it with mesh-aware optimization, the stages are decoupled by kinematic_motion.h5, and stage two emits interaction_motion.h5 containing wrist-frame five-fingertip targets plus object 6D pose, exactly the co-tracking goals consumed by RL; the repo defaults to the WUJI Hand 2 Beta 2 right hand. The single-stage RL controller itself and its training code are not released.

Figure 1: TeleDexter learns diverse dexterous in-hand manipulation skills within a single-stage framework (paper Figure 1). Top left: the teleoperation interface with hand and object targets; bottom left: four long-horizon tool tasks (screwdriver tightening, brush sweeping, bulb install/uninstall, hammer nailing); right: diverse object and tool configurations.

One-Sentence Summary

TeleDexter reframes dexterous teleoperation from "map the human hand joints" to "co-track hand and object": the operator specifies only synchronized subgoal sequences of fingertip positions and object pose, a single-stage reinforcement-learning controller discovers how to realize them through real contact physics, and with random action masking plus domain randomization the controller transfers zero-shot to hardware, reaching 75.2% average success and 87.1% average task progress across seven in-hand reorientation and long-horizon tool-use tasks where every baseline nearly uniformly fails, while the collected demonstrations behavior-clone into autonomous policies.

Background and Motivation

Everyday human manipulation continuously coordinates three families of contact transition: in-hand translation, in-hand reorientation, and finger gaiting. The most direct route to giving robot hands this agility is teleoperation, with the operator teaching the robot in the loop. Yet existing dexterous teleoperation systems remain far from human-level in-hand dexterity, and the paper attributes the gap to structural defects in the two dominant paradigms.

The first paradigm is kinematic retargeting: vision-based tracking or exoskeleton gloves capture the human hand pose, which is then mapped onto the robot joint topology. The interface is intuitive, low-latency, and faithful to the operator's kinematic intent, but it ignores hand-object contact forces and object inertia entirely. High-acceleration maneuvers, non-prehensile interactions, and continuous finger gaiting therefore become highly unstable, with object slippage and drops as the default failure mode. The second paradigm, exemplified by DexGen, learns a dexterous action prior in simulation to map coarse teleoperation commands to contact-rich hand actions. It improves local contact robustness, but its training targets are synthetically generated grasp transitions with no physical-feasibility guarantee, and encoding the prior in a generative model introduces cascading trajectory drift and covariate shift during closed-loop execution, severely degrading long-horizon real-world performance.

The paper's entry point is a redistribution of responsibility: the operator declares what the hand and object should achieve, namely synchronized sequences of fingertip position targets and object pose targets; a learned low-level controller handles how to achieve them, i.e. the multi-contact dynamics trained in simulation. This casts teleoperation as hand-object co-tracking, supported by three technical designs: consecutive subgoal co-tracking, geometry-aware retargeting that constructs physically grounded reference motions, and zero-shot sim-to-real robustness centered on random action masking.

Relative to frame-wise trajectory imitation, the decisive property of consecutive subgoal co-tracking is the search space it leaves for contact strategies: the policy must reach each hand-object subgoal in order, but the contact transitions between subgoals are discovered by the policy itself during simulation rollouts. The paper argues that prior reference-motion-based dexterous manipulation learning fails to cover dynamic skills such as reorientation and gaiting precisely because dense frame-wise tracking is overly restrictive, locking single-stage policy learning into basic grasping and wrist motion.

Preliminaries

All quantities are expressed in the wrist frame $\mathcal{F}_{\mathrm{wrist}}$; the robot arm tracks the human wrist pose independently via inverse kinematics. Given hand-object reference motions, the goal is a co-tracking policy $\boldsymbol{a}_{t}=\pi_{\theta}(\boldsymbol{o}_{t},\,g_{t})$ where the observation $\boldsymbol{o}_{t}$ encodes the current hand-object state, the action $\boldsymbol{a}_{t}\in\mathbb{R}^{n_{\mathrm{dof}}}$ is a target joint position, and the co-tracking goal prescribes both fingertip positions and object pose:

$$g_{t}=\bigl(\hat{\boldsymbol{p}}^{\mathrm{tip}}_{t},\;\hat{T}^{o}_{t}\bigr),\quad \hat{\boldsymbol{p}}^{\mathrm{tip}}_{t}\in\mathbb{R}^{N_{f}\times 3},\quad \hat{T}^{o}_{t}=\bigl(\hat{\boldsymbol{x}}^{o}_{t},\,\hat{R}^{o}_{t}\bigr)\in SE(3)$$

During teleoperation $g_{t}$ is built from real-time captured hand-object poses, so dexterous teleoperation and co-tracking become formally identical problems. The observation $\boldsymbol{o}_{t}$ concatenates current hand joint positions $\boldsymbol{q}_{t}$, object pose $(\boldsymbol{x}^{o}_{t},R^{o}_{t})$, the gravity direction in the wrist frame, and the previous action $\boldsymbol{a}_{t-1}$: 142 dimensions for SharpaWave and 112 for LeapHand. Two design choices deserve emphasis. Every other entry lives in the wrist frame, so the gravity direction is the policy's only world-frame anchor, telling it the absolute orientation of the wrist. Joint velocities and contact forces are deliberately excluded because both are noisy or unavailable on hardware; the policy implicitly recovers them from kinematic state, previous action, and the gravity cue.

The evaluation protocol runs 15 trials per method per task, with each task decomposed into well-defined stages (Figure 3). Success rate (SR) is the fraction of trials completing all stages; task progress (TP) is the mean fraction of stages completed per trial, each stage weighted equally. A trial terminates on unrecoverable grasp loss or object drop. This protocol makes "where did it fail" readable information rather than a binary outcome.

Method

3.1 Consecutive Subgoal Co-tracking and the Hybrid Reward

Each reference trajectory is converted into a sequence of co-tracking subgoals sampled at varying intervals. For subgoal $g_{k}=(\hat{\boldsymbol{p}}^{\mathrm{tip}}_{k},\hat{T}^{o}_{k})$, the per-finger, object-position, and object-rotation tracking errors are

$$e^{f}_{t,k}=\left\|\boldsymbol{p}^{\mathrm{tip}}_{t,f}-\hat{\boldsymbol{p}}^{\mathrm{tip}}_{k,f}\right\|_{2},\quad e^{\mathrm{pos}}_{t,k}=\left\|\boldsymbol{x}^{o}_{t}-\hat{\boldsymbol{x}}^{o}_{k}\right\|_{2},\quad e^{\mathrm{rot}}_{t,k}=\left\|\mathrm{Log}\!\left(\hat{R}^{o\top}_{k}R^{o}_{t}\right)\right\|_{2}$$

A subgoal counts as reached when, for $N_{\mathrm{stay}}\sim\mathcal{U}\{5,15\}$ consecutive frames, all of $\max_{f}e^{f}_{t,k}<\epsilon_{\mathrm{tip}}=3$ cm, $e^{\mathrm{pos}}_{t,k}<\epsilon_{\mathrm{pos}}=1$ cm, and $e^{\mathrm{rot}}_{t,k}<\epsilon_{\mathrm{rot}}=10^{\circ}$ hold simultaneously. Randomizing the dwell length prevents the policy from learning a graze-and-leave exploit. The reward couples sparse subgoal reaching, dense tracking, and a time penalty:

$$r_{t}=\underbrace{\mathbbm{1}_{\mathrm{reach}}(t)\,w_{\mathrm{step}}(t)\,r_{\mathrm{score}}(t)}_{\text{sparse subgoal}}+\underbrace{\alpha_{\mathrm{dense}}\,r_{\mathrm{dense}}(t)}_{\text{dense tracking}}-\underbrace{c_{\mathrm{time}}}_{\text{time}},\quad \alpha_{\mathrm{dense}}=0.1,\ c_{\mathrm{time}}=0.1$$

The subgoal score measures how well hand and object match the active subgoal as a weighted sum of exponential kernels over per-finger and object errors:

$$r_{\mathrm{score}}=\alpha_{s}\Bigl[w^{s}_{\mathrm{tip}}\sum_{f\in\mathcal{F}}\rho_{f}+w^{s}_{\mathrm{obj}}\,(\rho_{\mathrm{pos}}+\rho_{\mathrm{rot}})\Bigr],\quad \rho_{f}=\exp\!\bigl(-\beta_{f}\,e^{f}_{t,k}\bigr)$$

with outer scale $\alpha_{s}=1.5$, $w^{s}_{\mathrm{tip}}=0.5$, $w^{s}_{\mathrm{obj}}=2.0$. The decay rates $\beta$ differ per error type: thumb 100, other fingers 90, MCP root-knuckle level 50, medial-knuckle level 40, object position 80, object rotation 3 (rotation error is in radians, hence the small rate). The object weight being four times the fingertip weight reflects that object pose, not finger pose, is the final arbiter of task success in co-tracking. The dense term $r_{\mathrm{dense}}$ reuses the same per-finger and object kernels at every timestep, but its object part is gated by $(1-\sigma_{t})$: early in training the curriculum factor $\sigma_{t}$ is small and the dense object signal is off, ramping in as the curriculum hardens so that unreachable object targets do not suppress early exploration.

The step weight $w_{\mathrm{step}}$ is an easily missed but essential anti-exploit detail. After subgoal $g_{k}$ is hit, the next subgoal is drawn from the same reference trajectory at index $k^{\prime}=k+\Delta k$, with the range of $\Delta k$ expanding under the curriculum; setting $w_{\mathrm{step}}=|\Delta k|+5$ makes larger jumps yield proportionally larger rewards so the policy is not biased toward exploiting trivially close subgoals. On a cross-trajectory switch, $w_{\mathrm{step}}$ takes a fixed large value of 100 to compensate the distribution jump.

3.2 Curriculum, Resets, and Random Action Masking

The curriculum advances along three difficulty axes: gravity anneals from 0 to $-9.8$ over roughly 32K environment frames to ease initial contact establishment; the subgoal tolerances $(\epsilon_{\mathrm{tip}},\epsilon_{\mathrm{pos}},\epsilon_{\mathrm{rot}})$ start permissive and tighten progressively; and the inter-subgoal step $\Delta k$ grows from a maximum of 40 to 80 reference frames at the 60 Hz control rate, so the policy first masters fine-grained local tracking and later handles longer-horizon goal jumps. The dense-reward object gate $\sigma_{t}=1-(1-0.7)\min(t/25600,1)$ and the action-mask duration growing from 1 to 10 frames belong to the same schedule.

Resets cooperate with the curriculum: with probability 0.9 the environment continues in the same trajectory at $k+\Delta k$, and with probability 0.1 it switches across trajectories; after a switch the reward is flattened for 300 frames, failed resets carry penalty factor $\eta_{\mathrm{fail}}=1.5$, and velocity-bound violations or object position errors beyond 15 cm terminate the episode. The first 90% of reference-motion frames serve random-start initialization, ensuring the late segments of long horizons receive sufficient visits.

Random action masking is the single most impactful sim-to-real intervention in the paper. Given the policy action $\boldsymbol{a}_{t}$, a binary mask $\boldsymbol{m}_{t}\in\{0,1\}^{n_{\mathrm{dof}}}$ is sampled and the executed command is

$$\tilde{\boldsymbol{a}}_{t}=\boldsymbol{m}_{t}\odot\boldsymbol{a}_{t}+(1-\boldsymbol{m}_{t})\odot\tilde{\boldsymbol{a}}_{t-1}$$

i.e. with probability 0.15 three degrees of freedom are chosen and their joints are frozen at the previous command for a randomly sampled duration. This forces the policy to recover useful contact configurations even when subsets of joints respond late or not at all, directly matching the dominant hardware failure modes (motor lag, backlash, occasional missed commands in the SharpaWave SDK). The action parameterization follows HORA's residual joint targets with a soft deadzone, $\boldsymbol{a}^{\mathrm{cmd}}_{t}=\mathrm{clip}\bigl(\boldsymbol{a}^{\mathrm{cmd}}_{t-1}+0.1\cdot\mathrm{deadzone}_{0.1}(\boldsymbol{a}_{t})\bigr)$, giving the Gaussian policy an explicit hold-still region instead of requiring it to emit exactly zero.

3.3 Geometry-Aware Retargeting: From Mocap to Reference Motions

Reference motions come from motion-capture recordings of human hand-object interaction spanning three categories: in-hand translation, in-hand rotation, and free-play combining arbitrary grasps, finger gaiting, and tool-use sequences. Conversion proceeds in two stages. Stage one follows vector-based retargeting, optimizing robot joint angles $\boldsymbol{q}_{1:T}$ to match human hand geometry via directional and inter-finger vector alignment. Stage two refines the result with a geometry-aware optimization that incorporates the object mesh:

$$\boldsymbol{q}^{*}_{1:T}=\arg\min_{\boldsymbol{q}_{1:T}}\sum_{t=1}^{T}\left(\mathcal{L}_{\mathrm{vec}}^{t}+\lambda_{\mathrm{surf}}\mathcal{L}_{\mathrm{surf}}^{t}+\lambda_{\mathrm{pen}}\mathcal{L}_{\mathrm{pen}}^{t}+\lambda_{\mathrm{col}}\mathcal{L}_{\mathrm{col}}^{t}\right)+\lambda_{\mathrm{smooth}}\mathcal{L}_{\mathrm{smooth}}$$

The surface-attraction term $\mathcal{L}_{\mathrm{surf}}^{t}=\frac{1}{|\mathcal{S}_{t}|}\sum_{p\in\mathcal{S}_{t}}\mathrm{ReLU}(\mathrm{sdf}_{\mathcal{O}_{t}}(p))$ pulls near-contact hand points ($\mathcal{S}_{t}=\{p:\mathrm{sdf}<\tau_{\mathrm{surf}}=2\}$ mm) toward the object surface; the penetration penalty with $\lambda_{\mathrm{pen}}=2$ acts on $\mathrm{ReLU}(-\mathrm{sdf})$; the cross-finger collision term uses $\lambda_{\mathrm{col}}=10$ over fingertip spheres; $\lambda_{\mathrm{smooth}}=0.1$ applies a temporal smoothness energy to suppress capture jitter; differentiable SDF queries come from Kaolin. Stage one runs Adam at learning rate 1e-3 for 6000 steps, stage two at 3e-3 for 160 steps with gradient clipping at 1.0. The output $\{(\boldsymbol{q}^{*}_{t},T^{o}_{t})\}$ gives reference motions in the wrist frame, and the co-tracking goals are simply $\hat{\boldsymbol{p}}^{\mathrm{tip}}_{t}=\mathrm{FK}_{\mathrm{tip}}(\boldsymbol{q}^{*}_{t})$ and $\hat{T}^{o}_{t}=T^{o}_{t}$. In the released repository, object_aware.py implements stage two with exactly these $\lambda$ weights and kinematic/pipeline.py implements stage one.

Figure 2: Method overview (paper Figure 2). Left: hand-object interaction reference motions. Middle: the single-stage RL loop with consecutive subgoal co-tracking, hybrid reward (sparse reaching plus dense tracking), action masking (86% on the real robot), curriculum (gravity, failure tolerance, subgoal step), and in-trajectory plus cross-trajectory resets. Right: zero-shot deployment with the operator driving the robot hand and object through the TeleDexter policy.

3.4 Single-Stage Training and Real-World Deployment

Each controller is trained in a single RL stage with no staged skill decomposition and no task-specific reward engineering. Simulation uses Isaac Gym with all reference motions for one object (about 50 minutes at 30 Hz, 150 trajectories per object, captured with a NOKOV mocap system) loaded simultaneously. The optimizer is SAPG (6 attention blocks, LSTM 512, MLP [512,1024,1024,512,512] with ELU, learning rate 2e-4, $\gamma=0.99$, $\lambda=0.95$) on 4 RTX 5090 GPUs with 15,600 parallel environments per GPU, 62,400 in total, converging within roughly $10^{10}$ environment steps, about one day. Domain randomization covers object and hand shape and dynamics properties, random external forces on the object, observation noise, and latency.

At deployment a real-time system captures the operator's wrist, fingertip, and object poses: the arm tracks the wrist via IK while fingertip and object poses form the co-tracking goal for the policy. For contact initialization, kinematic retargeting handles pre-grasp positioning; once stable contact is established the operator switches to the co-tracking controller for dexterous manipulation. This kinematics-then-learned switch is the only point in the human-machine interface that requires operator judgment.

Experimental Results

The hardware is a Franka FR3 arm with two dexterous hands: the 16-DoF four-finger LeapHand and the 22-DoF five-finger human-like SharpaWave. Seven tasks span two families: three in-hand reorientation tasks (CylinderReorient, CuboidReorient, BunnyReorient) and four long-horizon tool-use tasks (HammerUse, ScrewdriverUse, BrushSweep, BulbReplace). In the main results, TeleDexter averages 75.2% SR and 87.1% TP across the seven tasks; kinematic retargeting baselines DexRT and GeoRT average 5.7/37.6 and 0/28.5, the learned-prior baseline DexGen averages 0/25.0, and the non-teleoperation SimToolReal variants reach only 8.9/28.9 and 6.7/20.6.

Task (SharpaWave)	DexRT	GeoRT	DexGen	SimToolReal†	SimToolReal‡	TeleDexter
CylinderReorient	6.7 / 37.8	0.0 / 24.4	0.0 / 31.1	—	—	80.0 / 86.7
CuboidReorient	26.7 / 51.1	0.0 / 33.3	0.0 / 28.9	—	—	80.0 / 86.7
BunnyReorient	0.0 / 35.6	0.0 / 31.1	0.0 / 26.7	—	—	66.7 / 77.8
HammerUse	0.0 / 26.7	0.0 / 30.5	0.0 / 26.7	0.0 / 27.6	20.0 / 36.2	66.7 / 86.7
BrushSweep	0.0 / 39.0	0.0 / 29.5	0.0 / 8.6	26.7 / 41.9	0.0 / 5.7	73.3 / 89.5
ScrewdriverUse	6.7 / 37.3	0.0 / 25.3	0.0 / 33.3	0.0 / 17.3	0.0 / 20.0	73.3 / 86.7
BulbReplace	0.0 / 35.6	0.0 / 25.6	0.0 / 20.0	—	—	86.7 / 95.6
Average	5.7 / 37.6	0.0 / 28.5	0.0 / 25.0	8.9 / 28.9	6.7 / 20.6	75.2 / 87.1

Table 1: Dexterous teleoperation results on SharpaWave (paper Table 1); each cell is SR / TP in percent. SimToolReal is not a teleoperation method; † is category-specific and ‡ all-categories, averaged over the three tool tasks only.

The SR-to-TP gap reveals where failures occur: TeleDexter's 75.2 versus 87.1 shows most failures happen at late task stages, whereas baselines collapse at the first stage requiring in-hand reorientation or finger gaiting. The stage-wise data (Figure 6 in the paper) support this reading: on BulbReplace, TeleDexter passes both screw-in and unscrew rotation stages in 15/15 trials and loses only at final placement (13/15); on ScrewdriverUse, 13 of 15 trials sustain continuous finger gaiting through the tightening stage, a contact mode no baseline can execute. On LeapHand the same training pipeline transfers by swapping only the geometry-aware retargeting stage: 60.0/73.3 on CylinderReorient and 73.3/82.2 on CuboidReorient, with no re-collection of human reference motions.

Figure 3: Task descriptions and stage decomposition (paper Figure 3). Top row: the three reorientation tasks; bottom row: the four tool tasks. Each thumbnail sequence corresponds to one well-defined stage, and SR/TP score against exactly these stages.

From teleoperation to autonomy: demonstrations collected with TeleDexter train Conv-UNet Diffusion Policies conditioned on third-person and wrist RGB. Each task uses 50 expert demonstrations and 15 real-world evaluation trials. HammerDriver reaches 73.3% SR with perfect grasping and reorientation (15/15 through stage 2), failing only at the nail-driving stage where the policy must sustain repeated contact force against the foam target. BulbInstall reaches 46.7%: grasping and reorientation succeed reliably, precision alignment is the bottleneck (8/12), and once aligned the install succeeds in 7/8 trials. BrushForward reaches 40.0%: the thin, irregular brush handle makes grasping the dominant failure point (7/15), yet trials that survive the grasp nearly all complete rotation and sweep (6/7). The decisive point is that no baseline teleoperation system in Table 1 can reliably complete any of these three tasks, so comparable demonstration data cannot be collected with existing methods at all: TeleDexter is simultaneously a teleoperation interface and a data-collection pipeline for autonomous dexterous manipulation.

Autonomous task	Stage chain (surviving trials)	SR
BulbInstall	13/15 pick up, 12/13 reorient, 8/12 align, 7/8 install	46.7%
HammerDriver	15/15 pick up, 15/15 rotate, 11/15 drive nails	73.3%
BrushForward	7/15 pick up, 7/7 rotate, 6/7 sweep forward	40.0%

Table 2: Stage-wise success of the autonomous policies (paper Table 3 and Section 4.3). 50 demonstrations and 15 real trials per task; the chain lists trials surviving each stage.

Ablation one compares consecutive subgoal tracking against dense frame-wise tracking in simulation. Even under dense evaluation, which favors the dense baseline, sparse subgoal tracking achieves far longer episodes (Cuboid 378.6 versus 115.8); under sparse evaluation the gap widens to an order of magnitude (Cuboid 32.6 versus 2.6 consecutive subgoals reached; Hammer 186.6 versus 2.7). Dense frame-wise tracking forces step-by-step replication of the reference, leaving no tolerance to discover physically feasible contact strategies, and terminates early. Ablation two removes random action masking on the real robot: HammerUse drops from 66.7/86.7 to 33.3/57.1, ScrewdriverUse from 73.3/86.7 to 0.0/36.0, and CuboidReorient from 80.0/86.7 to 26.7/51.1. Masking acts as an action-space regularizer that stops the policy from exploiting simulation-specific dynamics, and it is the precondition for zero-shot deployment.

Ablation	Setting	With design	Without design
Subgoals vs dense (sim)	Cuboid: dense-eval EpLen / sparse-eval goals	378.6 / 32.6	115.8 / 2.6
Subgoals vs dense (sim)	Hammer / Screwdriver sparse-eval goals	186.6 / 178.5	2.7 / 2.7
Action masking (real SR/TP)	Hammer / Screwdriver / Cuboid	66.7/86.7, 73.3/86.7, 80.0/86.7	33.3/57.1, 0.0/36.0, 26.7/51.1

Table 3: The two core ablations (paper Tables 4 and 5). EpLen is episode length and Goals is consecutive subgoals reached; both are higher-is-better.

flowchart TB
  subgraph DATA["Reference motion construction"]
    A[Mocap human hand-object trajectories
150 trajs per object, 20 s at 30 Hz] --> B[Stage 1 vector-alignment retargeting
Adam 1e-3, 6000 steps]
    B --> C[Stage 2 geometry-aware refinement
surface 10, penetration 2, collision 10, smooth 0.1]
    C --> D[Wrist-frame fingertip targets
plus object 6D pose]
  end
  subgraph RL["Single-stage RL in Isaac Gym"]
    D --> E[Consecutive subgoal co-tracking
dwell 5-15 frames, tip 3 cm, pos 1 cm, rot 10 deg]
    E --> F[Hybrid reward
sparse reach plus 0.1 dense minus 0.1 time]
    F --> G[Random action masking
p 0.15, 3 DoFs frozen stale]
    G --> H[Curriculum
gravity, tolerances, step 40 to 80 frames]
    H --> I[SAPG, 62400 envs, 4x RTX 5090, 1e10 steps]
  end
  subgraph DEPLOY["Zero-shot real world"]
    I --> J[Kinematic retarget pre-grasp]
    J --> K[Operator switches to co-tracking controller]
    K --> L[7 tasks, 75.2 percent SR, 87.1 percent TP]
    L --> M[50 demos per task train Diffusion Policy
46.7 to 73.3 percent autonomous SR]
  end

Pipeline reconstruction from Sections 3.1-3.4 and 4.3: reference motions, single-stage co-tracking RL, and zero-shot deployment feeding autonomous policy learning.

Figure 4: Real-world failure cases (paper Figure 7). (a) Interaction perturbation: grasp destabilizes after an external contact disturbance. (b) Contact transition jam: the object wedges in the grasping geometry during a gaiting switch. (c) Tracking stall: without tactile feedback the policy stops correcting an already slipped object. All three failure families sit at contact transition or contact maintenance, not at the subgoal semantics.

Figure 5: Hand-object motion capture setup (paper Figure 8). Tripod-mounted NOKOV infrared cameras enclose a dedicated capture volume; the offline reference-motion glove carries dense markers on wrist, palm, and all finger joints, while the online teleoperation glove trims markers to cut latency.

Limitations

The authors state two boundaries. First, TeleDexter currently learns an object-specific controller: a new object requires collecting fresh human hand-object interaction data and training a dedicated policy, and scaling to a unified object-conditioned controller that generalizes across categories without per-object collection and training is named as promising future work. Second, real-world deployment relies on a motion-capture system for real-time hand and object pose estimation; replacing it with markerless vision-based tracking would significantly lower the deployment barrier and broaden practical applicability.

Three independent judgments supplement the authors' list. First, the SR/TP protocol depends on manually defined stage decompositions, and stage granularity directly moves both metrics, so cross-paper comparisons need care; with 15 trials per condition, the difference between 66.7% and 73.3% sits inside overlapping binomial confidence intervals. Second, the kinematic-pre-grasp to co-tracking-controller switch is left to operator judgment: the paper neither quantifies how switch timing affects success nor reports the share of failures attributable to bad switches, even though this is the most error-prone point of the human-machine interface. Third, the autonomy experiments cover only the contact-intensive sub-stages of three tasks with 50 demonstrations each, and the BrushForward grasping bottleneck (7/15) shows RGB observations resolve finger placement on thin handles poorly; no depth or tactile channel is attempted and no demonstration-scaling curve is reported.

Conclusion and Outlook

The contribution of TeleDexter is a redistribution of control authority in dexterous teleoperation: the operator declares only synchronized geometric targets for fingertips and object, while the full complexity of contact strategy is left to single-stage RL searching in simulation. The consecutive subgoal formulation leaves the policy freedom over contact transitions, geometry-aware retargeting keeps subgoals physically feasible, and random action masking turns the actuation gap between simulation and hardware into part of the training distribution. Seven tasks at 75.2% average success against near-total baseline failure, plus autonomous policies behavior-cloned from 50 demonstrations, show the pipeline answers both long-standing questions at once: whether teleoperation can be dexterous enough, and where dexterous data is supposed to come from.

The open problems are equally concrete. Object-specific controllers mean every new object restarts the collection-and-training loop, making object conditioning or object-geometry encoding the natural next step. Mocap dependence locks deployment inside a laboratory volume, so markerless visual tracking is the precondition for practical use. And sensitivity analyses of the SR/TP protocol, switch timing, and demonstration scale will decide whether this evaluation can become a community-reusable benchmark. If those three are resolved in turn, human-level in-hand dexterity would for the first time have a scalable supply chain of both data and controllers.

Golden Quote

The operator only needs to say where the hand and the object should be; how the contact happens is what the controller teaches itself in simulation. The bottleneck of dexterous teleoperation was never intent, but the execution gap between intent and multi-contact physics.