Towards Human-level Dexterous Teleoperation
Humans reorient, translate, and regrasp objects within a single hand by orchestrating continuous contact transitions, yet existing dexterous teleoperation either retargets kinematics while ignoring contact forces and object inertia, or relies on generative action priors that drift in closed loop. TeleDexter formulates dexterous teleoperation as hand-object co-tracking: the operator specifies synchronized subgoal sequences of fingertip positions and object pose, and a single-stage reinforcement-learning controller learns in simulated contact how to realize them. A hybrid reward couples sparse subgoal reaching with dense tracking, geometry-aware two-stage retargeting converts mocap human hand-object data into physically feasible reference motions, and random action masking plus domain randomization enable zero-shot sim-to-real transfer. On seven reorientation and long-horizon tool-use tasks with a Franka FR3 and SharpaWave or LeapHand hands, TeleDexter averages 75.2% success and 87.1% task progress where baselines nearly uniformly fail, and 50 demonstrations per task behavior-clone into autonomous policies reaching 46.7-73.3% success.
