Towards Miniature Humanoid Tele-Loco-Manipulation Using Virtual Reality and Reinforcement Learning

1. Overview: Miniature Humanoid Tele-Loco-Manipulation via VR and RL

This paper builds a complete tele-loco-manipulation system for DYNAMIXEL-based miniature humanoid robots, fusing VR teleoperation manipulation with RL walking balance into whole-body control. The motivation is the data bottleneck in imitation learning — full-scale humanoid hardware is expensive and difficult to work with, so miniature platforms like ROBOTIS OP3 offer a far cheaper and more tractable research vehicle. The four contributions: (1) a software architecture for full-body tele-loco-manipulation control of humanoid robots; (2) data-driven DYNAMIXEL XM430 actuator modeling for torque control; (3) a PD-based impedance controller for DYNAMIXEL joint chains; (4) experimental validation on ROBOTIS OP3. The VR side uses a VIVE Pro 2 HMD and hand controllers to track operator head/hand motion, mapped to arm joints via inverse kinematics (IK); the RL policy is trained in Isaac Lab and deployed zero-shot for real-robot walking, enabling independent upper/lower body control.

2. Core Problem: Teleoperation and Walking Fusion on Miniature Humanoids

Classical humanoid loco-manipulation typically uses position-controlled robots that overpower external dynamics, treating physical interactions as disturbances to the walking gait. But this compromises whole-body walking stability, and full-scale hardware is expensive and scarce. VR teleoperation has advanced greatly (driven by COVID-19 and the ANA Avatar XPRIZE, with haptic gloves and upper-body motion retargeting), but relies on rigid position control, simplifying operator motion retargeting at the cost of compliance. For RL walking, massively parallel simulation can rapidly converge to deployable "blind" walking policies on consumer GPUs. The core problem this paper addresses: how to fuse VR upper-body teleoperation with RL lower-body walking balance on a miniature humanoid, independently yet coordinately, while bridging the DYNAMIXEL torque-control sim2real gap.

3. Method: VR Control + RL Locomotion + Torque Control

3.1 VR Teleoperation and Motion Retargeting

The operator wears a VIVE Pro 2 HMD with hand controllers, optically tracked by four SteamVR basestations. Unity serves as the development platform for VR scene creation and physics simulation. Operator hand positions are processed by an inverse kinematics (IK) solver to generate robot arm joint positions. A scaled virtual robot model is superimposed over the operator's torso for embodiment. The robot is a ROBOTIS OP3 (20 DoF: two 6-DoF legs, two 3-DoF arms, one 2-DoF pan/tilt head), with an i7 NUC, 9-DoF IMU, and a replaced VR180 dual-fisheye USB camera for enhanced telepresence. The camera video stream (3840×1080@60Hz) is projected onto virtual spherical displays surrounding the operator.

3.2 PD Impedance Torque Controller

IK solutions are tracked by a 200 Hz torque controller (bandwidth limited by TTL communication speed). DYNAMIXEL XM430 servos achieve open-loop torque control via hardware-level current control, assuming a linear relationship between input current and output shaft torque. The control law is:

$$\boldsymbol{\tau}_{cmd} = K_P (\boldsymbol{q}_{des} - \boldsymbol{q}) - K_D \dot{\boldsymbol{q}} \tag{1}$$

where $\boldsymbol{\tau}_{cmd} \in \mathbb{R}^{20}$ is the command torque vector, $\boldsymbol{q}_{des} \in \mathbb{R}^{20}$ is the desired joint position from Unity, $\boldsymbol{q} \in \mathbb{R}^{20}$ is joint position feedback, $\dot{\boldsymbol{q}}$ is joint velocity from first-order finite difference, $K_P \in \mathbb{R}^{20\times20}$ is the stiffness matrix, and $K_D$ is the damping matrix. The servo torque-current linear model:

$$\tau = k_m \cdot I_{motor}, \quad k_m = \frac{\tau_{stall}}{I_{stall}} \tag{2}$$

The torque controller's output combined with the current-torque mapping constitutes impedance control. An Exponential Moving Average (EMA) filter smooths spherical display rotations to match mechanical delay (~100 ms), reducing VR motion sickness.

3.3 RL Locomotion Policy and Sim2Real

The RL policy is modeled as a Markov Decision Process (MDP) maximizing the reward function $G_t$:

$$G_t = \mathbb{E}\left[\sum_{k=0}^{\infty} \gamma^k \mathcal{R}(s_k, a_k)\right], \quad \pi^* = \arg\max_\pi \mathbb{E}[G_t] \tag{3}$$

The observation space comprises 5 terms (angular velocity, projected gravity in body frame, joint positions, etc.), each stacked with 9 previous timesteps, forming a 330×1 vector. History stacking reduces action noise during sim2real inference. Domain randomization varies mass, center of mass, PD gains, etc., to handle ZMP shifts during teleoperation. The projected gravity $xy$ penalty ensures rapid foot adjustment during CoM changes:

$$r_{pg} = -\exp\left(-\frac{g_{xy}^2}{\sigma_{pg}^2}\right), \quad r_{total} = \sum_i w_i r_i \tag{4}$$

Accurate actuator dynamics modeling is crucial for zero-shot transfer, using Isaac Lab's actuator model with manual PD tuning via sinusoidal and step input responses. The walking ZMP constraint:

$$p_{ZMP} = p_{CoM} - \frac{h}{g} \cdot \ddot{p}_{CoM} \tag{5}$$

where $h$ is CoM height and $g$ is gravitational acceleration. Domain randomization handles $p_{ZMP}$ shifts during teleoperation.

4. Experiments

4.1 Telemanipulation Validation

The operator traces a circle with both arms 10 times rapidly, recording VR controller and robot end-effector positions at 50 Hz. Motion latency is computed via cross-correlation peak between human and robot hand positions, yielding 220 ms in the y-direction. Position tracking shows over 15% RMSE instantaneous error, but most is attributable to motion latency (~220 ms), and the upper body still reproduces the operator's circular arm motions.

Experiment	Latency	Position Error	Assessment
Circle telemanipulation	~220 ms	>15% RMSE	Reproduces circular motion
Locomotion disturbance	—	Projected gravity ≈ -1	Stable
Block relocation	—	2/6 success	Partial success

4.2 Locomotion and Tele-Loco-Manipulation

The locomotion policy receives random whole-body velocity commands every 3 s for 80 s total. The normalized projected gravity stays near -1, indicating the RL policy maintains stable torso orientation under abrupt velocity changes. The final tele-loco-manipulation experiment requires the robot to relocate 6 blocks within 10 minutes: walking ~5 m per trial (average 0.35 m/s), picking up a 40 g PLA cube with both arms, and dropping it in a container. The result of 2/6 successful relocations reveals the full-stack system's capability boundary.

Metric	Telemanipulation	Locomotion	Block Relocation
Latency	~220 ms	—	—
Accuracy	>15% RMSE	Projected gravity ≈ -1	—
Success rate	Circle reproduced	80 s stable	2/6 (33%)
Speed	50 Hz sampling	0.35 m/s avg	5 m/trial

5. Limitations

• Sim2real gap: The current actuator model is rudimentary, requiring excessive domain randomization; DYNAMIXEL TTL communication bandwidth limits torque control to 200 Hz, with instabilities at high joint speeds.
• Overly soft torque control: The teleoperation torque controller was too soft during trials, requiring the operator to cross their arms to generate enough force to hold objects, indicating impedance parameters need optimization.
• Basic locomotion policy: Proprioceptive walking is validated only on simple terrain, lacking more difficult terrain and additional reward shaping; the 33% block relocation success rate shows the full-stack integration needs improvement.

6. Conclusion

This paper builds a tele-loco-manipulation system for the miniature humanoid ROBOTIS OP3, fusing VR upper-body teleoperation with RL lower-body walking. The PD impedance torque controller (200 Hz) achieves DYNAMIXEL XM430 torque control, and the IK solver maps operator hand motion to robot arms. The RL policy is trained in Isaac Lab, using domain randomization and observation history stacking for zero-shot sim2real transfer. Experiments validate motion latency (~220 ms), locomotion stability (projected gravity ≈ -1), and full-stack capability (2/6 block relocations). The core insight is that miniature humanoid platforms can achieve whole-body tele-loco-manipulation at very low cost through a decoupled "VR upper body + RL lower body" architecture — the key is not robot size but the hierarchical decoupling of the control architecture and accurate actuator torque modeling.

flowchart TD
    A["Operator: VIVE Pro 2 + controllers"] --> B["Unity VR scene"]
    B --> C["IK solver: hand position → joint angles"]
    C --> D["PD impedance torque controller 200Hz"]
    D --> E["DYNAMIXEL XM430 torque control"]
    F["Isaac Lab RL training"] --> G["Domain randomization + history stacking"]
    G --> H["Zero-shot sim2real walking policy"]
    H --> I["OP3 lower body walking balance"]
    E --> J["OP3 upper body teleoperation"]
    I --> K["Whole-body tele-loco-manipulation"]
    J --> K
    K --> L["Block relocation: 2/6 success"]