Let the Body Follow: Coupled Egocentric Control for Whole-Body Robot Teleoperation

Background and Motivation

Whole-body teleoperation requires users to coordinate perception, manipulation, posture, and mobility across multiple robot components—head, arms, torso, and base. This coordination is challenging because users must simultaneously control the robot's head (observation), arms (manipulation), torso (posture adjustment), and base (mobility), while maintaining task awareness and avoiding kinematic or environmental constraints. Existing interfaces typically require explicit touchpad control of torso and base, leading to high cognitive load and low efficiency—users must frequently switch attention between gaze/hand control and torso/base adjustment, and this context switching disrupts the manipulation flow.

This paper proposes coupled egocentric control, a "body-following" teleoperation approach where the robot's torso and base automatically respond to the operator's head and arm motions. Head pitch adjusts torso height, head yaw drives base rotation, end-effector height adjusts torso motion, and end-effector workspace boundaries trigger base translation. Users focus only on gaze and hand control while the system automatically coordinates the rest of the body to maintain viewpoint, reachability, and maneuverability. In a user study (12 participants) on a TIAGo mobile manipulator with home-care-inspired tasks, this approach significantly reduced mental demand (p<.01), improved ease of use (p<.05) and confidence (p<.05), with 5x fewer button presses and 3x fewer arm joint limits/singularities compared to the baseline.

Coupled Egocentric Control Method

Design Philosophy

The core idea is simple yet powerful: "let the body follow the operator's gaze and hands." Rather than requiring explicit control of every robot component, coupling rules let the torso and base automatically respond to head and arm motions. This simplifies whole-body teleoperation from "explicit multimodal control" to "egocentric control + automatic body coordination"—users control only the two most natural dimensions (where to look, where to place hands), and the system handles the rest. Head motions support perception-centered control (adjusting torso height and base rotation), arm motions support manipulation-centered control (adjusting torso height and base translation).

Control Mapping Formulas

Head pitch $\theta_t$ and yaw $\theta_p$ drive torso height rate and base rotation angular velocity. A dead-zone sign function avoids micro-jitter:

$$\Gamma(s; \tau, u) = \begin{cases} u, & s > \tau \\ -u, & s < -\tau \\ 0, & \text{otherwise} \end{cases}$$

where $s$ is the input signal, $\tau$ is the dead-zone threshold, $u$ is the output speed. Head-driven control mappings:

$$$\dot{T}_z^h = \Gamma(\theta_t; \lambda_\theta \theta_t^{\max}, v_T), \quad \dot{\psi}_b = \Gamma(\theta_p; \lambda_\theta \theta_p^{\max}, \alpha_M \omega_b)$$$

where $\lambda_\theta = 0.5$ is the threshold scaling factor (triggering at 50% of maximum head angle), $v_T$ is torso speed, $\omega_b$ is base rotation speed, $\alpha_M$ is a scaling coefficient. Arm-driven control mappings are based on end-effector position $\mathbf{p}_{ee}^i = [x_{ee}^i, y_{ee}^i, z_{ee}^i]^\top$ ($i \in \{L, R\}$):

$$\dot{T}_z^{a,i} = \begin{cases} +v_T, & z_{ee}^i > z_{ee}^{\max} \\ -v_T, & z_{ee}^i < z_{ee}^{\min} \\ 0, & \text{otherwise} \end{cases}$$

When the end-effector approaches vertical workspace boundaries, the torso automatically adjusts to extend reach. Similarly, base translation is triggered by horizontal workspace boundaries:

$$\mathbf{v}_b^i = \begin{cases} [+v_x, 0]^\top, & x_{ee}^i > x_{ee}^{\max} \\ [-v_x, 0]^\top, & x_{ee}^i < x_{ee}^{\min} \\ [0, +v_y]^\top, & y_{ee}^i > y_{ee}^{\max} \\ [0, -v_y]^\top, & y_{ee}^i < y_{ee}^{\min} \end{cases}$$

Total torso speed is the saturated sum of head and arm driven components: $$\dot{T}_z = \text{sat}_{[-v_T, v_T]}\left(\dot{T}_z^h + \sum_{i \in \mathcal{A}} \dot{T}_z^{a,i}\right)$$, total base velocity is the projected fusion of arm-triggered directions $\mathbf{v}_b = \Pi_\rho(\{\mathbf{v}_b^i | i \in \mathcal{A}, \mathbf{v}_b^i \neq \mathbf{0}\})$, where $\Pi_\rho$ projects multiple directions to the nearest non-conflicting direction, $\rho = 90°$ is the conflict angle threshold. Arm joint angles are solved via inverse kinematics: $\mathbf{q}_a^i \leftarrow \text{IK}(\mathbf{x}_{ee}^i)$.

From a human factors engineering perspective, coupled egocentric control's design deeply understands the cognitive bottleneck of human multitask control. Human working memory capacity is limited (Miller's "7±2" rule), and simultaneously managing multiple explicit control channels (head pitch/yaw, arm position, torso height, base rotation/translation) rapidly exhausts cognitive resources. Traditional interfaces require users to interrupt arm operation to press buttons adjusting torso or base—this attention switching has not only time costs but also cognitive re-orientation costs—users must switch from "manipulation" to "navigation" mindset and back. Coupled control eliminates these switches by automating torso and base adjustments as side effects of head and arm motion—users maintain the natural "look + grasp" operation flow while body coordination is implicitly handled. This explains why button usage dropped 5x while performance improved—the reduction is exactly the overhead of these cognitive switches. Joint limits/singularities dropping 3x also shows automatic body coordination helps maintain good arm configurations—when torso and base automatically follow the arm's reach, the arm doesn't need to extend to extreme positions, avoiding singularities and limits. Organizing task (torso-dominant) completion time significantly reduced because that task most depends on torso adjustment, and coupled control transforms torso adjustment from explicit button operation to automatic result of arm motion, yielding the most obvious efficiency gain.

Control Mapping Design Principles

These mappings follow the "natural correspondence" principle—control mappings should align with human intuition, so operators don't need to learn counter-intuitive relationships. Head pitch to torso height leverages the natural "look up→need to rise" association; end-effector workspace boundaries to base/torso leverage the natural "can't reach→body follows" response. The dead-zone design ($\lambda_\theta = 0.5$) ensures only deliberate head deflections trigger body movement, avoiding accidental motion from micro-jitter. Saturated fusion ensures multiple triggers don't cause overspeed. Direction projection fusion $\Pi_\rho$ handles cases where both hands trigger different directions, using 90° as conflict threshold to select the nearest non-conflicting direction.

flowchart TD
    HEAD["VR head pose
pitch θ_t, yaw θ_p"] --> DEAD1["Dead-zone function Γ(s;τ,u)"]
    DEAD1 --> TORSO_H["Torso height ΔT_z^h"]
    DEAD1 --> ROT["Base rotation ψ_b"]
    ARM["VR controllers
EE position p_ee"] --> IK["Inverse kinematics IK"]
    IK --> ARM_ACT["Arm joints q_a"]
    ARM --> BOUND_Z["Vertical boundary check
z_ee ∈ [min,max]?"]
    BOUND_Z --> TORSO_A["Torso height ΔT_z^a"]
    ARM --> BOUND_XY["Horizontal boundary check
x,y_ee ∈ [min,max]?"]
    BOUND_XY --> BASE_TRANS["Base translation v_b"]
    TORSO_H --> SAT["Saturated fusion
sat(ΔT_z^h + ΣΔT_z^a)"]
    TORSO_A --> SAT
    SAT --> TORSO_OUT["Torso motor"]
    ROT --> BASE_ROT["Base rotation motor"]
    BASE_TRANS --> BASE_TRANS_OUT["Base translation motor"]

User Study

Participants and Tasks

12 participants (5 male, 7 female, ages 19-39, M=27.92) recruited, all having completed a prior study with the same baseline interface and highly familiar with it (M=4.33/5), providing a conservative comparison. Study lasted 90 minutes, $15 compensation. Phase I had three tasks emphasizing different control demands: Collecting (base-dominant, lateral move to reach bottle), Organizing (torso-dominant, lower to upper shelf), Transferring (rotation-dominant, pick up then 90° rotation to place). Phase II was a composite cleaning task (rigid bottles + deformable cloth, multiple workspaces).

Main Results

Figure 1: Coupled egocentric control—robot body automatically follows operator's gaze and hands

Figure 2: Control mappings—head and arm motions drive torso and base

Figure 3: Home-care-inspired evaluation tasks—collecting, organizing, transferring, and composite cleaning

From a broader perspective, this work represents a paradigm shift in teleoperation interface design from "explicit compartmentalized control" to "implicit coupled control." Traditional interfaces decompose the robot into independently controlled subsystems (head, arms, torso, base), each with dedicated control channels requiring explicit switching and management. This "compartmentalized" design is conceptually clear but cognitively demanding—users must not only execute tasks but also manage control mode switching. Coupled egocentric control breaks this compartmentalization—by automating low-level coordination, users focus only on high-level intent (where to look, what to grasp) while the system handles "how to get there" details. This design philosophy aligns with the "intent interface" concept in autonomous driving—users specify goals rather than every step. Technically, the coupling rules' simplicity (threshold-triggered velocity commands) makes them easy to implement and debug, and makes behavior predictable—users can quickly build a mental model of how the body follows. Predictability is crucial for establishing trust in teleoperation—if body-following behavior is unpredictable, users feel loss of control and load increases. Dead-zone, saturation, and projection fusion design details all ensure coupling behavior's predictability and safety. Phase II composite cleaning task results further validate effectiveness in real complex scenarios—participants not only handled rigid objects but also deformable cloth, showing coupled control remains effective with diverse object interactions. Participants' strong subjective preference for coupled control is an important indicator for real-world deployment acceptance.

Limitations and Future Work

Limitation 1: Relies on handheld VR controllers. Current arm input relies on handheld VR controllers; a more natural interface might be controller-free hand tracking or gesture input. Future work will explore combining hand tracking with coupled egocentric control.

Limitation 2: Platform and coupling rule limitations. Only validated on TIAGo mobile manipulator; future work should extend to humanoid robots and legged mobility platforms. Coupling thresholds and mappings are fixed; future systems could adapt them based on task context, user preference, or robot state. Integration with ergonomic and workload-aware models to reduce fatigue during prolonged teleoperation is also a direction.

Summary and Insights

This paper proposes coupled egocentric control for whole-body robot teleoperation, with the core idea "let the body follow gaze and hands" being simple yet effective. Technical contributions include: dead-zone sign function $\Gamma(s; \tau, u)$ avoiding jitter; head pitch/yaw to torso height/base rotation mappings $\dot{T}_z^h = \Gamma(\theta_t; \lambda_\theta \theta_t^{\max}, v_T)$; end-effector workspace boundary to torso/base trigger mappings; saturated fusion $\dot{T}_z = \text{sat}_{[-v_T,v_T]}(\dot{T}_z^h + \sum \dot{T}_z^{a,i})$ and direction projection fusion $\mathbf{v}_b = \Pi_\rho(\{\mathbf{v}_b^i\})$. A 12-participant user study on home-care tasks showed: mental demand significantly reduced (p<.01), ease of use/learning/confidence significantly improved, button usage reduced 5x, joint limits reduced 3x. Organizing (torso-dominant) task completion time significantly reduced, object manipulation time significantly reduced across all tasks—showing body-following is most effective during interaction-rich portions. Participant feedback: "coupled control helped me seamlessly approach and grasp/place targets" and "it was useful that base/torso moved in sync with the robot hand, as I often forgot which button to press." This work provides a practical solution for reducing cognitive load in whole-body teleoperation, with the "body-following" design principle having broad reference value for service robotics and remote operation. From a design philosophy perspective, simplifying explicit multimodal control to egocentric control + automatic coordination embodies the wisdom of "subtractive design"—not adding more control options, but letting the system handle low-level coordination, freeing users' cognitive resources for high-level task decisions.