Deep Reinforcement Learning for Adaptive Gain Tuning in Control of Teleoperation Manipulators with Joint Flexibility and Time-Varying Delays

Background and Motivation

Bilateral teleoperation systems consist of a human operator, a local master robot, a communication channel, and a remote slave robot, widely applied in space exploration, rehabilitation, and minimally invasive surgery. Commercial platforms like the da Vinci surgical system and the Sina teleoperation system demonstrate the practical value of this technology. However, real teleoperation systems face two core challenges: joint flexibility and time-varying communication delays. Joint flexibility arises from elastic deformation in transmission components such as harmonic reducers and flexible couplings, being particularly significant in precision scenarios like surgery and space operations—the motor side and link side are elastically coupled, causing vibrations and imprecise position tracking. Time-varying communication delays stem from internet or satellite link uncertainty, where delay fluctuations disrupt master-slave synchronization and can cause system instability.

From a historical perspective, teleoperation control research has evolved from scattering transforms to adaptive control and wave-variable methods. Anderson and Spong's scattering transform in the late 1980s established passivity theory foundations, but position drift has persistently plagued this direction. Chopra et al.'s adaptive control and Nuño et al.'s position synchronization strategies partially alleviated drift but still face ambiguous stability boundaries under time-varying delays. Recently, reinforcement learning has matured in control applications—Xu et al. applied TD3 to circulating cooling water system adaptive control, demonstrating TD3 outperforms PID, fuzzy PID, and DDPG in nonlinear environments; Yin et al. used TD3 for permanent magnet synchronous motor vector control, showcasing DRL's potential in slow-dynamics uncertain systems. These works inspired the approach of introducing TD3 into teleoperation gain tuning, but this paper's innovation lies in embedding DRL within a P+d framework with rigorous stability proofs, rather than completely replacing the traditional controller, thus combining theoretical guarantees with data-driven optimization.

Existing methods have limitations: scattering-transform-based methods suffer from position drift; adaptive control reduces drift but struggles with time-varying delays; wave-variable methods degrade under large delays. More critically, most methods assume rigid joint models, ignoring transmission elasticity, while flexible-joint models more faithfully represent actual teleoperation dynamics. P+d (Proportional plus damping injection) controllers have been proven to guarantee stable motion and asymptotic position tracking under bounded delays, but their fixed gains cannot simultaneously accommodate stability and tracking precision under varying delay conditions—excessive gains cause vibration, while insufficient gains lead to sluggish tracking. This paper proposes combining a P+d controller with a TD3-based deep reinforcement learning agent that adaptively tunes remote-side proportional gain K and damping gain B in real time, optimizing tracking performance while guaranteeing stability.

Dynamic Model

The system comprises a rigid 2-DoF master (local) robot and a flexible-joint 2-DoF slave (remote) robot. The remote robot dynamics are split into link-side and motor-side, coupled through stiffness matrix $S_r$:

$$M_r(q_r)\ddot{q}_r + C_r(q_r, \dot{q}_r)\dot{q}_r + S_r[q_r - \theta_r] = -\tau_e$$

$$J_r\ddot{\theta}_r + S_r[\theta_r - q_r] = \tau_r$$

where $q_r, \dot{q}_r, \ddot{q}_r \in \mathbb{R}^n$ are link position, velocity, and acceleration, $\theta_r \in \mathbb{R}^n$ is motor position, $M_r \in \mathbb{R}^{n \times n}$ is the link inertia matrix, $C_r$ is the Coriolis matrix, $J_r$ is the motor inertia matrix, $S_r$ is the diagonal stiffness matrix, $\tau_r$ is the motor torque, and $\tau_e$ is the environmental contact torque. The local robot dynamics are $M_l(q_l)\ddot{q}_l + C_l(q_l, \dot{q}_l)\dot{q}_l = \tau_h - \tau_l$, where $\tau_h$ is the operator-applied torque and $\tau_l$ is the local control torque. The inertia matrices satisfy the skew-symmetry condition $\dot{M}_i(q_i) = C_i(q_i, \dot{q}_i) + C_i^T(q_i, \dot{q}_i)$, which is fundamental to Lyapunov stability analysis. The stiffness matrix $S_r = 100I$ N·m represents the joint elastic coupling strength—higher stiffness means weaker flexibility effects, but vibrations can still occur under high-frequency excitation.

The physical meaning of the flexible-joint model deserves deeper understanding. In real robots, motors drive links through reducers (e.g., harmonic drives) that are not ideally rigid—their elastic deformation causes motor angle $ heta_r$ to differ from link angle $q_r$, with the difference $ heta_r - q_r$ generating an elastic restoring torque $S_r[ heta_r - q_r]$ that both drives link motion and reacts on the motor. When the system is subjected to external disturbances or high-frequency excitation, this elastic coupling produces sustained vibrations that decay slowly under low damping, severely degrading end-effector precision. The stiffness $S_r = 100I$ N·m is a moderate value, exhibiting clear flexibility effects (unlike rigid joints with $S_r o \infty$) without being so soft as to make control intractable. In teleoperation, flexibility effects compounded with communication delays make the problem more complex—delay causes master-slave synchronization information to lag, while flexibility causes the slave response to oscillate, and the combination can destabilize the system.

Control Strategy

P+d Controller

The local and remote control torques are defined as:

$$\tau_l = K_l\left[q_l(t) - \theta_r(t - T_r(t))\right] + B_l\dot{q}_l(t)$$

$$\tau_r = -K_r\left[\theta_r(t) - q_l(t - T_l(t))\right] - B_r\dot{\theta}_r(t)$$

where $K_l, K_r$ are proportional gains, $B_l, B_r$ are damping gains, and $T_l(t), T_r(t)$ are time-varying forward and backward communication delays. The error signals are $e_l = q_l - \theta_r(t - T_r(t))$ and $e_r = \theta_r - q_l(t - T_l(t))$. The P+d controller dissipates system energy through damping injection, ensuring stability under bounded delays. The limitation of fixed gains is: when delay increases, larger damping is needed for stability, but excessive damping reduces tracking response speed; when delay decreases, damping can be reduced to improve tracking precision, but fixed gains cannot automatically adapt to such changes.

DRL Adaptive Gain Tuning

The TD3 agent's state space includes remote motor position $\theta_r(t)$, velocity $\dot{\theta}_r(t)$, delayed local link position $q_l(t - T_l(t))$, tracking error $e_{\text{tracking}} = \theta_r - q_l(t - T_l(t))$ and its integral, damping error $e_{\text{damping}} = \theta_r - q_r$ and its integral:

$$\mathbf{s}_t = \left[\theta_r(t), \dot{\theta}_r(t), q_l(t - T_l(t)), e_{\text{tracking}}(t), \int e_{\text{tracking}} d\tau, e_{\text{damping}}(t), \int e_{\text{damping}} d\tau\right]$$

The action space is the adaptive gains for each remote joint:

$$\mathbf{a}_t = [K_r^i, B_r^i], \quad i \in \{1, 2, \ldots, n\}$$

The reward function comprehensively considers tracking error, damping error, and action magnitude:

$$r_t = -\left(\alpha \|e_{\text{tracking}}\|^2 + \beta \|e_{\text{damping}}\|^2 + \gamma \|\mathbf{a}_t\|^2\right)$$

where $\alpha, \beta, \gamma$ are weight coefficients, with the third term penalizing excessive gain outputs to avoid control signal saturation. Gains are constrained within a safe set $\mathcal{S}$: $K_i^{\min} \leq K_i(t) \leq K_i^{\max}$, $B_i^{\min} \leq B_i(t) \leq B_i^{\max}$, satisfying the stability condition $4B_l(t)B_r(t) \geq (T_l^\ast + T_r^\ast)^2 K_l(t)K_r(t) + \varepsilon$ ($\varepsilon > 0$), ensuring negative-definite Lyapunov derivative. TD3 employs twin critic networks to reduce Q-value overestimation, with the actor network's final layer using Softplus activation to ensure positive, smooth gains.

flowchart LR
    H["Operator"] --> ML["Local Master
Rigid 2-DoF"]
    ML -->|"\tau_l = K_l·e_l + B_l·\dot{q}_l"| CH["Comm Channel
Time-varying delay T_l, T_r"]
    CH -->|"q_l(t-T_l)"| DRL["TD3 Agent
state s_t → action K_r, B_r"]
    DRL -->|"\tau_r = -K_r·e_r - B_r·\dot{\theta}_r"| RR["Remote Slave
Flexible-joint 2-DoF"]
    RR -->|\theta_r, q_r| DRL
    RR --> CH
    CH --> ML

Lyapunov-Krasovskii Stability Analysis

A Lyapunov-Krasovskii functional $V = V_l + \eta V_r + E_h + \eta E_e + \frac{\eta K_l}{2}|q_l - \theta_r|$ is constructed, where $V_l = \frac{1}{2}\dot{q}_l^T M_l(q_l) \dot{q}_l$ is local kinetic energy, $V_r$ is remote kinetic energy plus elastic potential energy, and $E_h$ and $E_e$ are operator and environment energy terms. Using the delay integral inequality $\int_0^t x^T(\sigma) \int_{-T(\sigma)}^0 y(\sigma+\theta) d\theta d\sigma \leq \frac{\alpha}{2}\|x\|^2 + \frac{1}{2\alpha}\|y\|^2$, it can be proven that within the gain constraint set $\mathcal{S}$, the Lyapunov derivative satisfies:

$$\dot{V} \leq -\lambda_l \|\dot{q}_l\|^2 - \lambda_r \|\dot{\theta}_r\|^2, \quad \lambda_l, \lambda_r > 0$$

This guarantees asymptotic stability—i.e., $\dot{q}_l \to 0$ and $\dot{\theta}_r \to 0$, with master-slave velocities synchronously converging to zero. The DRL agent's adaptive gains always remain within the constraint set $\mathcal{S}$, thus not disrupting the stability guaranteed by the P+d controller, but rather searching for optimal gain configurations within the stable region.

The construction of the Lyapunov-Krasovskii functional is central to the theoretical contribution. Unlike Lyapunov functions for delay-free systems, time-delay systems require Krasovskii functionals to handle delay terms—the functional includes not only current state energy but also integral terms of past state history, characterizing the influence of delay on system energy. In this paper's functional, $E_h = -\int_0^t \dot{q}_l^T(\sigma) au_h(\sigma) d\sigma + \kappa_l$ and $E_e = \int_0^t \dot{q}_r^T(\sigma) au_e(\sigma) d\sigma + \kappa_r$ represent operator-injected and environment-dissipated energy, with $\eta > 0$ as the remote weight. The elastic potential energy $rac{1}{2}(\dot{q}_r^T M_r \dot{q}_r + heta_r^T S_r heta_r)$ is included in $V_r$. The key is proving $\dot{V} \leq -\lambda_l \|\dot{q}_l\|^2 - \lambda_r \|\dot{ heta}_r\|^2$, which requires the gain constraint $4B_l B_r \geq (T_l^st + T_r^st)^2 K_l K_r + arepsilon$—physically meaning the damping product must be large enough to overcome energy injection from delay times stiffness. The DRL agent must satisfy this constraint when searching for optimal gains, implemented through projection or clipping on action outputs.

Experimental Results

Simulation Setup

Simulations are implemented in MATLAB/Simulink with total duration $T_f = 40$s and sampling time $T_s = 0.01$s. Both master and slave are 2-DoF, with link lengths $l_1 = l_2 = 0.38$m, link masses $m_{1l} = m_{1r} = 0.5$kg, $m_{2l} = m_{2r} = 0.35$kg, and remote motor inertia $J_r = 0.3I$ kg·m². Local gains are fixed at $K_l = 25$, $B_l = 10$. Communication delay bounds are $T_l^\ast = T_r^\ast = 1$s. The operator torque model is $\tau_h = K_s(q_h - q_l) - K_d \dot{q}_l$ ($K_s = 10, K_d = 2$), with environment torque $\tau_e = 0$ (free motion). TD3 hyperparameters: discount factor $\gamma = 0.99$, batch size 512, learning rate $1 \times 10^{-4}$, target network soft update coefficient $\tau = 0.005$, exploration noise standard deviation decaying from 4 to 0.05 (decay rate $1 \times 10^{-7}$).

Performance Evaluation

Figure 1: Schematic of the 2-DoF flexible joint manipulator

Figure 2: Control framework overview—bilateral teleoperation with DRL adaptive gain tuning

Figure 3: Time response of remote joint positions and delayed local link positions

Experimental results show the DRL controller effectively suppresses oscillatory behavior in flexible-joint systems under delay conditions. Remote joint positions $\theta_r(t)$ and delayed local link positions $q_l(t-T_l)$ rapidly converge to synchronized steady state. Position tracking error $\theta_r - q_l(t-T_l)$ converges swiftly to near zero for both joints with negligible steady-state error. The rapid attenuation of remote link-joint damping error $\theta_r - q_r$ confirms the adaptive gain tuning mechanism successfully emulates virtual damping, mitigating the destabilizing influence of joint elasticity. DRL-generated adaptive gains $K_r^i$ and $B_r^i$ adjust in real time with delay variations—damping gains automatically increase when delay grows to maintain stability, while proportional gains moderately increase when delay decreases to improve tracking response.

TD3 Hyperparameters and Simulation Parameters

Several key phenomena can be observed from simulation results: first, the DRL-generated adaptive gains are not constant but vary dynamically with system state—when tracking error is large, proportional gain $K_r$ increases to accelerate convergence; when vibration trends appear, damping gain $B_r$ increases to suppress oscillation. Second, gain changes are smooth, thanks to the Softplus activation function and TD3's target policy smoothing, avoiding control signal jumps from sudden gain changes. Third, twin critic networks effectively mitigate Q-value overestimation, making the gain tuning policy more robust. Compared to fixed-gain P+d controllers, the DRL adaptive scheme significantly reduces tracking error under 1s large delays with faster flexibility vibration attenuation, validating the effectiveness of data-driven optimization within the stable region.

From a control engineering practice perspective, the design philosophy of this framework has important methodological significance. Traditional control theory provides stability guarantees but with fixed gains that cannot adapt to varying conditions; pure DRL may learn unstable policies but offers strong adaptability. This hybrid approach combines both advantages—the P+d controller defines a safety region, and DRL optimizes performance within it. This "safety barrier + intelligent optimization" paradigm can be generalized to other control systems requiring both theoretical guarantees and performance, such as robot contact force control and UAV formation control. Notably, the DRL agent only tunes remote-side gains while keeping local-side gains fixed—a conservative but safe design. If both sides adapt simultaneously, stability constraints become more complex and coupled. The authors note in conclusion that future multi-agent frameworks may achieve more comprehensive bilateral adaptation, but require more refined stability analysis to ensure cooperative optimization does not violate safety constraints.

Limitations and Future Work

Limitation 1: Simulation-only validation. All experiments are conducted in MATLAB/Simulink simulation without physical teleoperation platform validation. Real-world factors like sensor noise, friction nonlinearity, and model mismatch may affect the DRL agent's generalization, making sim-to-real transfer a challenge.

Limitation 2: Remote-side only adaptation. The DRL agent only tunes remote-side gains $K_r, B_r$, while local-side gains $K_l, B_l$ remain fixed. Future work plans to extend adaptive gain tuning to the master side, incorporate impedance adaptation for interactions with stiff environments, and explore a multi-agent framework to coordinate independent reinforcement learning on both master and slave sides.

From a broader perspective, this work sits at the intersection of control theory and machine learning. Traditional control theory emphasizes provable stability and performance guarantees but relies on precise models and conservative design; deep reinforcement learning excels at discovering complex nonlinear mappings and adaptive strategies from data but lacks rigorous safety guarantees. The "learning-enhanced control" paradigm that fuses both is becoming an important trend in robot control—controllers provide safety baselines, and learning modules optimize performance within safety constraints. This paper's P+d+TD3 framework is a typical instance of this paradigm: Lyapunov-Krasovskii analysis provides the $\dot{V} \leq 0$ stability guarantee, while TD3 searches within the gain safety set $\mathcal{S}$ for gain configurations that minimize tracking error and vibration. This design also offers good interpretability—the DRL-learned gain tuning strategy can be compared with traditional control intuition; for example, the behavior of increasing damping with larger delay is consistent with manual tuning experience, but DRL adapts more finely to instantaneous delay variations rather than relying on fixed empirical values. For practical deployment, the modular design also facilitates engineering implementation—the P+d controller serves as a safety fallback, so even if the DRL module fails, the system still operates stably under conservative gains, which is critical for safety-critical applications like surgical robotics.

Summary and Insights

This paper proposes a hybrid control framework combining a P+d controller with TD3 deep reinforcement learning to address the dual challenges of joint flexibility and time-varying delays in bilateral teleoperation. The P+d controller guarantees asymptotic stability under bounded delays through Lyapunov-Krasovskii analysis, while the DRL agent real-time tunes remote-side proportional and damping gains within the stable region, optimizing tracking precision and vibration suppression. Key technical contributions include: incorporating the flexible-joint model (link-side $q_r$ and motor-side $\theta_r$ coupled through stiffness $S_r$) into teleoperation control design; constructing a Lyapunov-Krasovskii functional $V = V_l + \eta V_r + E_h + \eta E_e + \frac{\eta K_l}{2}|q_l - \theta_r|$ with energy and elastic potential terms and proving $\dot{V} \leq -\lambda_l\|\dot{q}_l\|^2 - \lambda_r\|\dot{\theta}_r\|^2$; TD3 state space including tracking error and its integral, damping error and its integral, with reward $r_t = -(\alpha\|e_{\text{tracking}}\|^2 + \beta\|e_{\text{damping}}\|^2 + \gamma\|\mathbf{a}_t\|^2)$ balancing precision, vibration suppression, and control energy; gains constrained within stability set $\mathcal{S}$ ensuring DRL does not compromise theoretical stability guarantees. In 2-DoF flexible-joint teleoperation simulation, fast stabilization, accurate tracking, and effective vibration suppression are achieved under 1s time-varying delays. This work provides a practical solution for precision remote manipulation scenarios like surgical robotics and space teleoperation, combining theoretical stability with data-driven optimization.

Furthermore, TD3's improvements over DDPG are critical for this application: twin critic networks taking the minimum value avoid gain divergence from Q-value overestimation, delayed policy updates reduce training instability, and target policy smoothing suppresses gain jumps. These features together ensure that under the complex dynamics of time-varying delays and flexible vibrations, the gain tuning policy converges stably rather than oscillating divergently, which is key to reliable operation of practical teleoperation systems.