Actuator Reality Shaping for Zero-Shot Sim-to-Real Robot Learning

Actuator Reality Shaping: Zero-Shot Sim-to-Real Robot Learning

Paper: Actuator Reality Shaping for Zero-Shot Sim-to-Real Robot Learning · Authors: Satoshi Yamamori, Koji Ishihara, Kenjiro Minamikawa, Ryosei Ohmori, Taiyo Yasaki, Norikazu Sugimoto, Jun Morimoto (Kyoto University / ATR) · arXiv: 2607.02205 · Project page (no public code yet) · Keywords: reinforcement learning, sim-to-real, actuator reality shaping, two-degree-of-freedom control, disturbance observer

One-Sentence Summary

Instead of making the simulator match the real motors, this paper makes the real motors behave like the simulator: a per-joint two-degree-of-freedom feedforward–feedback controller plus a disturbance observer forces every physical actuator to reproduce the ideal second-order reference dynamics used in simulation, so that policies trained purely in simulation deploy zero-shot on real hardware — with no hardware-specific data collection, no learned actuator model, and no task-level fine-tuning.

1. Background and Motivation

GPU-accelerated physics simulation has produced striking locomotion and manipulation controllers trained with reinforcement learning (RL). Yet the sim-to-real gap — the performance drop when a policy trained in simulation is deployed on real hardware — remains the central obstacle of robot learning, and this paper argues that a major part of the gap lives at the actuator level. Rigid-body simulators model joints as ideal torque sources or simplified servo systems, while real actuators exhibit friction, stiction, backlash, transmission compliance, inertia mismatch, saturation, and electromechanical delays. These unmodeled effects distort the relation between commanded and realized joint motion, so a policy optimized in simulation drives the physical robot along unintended trajectories, degrading performance or even destabilizing dynamic motions.

The dominant strategy has been to make the simulator more realistic. Domain randomization (DR) trains policies over randomized physical parameters so the real system is likely to fall inside the training distribution, but this robustness is bought at the cost of task-specific performance. A complementary line augments the simulator with physics-based or learned actuator models identified from input–output data, or learns online adaptation modules that infer latent environment/actuator parameters at deployment. All of these share one structural dependency: they adapt the simulation or the policy to a particular hardware distribution, so hardware-specific data collection or identification must be repeated whenever actuators are replaced, degrade over time, or differ across platforms. This limits the reusability of policies trained in simulation.

Figure 1: The idea of actuator reality shaping — rather than making the simulator imitate reality (right branch), the physical actuator is shaped to imitate the simulator's ideal model (left branch), so that a policy trained against the reference model transfers zero-shot.

Related-work analysis sharpens the point. Learned actuator models such as the actuator network in Hwangbo et al. (2019) or ASAP's Delta Action policy correct the dynamics residual at the policy level, but most demonstrations rely on quasi-direct-drive actuators with low gear ratios, whose torque response approximates an ideal current source. High-gear-ratio joints violate this assumption: reflected friction, stiction and backlash dominate the joint torque and cannot be captured as a small parametric perturbation. System-identification and online-adaptation approaches (RMA-style latent codes, adaptive controllers) work, but every hardware change forces a new identification round. The authors therefore take the dual perspective: rather than identifying the hardware so the policy adapts to it, shape the hardware's closed-loop response at the driver level so the policy perceives a fixed, simulator-defined plant on every platform.

The classical building blocks already exist. Two-degree-of-freedom (2-DoF) control decouples reference tracking from disturbance rejection through independent feedforward and feedback design, and is commonly combined with a disturbance observer (DOB) to reject friction and load. But prior work operates at the level of individual joint control and was never established as a sim-to-real interface layer for whole-body learned motion. This paper's design shares the reference-model layer idea with a fixed-gain 2-DoF controller derived from a minimal actuator model — no convergence monitoring, no gain scheduling, no adaptation–saturation stability issues, and no hardware adaptation data.

2. Problem Formulation: MDP in the Actuator and the Actuator Gap

Consider a multi-DoF robot with $n$ revolute joints actuated by DC motors. An RL policy $\pi:\mathcal{O}\to\mathcal{A}$ maps observation $o_t$ to a goal joint angle $\bar{\theta}_t$. In simulation, joint $j$ is governed by a PD controller:

$$I_{\mathrm{sim}}\ddot{\theta}_j = K_p(\bar{\theta}_j - \theta_j) - D\dot{\theta}_j + f_j$$

where $I_{\mathrm{sim}}$, $K_p$, $D$ are the simulated inertia, proportional gain and damping, and $f_j$ is the net external joint torque. The MDP transition operator is defined by this dynamics, and the policy is optimized against it (e.g., with PPO). The formulation implicitly assumes the commanded torque $\tau_{\mathrm{cmd}} = K_p(\bar{\theta}_j-\theta_j) - D\dot{\theta}_j$ is produced instantaneously at the joint. On real hardware, joint $j$ is driven by a motor with its own dynamics:

$$J_{\mathrm{real}}\ddot{\theta}_j = K_{\tau i}(i_j - i_{d,j}) - d_{\mathrm{real}}\dot{\theta}_j - \tau_{f,j}(\dot{\theta}_j) - \tau_{\mathrm{ext},j}$$

with actual rotor inertia $J_{\mathrm{real}}$ (including gear reflection), torque constant $K_{\tau i}$, current command $i_j$, disturbance current $i_{d,j}$ (e.g., cogging), viscous damping $d_{\mathrm{real}}$, velocity-dependent friction $\tau_{f,j}$ (Coulomb friction and stiction), and external load torque $\tau_{\mathrm{ext},j}$. Linearizing around the operating point, each joint is described by plant transfer functions $P_{\mathrm{sim}}(s) = 1/(I_{\mathrm{sim}}s^2 + Ds)$ and $P_{\mathrm{real}}(s) = 1/(J_{\mathrm{real}}s^2 + d_{\mathrm{real}}s)$, with the nonlinear terms entering as a lumped disturbance torque. The actuator gap is the multiplicative uncertainty of the real plant relative to the simulated one:

$$\Delta P_j(s) = \frac{P_{\mathrm{real}}(s)}{P_{\mathrm{sim}}(s)} - 1 = \frac{I_{\mathrm{sim}}s + D}{J_{\mathrm{real}}s + d_{\mathrm{real}}} - 1$$

This dimensionless gap captures inertia mismatch and damping mismatch jointly; on high-gear-ratio servos it can be very large because the motor-side inertia reflected through $N^2$ dominates joint-level dynamics.

3. Method: Per-Joint Actuator Reality Shaping

Figure 2: The actuator reality shaping architecture for a single joint — a cascaded 2-DoF controller (position outer loop wrapping a velocity inner loop) augmented with a disturbance observer drives the real motor so that its closed-loop response tracks the simulator's reference model.

flowchart TB
    A["RL policy pi
goal joint angle theta_bar"] --> B["Outer loop 2-DoF
F1: second-order reference model"]
    B --> C["Goal velocity theta_dot_bar"]
    C --> D["Inner loop 2-DoF
F2: velocity reference model"]
    E["DOB: disturbance estimate
low-pass cutoff alpha_dob"] --> F
    D --> F["Sum and saturate
i = clip((tau_ff + tau_fb - tau_d_hat) / K_tau_i)"]
    F --> G["Real motor + gear
friction, backlash, load"]
    G -->|"theta, theta_dot at 500 Hz"| D
    G -->|"theta, theta_dot at 500 Hz"| E
    style B fill:#e0f2fe,stroke:#0284c7
    style D fill:#e0f2fe,stroke:#0284c7
    style E fill:#fef9c3,stroke:#ca8a04

Two-degree-of-freedom controller. Adopting the standard 2-DoF structure, the control input is

$$u = \frac{F(s)}{\hat{P}(s)} r + K(s)e, \qquad e = F(s)r - y$$

where $r$ is the reference (the policy goal angle $\bar{\theta}$), $y$ is the measured angle $\theta$, $F(s)$ is the desired response block, $K(s)$ the feedback block, and $\hat{P}(s)$ a nominal plant. Writing the true plant as $P(s) = (1+\Delta P(s))\hat{P}(s)$, the closed-loop from reference and disturbance to output becomes (derivation in Appendix B):

$$y = \underbrace{\left(1 + \frac{\Delta P}{1 + PK}\right)}_{\approx\,1} F r + \underbrace{\frac{P}{1 + PK}}_{\text{sensitivity}} d$$

When the feedback gain $K$ is large enough to compensate the model error, the output tracks $y \approx Fr$ regardless of the disturbance loop, so $F$ can be designed independently to match the simulator's reference dynamics. The authors choose

$$F_1(s) = \frac{K_p}{I_{\mathrm{sim}}s^2 + Ds + K_p}, \qquad F_2(s) = \frac{\alpha}{s + \alpha}$$

so the outer-loop plant is $P_1 \approx F_2/s$ because the inner velocity loop closely follows its reference model $F_2$. By the internal model principle the feedback block must contain an integrator for zero steady-state error under constant disturbances; a PID controller is used: $K(s) = k_p + k_i/s + k_d s$.

Disturbance observer. The 2-DoF controller is augmented with a DOB that estimates the lumped disturbance torque from the commanded torque $\tau = K_{\tau i} i$ and measured velocity $\dot{\theta}$. From the motor equation $J\ddot{\theta} = \tau - d\dot{\theta} - \tau_d$, the disturbance can be extracted as $\tau_d = \tau - (sJ + d)\dot{\theta}$; since differentiating velocity amplifies encoder noise, the estimate is low-pass filtered:

$$\hat{\tau}_d = \frac{\alpha_{\mathrm{dob}}}{s + \alpha_{\mathrm{dob}}}\left(\tau - (sJ + d)\dot{\theta}\right)$$

where $\alpha_{\mathrm{dob}}$ is the DOB low-pass cutoff in rad/s. The estimated disturbance is cancelled by subtracting a compensation current $i_{\mathrm{comp}} = \hat{\tau}_d / K_{\tau i}$ from the command before it reaches the driver, attenuating residual model error below $\alpha_{\mathrm{dob}}$. Notably, only a coarse one-shot fit of the motor model ($K_{\tau i}$, $J$, $d$ from a pendulum free-response) is needed: the residual model error is cancelled at runtime by the DOB rather than removed offline — the authors characterize accuracy through the DOB's reconstruction error (0.37 N·m for YM070, 0.66 N·m for PH54, measured against a calibrated torque gauge).

Cascaded position–velocity control and 500 Hz loop. Position-controlled joints cascade two 2-DoF loops: an inner velocity loop with plant $\hat{P}_2 = 1/(Js + d)$ and reference $F_2$, and an outer position loop whose reference model $F_1$ is exactly the simulator's second-order response, outputting a goal velocity $\bar{\dot{\theta}}$ for the inner loop. The controller runs on the motor driver at 500 Hz, executing per step: (1) read encoder angle and compute velocity; (2) compute feedforward torque $\tau_{\mathrm{ff}} = (F/\hat{P})r$; (3) compute feedback torque $\tau_{\mathrm{fb}} = K(Fr - \theta)$; (4) compute the DOB estimate $\hat{\tau}_d$; (5) sum and saturate $i = \mathrm{clip}((\tau_{\mathrm{ff}} + \tau_{\mathrm{fb}} - \hat{\tau}_d)/K_{\tau i}, -i_{\mathrm{max}}, i_{\mathrm{max}})$; (6) send the current to the driver. The velocity fed to the DOB passes an extra low-pass filter at $0.5\alpha_{\mathrm{dob}}$ to reduce encoder noise.

4. Experimental Setup

The method is evaluated on platforms of increasing complexity: a single-joint testbench, a 7-DoF arm reaching task, and zero-shot demonstrations on a 31-DoF wheeled-legged robot and a 22-DoF humanoid — all three custom-built in-house. Because their Isaac Sim models come directly from CAD-derived URDFs without the implicit tuning of commercial simulation assets, the sim-to-real gap on these platforms is more severe than on off-the-shelf hardware, making them a stringent testbed. The single-joint experiments use the same actuator unit deployed on the robots: a high-gear-ratio servo with cycloidal reducer (Dynamixel YM070, 99:1), so identified parameters and gains transfer directly; the humanoid uses the Dynamixel PH54 (501.9:1). The high gear ratio reflects link-side inertia through $N^2$, so motor-side inertia dominates joint dynamics by orders of magnitude and inter-joint inertial coupling appears at the DOB as a low-frequency bounded perturbation within the rejection bandwidth — validating the per-joint decentralized design. Policies are trained with PPO in Isaac Sim/Isaac Lab against the reference dynamics; the humanoid walking task builds on the public BeyondMimic implementation, replacing only the actuator model and low-level controller. Baselines, all at the same 500 Hz driver frequency: PID Cascade (the servo's factory default cascaded position–velocity–current PID), Simple PD (single-loop, manually tuned), and ASAP (a real-to-sim-to-real Delta Action model trained on real rollouts of the arm).

5. Results

Single-axis tracking. On a 20 s sinusoidal reference (20° amplitude, 2.5 s period, 5 runs) with the YM070, phase portraits show PID Cascade and Simple PD leaving a large sim-to-real gap — they track the commanded goal angle rather than the reference dynamics, producing systematic bias along the velocity-error axis — whereas the 2-DoF and 2-DoF+DOB clouds stay near the origin because the feedforward filter $F$ removes phase lag without overshoot. Quantitatively, 2-DoF+DOB reduces the mean position deviation by 96.3% relative to PID Cascade and 94.5% relative to Simple PD, and cuts the mean velocity deviation — the cascade's dominant error mode — by 99.1%.

Method	$|\Delta\theta|$ tracking [rad]	$|\Delta\dot{\theta}|$ tracking [rad/s]	$\|\Delta p\|$ reaching [m]	$\|\Delta\dot{p}\|$ reaching [m/s]
PID Cascade	(4.9±2.7)e-2	3.6±1.5	(3.7±3.5)e-2	(6.1±2.8)e-1
Simple PD	(3.3±5.8)e-2	(1.7±7.5)e-1	(4.7±4.1)e-3	(6.5±5.8)e-2
2-DoF	(2.6±1.5)e-3	(1.3±1.4)e-2	(2.6±2.0)e-3	(1.0±0.6)e-1
2-DoF + DOB	(1.8±1.3)e-3	(3.2±3.1)e-2	(1.3±0.9)e-3	(1.8±0.7)e-2

Table 1: Mean ± std deviation from the simulated reference trajectory over 5 trials, in joint space for single-axis tracking and task space for the lemniscate reaching task.

Figure 3: Comparison with ASAP on the reaching task — end-effector position error (left) and orientation error (right). ASAP's residual reduces the gap but destabilizes training; actuator reality shaping attains good zero-shot performance without fine-tuning.

Reaching task. The 7-DoF arm tracks a lemniscate (figure-eight) trajectory; the policy trained against the ideal reference model in Isaac Sim transfers zero-shot via 2-DoF+DOB. In the steady-state window (5 runs), 2-DoF+DOB attains the lowest end-effector deviation — a 96.5% reduction relative to the factory PID cascade, with the DOB accounting for a further 50% over 2-DoF alone, and a 72% reduction over the tuned Simple PD baseline, because uncompensated friction in PD and cascade controllers introduces a systematic offset the policy cannot anticipate. Against ASAP, the Delta Action residual reduced the gap but had to output larger actions than the base policy, destabilizing simulation training (loss curves in Appendix H); real-robot performance was not substantially better than the untuned factory default.

Figure 4: Zero-shot demonstrations on other embodiments: (a) the wheeled-legged robot climbing a slope; (b) the humanoid robot's zero-shot walk.

Zero-shot transfer across embodiments. On the wheeled-legged robot, applying the 2-DoF driver layer to velocity-controlled wheel actuators lets a policy trained against the reference model climb a slope zero-shot — the slope height must be inferred from per-joint states and IMU posture alone, which is hard for poorly backdrivable high-gear-ratio servos, yet the controller succeeds by reproducing the simulator's compliant spring–damper behavior. On the humanoid (different servo, PH54), a walking policy trained with the off-the-shelf BeyondMimic framework — swapping only the actuator model for the ideal reference model — transfers without hardware-specific tuning and takes stable forward steps. The approach thus carries over to a high-DOF, dynamically balancing whole-body platform.

Task	Network (MLP)	PPO iterations	Rollout/env	Parallel envs
Reaching	[512, 256]	4,000	128	4,096
Rover	[512, 256, 128]	10,000	64	4,096
Humanoid	[512, 256, 128]	30,000	24	4,096

Table 2: PPO training hyperparameters per task (Appendix F).

Parameter	PH54	YM070
Torque constant $K_{\tau i}$ [N·m/A]	4.8	1.389
Nominal inertia $J$ [kg·m²]	0.0763	0.0152
Nominal viscous damping $d$ [N·m·s/rad]	0.02	1.0
Time constant for $F_2$: $\alpha$ [rad/s]	15	30
DOB cutoff $\alpha_{\mathrm{dob}}$ [rad/s]	10	200
PID gain (position) $k_p, k_i, k_d$	[15, 0, 0.02]	[50, 10, 1]
PID gain (velocity) $k_p, k_i, k_d$	[3, 3, 0.01]	[15, 8, 0.1]
Sim. spring gain $K_p$ [N·m/rad]	50	30
Sim. damping $D$ [N·m·s/rad]	3.16	3.46
Sim. inertia $I_{\mathrm{sim}}$ [kg·m²]	0.05	0.1

Table 3: Controller parameters used in experiments (Appendix E). Hardware parameters ($K_{\tau i}$, $J$, $d$) are identified from a pendulum free-response fit; controller and simulator parameters are design choices.

6. Limitations

(1) The authors explicitly note that the method treats joint-axis inertia as a constant $I_{\mathrm{sim}}$, whereas in reality it depends on whole-body configuration $I_{\mathrm{sim}}(q)$; this is negligible at high gear ratios but not for quasi-direct-drive actuators, where accounting for it (e.g., via model reference adaptive control) is a promising direction. (2) The nominal $J$ and $K_{\tau i}$ still require a coarse estimate — a pendulum free-response fit — so the approach is not completely identification-free. (3) Whether the approach extends to strongly nonlinear hardware remains open, per the authors' own statement. From a reviewer's perspective, additional caveats include: all platforms are in-house builds with the same high-gear-ratio servo family, so generality across actuator technologies (harmonic drives, tendon drives) is unverified; the DOB cutoff is much lower for the PH54 (10 rad/s) than the YM070 (200 rad/s), suggesting tuning sensitivity to motor characteristics; and the humanoid walking evidence is a qualitative zero-shot demonstration rather than a quantified benchmark against baselines.

7. Conclusion and Outlook

Actuator reality shaping relocates the sim-to-real adaptation burden from the simulator and the policy to a transparent, hardware-agnostic driver layer: a per-joint two-degree-of-freedom controller that, using only a minimal motor model, drives real actuators to replicate the simulator's second-order reference dynamics, enabling zero-shot transfer without hardware-specific data. For teams building custom robots — where URDFs come from CAD and commercial sim assets do not exist — this reframes the problem from "model the motor perfectly" to "make the motor behave like the simple model," which is a much more tractable engineering target.

Figure 5: Single-axis sinusoidal tracking (left) and per-joint tracking during reaching under external disturbances (right) — the 2-DoF+DOB response stays glued to the reference where the baselines drift.

Figure 6: ASAP Stage-A training-loss curves (paired runs) — evidence that fitting the Delta Action residual destabilizes simulation training, explaining why ASAP's real-robot gains were limited.

8. Golden Sentence

"Instead of modifying the simulator to match the real world, our method shapes the closed-loop behavior of physical actuators to match the idealized second-order reference dynamics used in simulation."

When the hardware becomes the interface, zero-shot is no longer luck — it is the default.