Skip to content
RobotWorld
Back to Papers

PAPER DEEP DIVE

步态优化强化学习SAC

FastDSAC: Enhancing Policy Plasticity via Constrained Exploration for Scalable Humanoid Locomotion

Scalable reinforcement learning has popularized high-throughput sampling architectures, which significantly compresses the training time for off-policy methods in robotic locomotion. However, the rapid increase of data volume and update frequency undermines the stability of value-based methods and diminishes the plasticity of policy networks. To address these challenges, this work presents FastDSAC, a fast and high-performance variant of the Distributional Actor-Critic algorithm designed for parallel sampling scenarios. Specifically, we introduce a truncated Gaussian distribution to approximate the learned policy, which effectively excludes out-of-distribution actions that strain target value estimation while keeping necessary stochasticity for exploration. The proposed action constraint functions as an implicit regularization, which counteracts the plasticity loss typically caused by aggressive gradient updates. This preservation of network adaptability enhances sample efficiency, particularly in scenarios with a high update-to-data ratio, and accelerates the early training process. In contrast to prior fast reinforcement learning approaches that rely on discrete value distributions, our method utilizes a continuous Gaussian representation equipped with adaptive variance regulation, which improves value estimation accuracy by sampling confident and informative transitions. Extensive experiments on MuJoCo Playground and HumanoidBench demonstrate that FastDSAC not only stabilizes the overall training process but also achieves superior asymptotic performance and faster convergence compared to state-of-the-art baselines.

Guanchen Lu, Yajuan Dun, Yi Zhou, Letian Tao, Jingliang Duan, Jie Li, Guofa LiJune 30, 20267 min read
中文

FastDSAC: Enhancing Policy Plasticity via Constrained Exploration for Scalable Humanoid Locomotion

Paper: FastDSAC: Enhancing Policy Plasticity via Constrained Exploration for Scalable Humanoid Locomotion
Authors: Chongqing University / Tsinghua University / USTB team
Link: arXiv:2606.31691 | Code: github.com/luge66/FastDSAC | Benchmarks: MuJoCo Playground + HumanoidBench

One-line summary: To address instability of off-policy value-based methods under high-throughput parallel sampling, a mean-centered truncated Gaussian constrains target actions, filtering extreme actions to reduce TD target variance while preserving exploration stochasticity, achieving faster convergence and higher asymptotic performance on humanoid locomotion tasks.

Background and Motivation

Scalable reinforcement learning has popularized high-throughput sampling architectures that significantly compress training time for off-policy methods in robotic locomotion. However, the rapid increase of data volume and update frequency undermines the stability of value-based methods and diminishes the plasticity of policy networks. Massively parallel simulation on a single workstation makes wall-clock efficiency and rapid iteration first-class objectives. PPO, SAC, and TD3 are increasingly evaluated under high-throughput training regimes.

The core problem stems from inaccurate target Q-value estimates. In off-policy actor-critic methods, target Q-values are estimated using target Q-networks. During high-throughput training, large mini-batches increase the number of target actions sampled from the target policy, making extreme low-probability actions more likely, yielding higher-variance target Q-value estimates and TD errors. This destabilizes critic learning and slows policy improvement. Additionally, aggressive gradient updates cause plasticity loss — the network loses the ability to adapt to new data.

Existing methods like Parallel Q-Learning study distributional critic scaling but rely on asynchronous parallelism. FastSAC starts fast but becomes unstable later. FastDSAC's core idea: decouple exploration from target Q-value estimation, imposing regularization only on the target action when the target Q-network forms the TD error — using mean-centered truncated Gaussian to filter extreme target actions while preserving mild exploration stochasticity.

Method Details

1. Gaussian Policy Parameterization

The policy network outputs diagonal Gaussian mean $\mu_{\phi}(s)$ and standard deviation $\sigma_{\phi}(s)$, i.e., $\pi_{\phi}(a|s)\sim\mathcal{N}(\mu_{\phi}(s),\sigma_{\phi}^{2}(s))$. The reparameterization trick enables gradient backpropagation:

$$a=\mu_{\phi}(s)+\sigma_{\phi}(s)\odot\epsilon,\quad\epsilon\sim\mathcal{N}(0,I)$$

Compared to the commonly used tanh-squashed parameterization, this design avoids vanishing gradients when actions approach bounds and avoids the log-likelihood correction numerical issues from tanh squashing. The policy mean $\mu_{\phi}(s)$ is bounded within $[m_{\min},m_{\max}]$ and standard deviation $\sigma_{\phi}(s)$ within $[\ell_{\min},\ell_{\max}]$ for stability.

2. Mean-Centered Truncation for Target Actions

Given next state $s'$, first sample a pre-truncation action:

$$u^{\prime}=\mu_{\phi}(s^{\prime})+\sigma_{\phi}(s^{\prime})\odot\epsilon,\quad\epsilon\sim\mathcal{N}(0,I)$$

Then obtain the target action via mean-centered truncation mapping:

$$a^{\prime}=\mu_{\phi}(s^{\prime})+c\,\tanh\!\big(u^{\prime}-\mu_{\phi}(s^{\prime})\big)$$

The elementwise $\tanh(\cdot)$ smoothly squashes the offset, bounding each component of $a'-\mu_{\phi}(s')$ strictly within $[-c,c]$. Compared to hard clipping, this mapping is continuous, avoids boundary discontinuities, and yields smoother target evaluation. This mapping is applied only when evaluating the target Q-value $Q_{\theta'}(s',a')$ at the next state.

FastDSAC framework overview

Figure 1: FastDSAC framework. Mean-centered truncation constrains target actions only during target Q-value evaluation; exploration actions are unaffected.

3. Induced Distribution Log-Probability

Since target Q-values are evaluated at the truncated action $a'$, the entropy term must use the log-probability of the corresponding induced action distribution. Via change-of-variables:

$$\log\pi_{\phi}(a^{\prime}|s^{\prime})=\log\pi_{\phi}(u^{\prime}|s^{\prime})-\log\!\left|\det\!\left(\frac{\partial a^{\prime}}{\partial u^{\prime}}\right)\right|$$

The Jacobian determinant is the product of truncation mapping derivatives, ensuring the entropy term is consistent with the constrained target action.

4. Critic and Actor Updates

The target return uses the minimum Q-value:

$$y=r+\gamma\big(Q_{\theta^{\prime}}^{\min}(s^{\prime},a^{\prime})-\alpha\log\pi_{\phi}(a^{\prime}|s^{\prime})\big)$$

The critic loss maximizes the likelihood of $y_z$ under the Gaussian parameterization:

$$J_{Z}(\theta)=\mathbb{E}\!\left[\frac{(y_{z}-Q_{\theta}(s,a))^{2}}{2\sigma_{\theta}^{2}(s,a)}+\log\sigma_{\theta}(s,a)\right]$$

The actor is learned by maximizing the critic mean objective:

$$J_{\pi}(\phi)=\mathbb{E}\!\left[Q_{\theta}(s,a)-\alpha\log\pi_{\phi}(a|s)\right]$$

Entropy temperature $\alpha$ adapts to match target entropy $\mathcal{H}^{\text{target}}$:

$$\alpha\leftarrow\alpha-\nabla_{\alpha}\frac{1}{|B|}\sum\big(\mathcal{H}^{\text{target}}-\mathcal{H}(s)\big)\cdot\alpha$$

5. Continuous Gaussian Distributional Critic

Unlike FastTD3 and FastSAC which use discrete value distributions with fixed intervals, FastDSAC employs a continuous Gaussian distributional critic with adaptively learned variance $\sigma_{\theta}(s,a)$. This improves value estimation accuracy and reduces overconfident updates in epistemically uncertain regions, retaining more high-reward transitions during interaction. Transition reward analysis shows FastDSAC achieves higher and more stable transition rewards than all baselines, indicating exploration is guided more consistently toward high-reward behaviors.

graph TD
  A[Sample exploration action a ~ N(μ,σ²)] --> B[Environment interaction + store in replay buffer]
  B --> C[Sample mini-batch from buffer]
  C --> D[Target action: a' = μ + c·tanh(u'-μ)]
  D --> E[Target Q: y = r + γ(Q'min - α·logπ')]
  E --> F[Update Critic: minimize TD error]
  F --> G[Update Actor: maximize Q-α·logπ]
  G --> H[Update entropy temperature α]
  H --> I[Soft update target networks]
  style D fill:#f5a623,stroke:#b97316,color:#fff
  style E fill:#4a90d9,stroke:#2c5f8a,color:#fff
  style F fill:#7ed321,stroke:#4a8a14,color:#fff

Experimental Results

Sample Efficiency and Time Efficiency

On four MuJoCo Playground locomotion tasks, PPO as an on-policy baseline improves slowly and plateaus at markedly lower returns. FastSAC rises quickly but becomes unstable later, showing plasticity loss signs leading to lower final returns. FastTD3 and DSAC-T have stable curves but more gradual improvement. FastDSAC combines fast early gains with stable late-stage learning, achieving the highest final return on all four tasks. On the more challenging G1JoystickRoughTerrain, FastDSAC matches FastSAC's early progress while continuing to improve without late-stage regressions.

MethodEarly ConvergenceLate StabilityFinal ReturnWall-clock Efficiency
PPOSlowLow plateauLowestSlowest
FastSACFastestUnstable/regressionLowerFast but high latency
FastTD3GradualStableMediumComparable to FastDSAC
DSAC-TGradualStableMedium-lowComparable to FastDSAC
FastDSACFastStable, no regressionHighestBest trade-off

Cross-Suite Generalization and Robustness

On six HumanoidBench locomotion tasks, FastSAC shows weaker and less consistent outcomes. FastDSAC remains competitive, exhibiting more stable learning and better preserving plasticity. FastDSAC is on par with or better than FastTD3, both substantially outperforming DSAC-T. Dynamics perturbation tests modifying mass parameters and ground friction show the robot maintains a stable, natural walking gait without posture degradation, providing preliminary qualitative evidence of policy robustness.

Learning curves comparison

Figure 2: Environment-step learning curves on four MuJoCo Playground tasks. FastDSAC combines fast early gains with stable late-stage learning.

Ablation: Truncation Radius c

Truncation radius $c\in\{10^{-2},10^{-3},10^{-4}\}$ mainly affects early learning speed while final returns remain relatively close. Larger $c$ preserves more target action variability, speeding early learning; smaller $c$ yields more conservative targets. $c=10^{-3}$ works well in simpler environments; $c=10^{-2}$ is better in complex environments. $c=10^{-3}$ is the robust default.

Ablation: Target Action Choice

Target Action ChoiceEarly ConvergenceComplex TerrainFinal Return
Mean-centered truncated a'FastBestHighest
Raw Gaussian sample u'CompetitiveSlower earlyMedium
Deterministic mean μ(s')SlowestWorstLowest

This shows retaining some stochasticity benefits target evaluation — fully deterministic mean loses diversity and exploration coverage, while unconstrained Gaussian increases variance. Mean-centered truncation balances both: preserving mild stochasticity while limiting extremes.

Truncation radius sensitivity

Figure 3: Sensitivity to truncation radius c. Only moderate sensitivity within practical range, mainly affecting early learning speed.

Limitations

Author-stated: Results suggest improved robustness under aggressive optimization but provide only indirect behavioral evidence of preserved network plasticity. Future work will incorporate plasticity-specific diagnostics, representation-level analyses, qualitative behavior evaluation, and sim-to-real experiments.

Analysis: Mean-centered truncation effectively reduces target Q-value variance, but the truncation radius $c$ still requires manual tuning or logarithmic sweep without an adaptive mechanism. The method constrains actions only during target Q-value evaluation, not addressing potential issues in the exploration policy itself. The Gaussian distributional critic, while more accurate than discrete distributions, has unverified variance estimation reliability in high-dimensional action spaces. Experiments are simulation-only; sim-to-real transfer is unverified. Plasticity loss mechanism analysis is insufficient — how truncation constraints specifically affect network gradient and representation dynamics needs deeper study. Systematic comparison with the latest large-scale training methods is limited.

Conclusion and Outlook

FastDSAC is a lightweight method for stabilizing entropy-regularized off-policy reinforcement learning under massively parallel sampling and large-batch updates. It applies mean-centered truncation exclusively to the target action in Bellman backups and computes the entropy term using the log-probability induced by the transformed distribution. By limiting extreme target actions, FastDSAC improves training stability with negligible computational overhead. Experiments demonstrate faster and more reliable learning, fewer late-stage regressions, and effectiveness at high update-to-data ratios. From a methodological perspective, the core idea of decoupling exploration from target evaluation — full stochasticity during exploration, constraints during target evaluation — provides a new paradigm for the stability-plasticity trade-off in high-throughput off-policy learning.

Explore boldly, but evaluate conservatively — leave aggressiveness to environment interaction, inject restraint into TD targets. This is FastDSAC's key insight for balancing stability and plasticity in high-throughput training.

Related Papers

Booster Lab: A Data-Centric Pipeline for Learning Deployable Humanoid Locomotion Policies

Booster Lab: A Data-Centric Pipeline for Learning Deployable Humanoid Locomotion Policies

Humanoid robot motion learning requires not only task-oriented control policies but also physically feasible and natural behaviors that can be transferred to real robots. However, robot-feasible motion data are often scarce: raw human demonstrations may be incompatible with the robot morphology, open-source clips vary in quality, and simulation-collected robot trajectories still require feasibility checking. To address these challenges, we propose a data-centric training and deployment pipeline that integrates motion data curation, real-to-sim model adaptation, AMP-based reinforcement learning, and sim-to-real deployment. We validate the framework on the Booster T1 robot and further provide preliminary cross-platform validation on Booster K1.

步态优化人形机器人AMPJun 26, 2026
X-Morph: Human Motion Priors for Scalable Robot Learning Across Morphologies

X-Morph: Human Motion Priors for Scalable Robot Learning Across Morphologies

Recent progress in humanoid behavior models has been driven in large part by abundant human motion data, but comparable motion data is scarce for non-humanoid legged robots such as quadrupeds, hexapods, and quadruped manipulators. A promising alternative is to repurpose human motion across embodiments; however, direct retargeting often produces motions that are visually plausible yet physically inconsistent or difficult to track under robot dynamics. We present X-Morph, a human-motion-to-robot-behavior pipeline that converts human motion into deployable locomotion and loco-manipulation policies for diverse non-humanoid legged morphologies. A cross-morphology retargeting stage converts human motions into kinematically plausible, intent-preserving robot references, which are then tracked by a privileged RL policy and distilled into a causal student policy. We evaluate X-Morph on three morphologically distinct platforms: a quadruped, a hexapod, and a quadruped equipped with a manipulator. The resulting policies track diverse retargeted motions, generalize to unseen human motions, and support downstream use cases including video-based teleoperation, behavior-prior control, and text-conditioned motion generation. These results suggest that large-scale human motion can serve as a substrate for learning broad, reusable behavior priors beyond humanoid robots. Project page: https://maker-rat.github.io/morph/

步态优化跨形态运动先验Jun 29, 2026
Multi-Rate Nonlinear Model Predictive Control for Wall-Supported Bipedal Locomotion of Quadrupedal Robots

Multi-Rate Nonlinear Model Predictive Control for Wall-Supported Bipedal Locomotion of Quadrupedal Robots

This paper presents a novel layered planning and control framework based on multi-rate nonlinear model predictive control (MR-NMPC) that enables quadrupedal robots to perform hybrid bipedal locomotion with wall-assisted support in constrained environments. Real-time trajectory optimization for this locomotion presents significant challenges, as the controller must simultaneously plan for both the contact points and the continuous trajectories of the robot's center of mass (CoM) and orientation within the robot's nonlinear dynamics while accounting for unilateral contact constraints, underactuation, and the switching nature of the robot's dynamics. At the high level of the control framework, an MR-NMPC is proposed, which dynamically plans both the discrete-time trajectories of the contact points and the continuous-time trajectories of the CoM and orientation, using a single rigid body (SRB) dynamics model. By incorporating contact-point planning within the multi-rate optimal control framework, this approach enhances dynamic stability compared to heuristic foot placement strategies. At the low level of the control framework, a nonlinear whole-body controller (WBC) based on virtual constraints and a quadratic program enforces full-order dynamics and tracks the MR-NMPC references. The proposed approach is validated through extensive numerical simulations demonstrating the robust wall-assisted bipedal locomotion of a Unitree A1 quadrupedal robot on rough terrains and under external disturbances in a constrained environment. Comparative analysis shows that the proposed MR-NMPC achieves a 2.9 times higher success rate compared to conventional MPC with heuristic-based foot placement strategies in negotiating irregular terrain at high speeds.

步态优化四足机器人MPCJul 2, 2026
Actuator Reality Shaping for Zero-Shot Sim-to-Real Robot Learning

Actuator Reality Shaping for Zero-Shot Sim-to-Real Robot Learning

Sim-to-real transfer in robot learning is often limited by discrepancies between the ideal actuator dynamics assumed during policy training and the nonlinear, hardware-dependent behavior of physical motors. While conventional approaches attempt to bridge this gap by increasing simulator fidelity through system identification, domain randomization, or learned actuator models, we introduce an alternative paradigm: actuator reality shaping. Instead of modifying the simulator to match the real world, our method shapes the closed-loop behavior of physical actuators to match the idealized second-order reference dynamics used in simulation. By equipping each joint with a two-degree-of-freedom feedforward--feedback controller, we decouple reference-response shaping from robust stabilization, thereby providing a standardized actuator interface for reinforcement learning policies. As a result, policies trained only with the prescribed reference model can be deployed zero-shot on real hardware without task-level fine-tuning or learned actuator models. We validate the approach on a single-joint high-gear-ratio servo under external loads and a 7-DOF robotic arm reaching task, where actuator reality shaping substantially reduces sim-to-real tracking error and improves zero-shot task performance compared with standard servo-control and representative real-to-sim-to-real baselines. We further demonstrate zero-shot transfer on a wheeled-legged robot driving over a slope and a humanoid robot walking, suggesting that actuator reality shaping can serve as a reusable interface for robot learning across diverse hardware platforms. Project page: https://syamamori.github.io/ActuatorRealityShaping.github.io/

步态优化Sim-to-Real执行器Jul 2, 2026