SteadyTray: Learning Object Balancing Tasks in Humanoid Tray Transport via Residual Reinforcement Learning

Paper: SteadyTray: Learning Object Balancing Tasks in Humanoid Tray Transport via Residual Reinforcement Learning (arXiv:2603.10306v1, 2026-03-11)
Authors: Anlun Huang*, Zhenyu Wu*, Soofiyan Atar, Yuheng Zhi, Michael Yip (UC San Diego; * equal contribution)
Code: released. The GitHub repository AllenHuangGit/SteadyTray (Apache-2.0) trains against the authors' IsaacLab fork AllenHuangGit/IsaacLab_SteadyTray, and ships four training tasks, a MuJoCo sim2sim deployment harness, and exported policy weights
Scope: humanoid bimanual loco-manipulation, residual reinforcement learning, privileged teacher-student distillation, and sim-to-real transfer on a 29-DoF Unitree G1

The Short Version

SteadyTray decomposes "carry a tray while walking" into two objectives that can be optimized separately. A pretrained gait policy is frozen and never touched again; a small residual module, made of a privileged observation encoder plus a zero-initialized adapter, is trained to cancel the oscillation that locomotion injects into the end effector; and DAgger then distills only the encoder so the deployed student needs nothing but proprioception and a camera estimate of the object pose. In simulation the residual variants reach 96.9% success on variable-velocity command tracking and up to 84.6% under pushes to the robot, versus 47.4% and 9.1% for the frozen base policy, and the same policy zero-shot carries a coffee cup, a water-filled wine glass, and surgical instruments on real G1 hardware.

Figure 1: the paper's teaser. The top row is a close-up sequence of the tray payload in simulation; the bottom row shows a Unitree G1 carrying a tray through an office environment, including a liquid-filled wine glass.

Background and Motivation

The paper opens with a labor statistic rather than a robotics one: in December 2025 U.S. employers reported 1.25 million openings in health care and social assistance and 0.81 million in accommodation and food services. What meal delivery, sterile instrument transport in an operating room, and elder-care assistance share is that locomotion and carrying happen simultaneously, inside spaces built for people, which means clutter, cables, and narrow corridors. A wheeled base cannot produce an active stabilization maneuver when it is shoved; a quadruped's payload geometry is constrained by its back. Bipedal humanoids, with a small planar footprint and the ability to actively balance, are argued to be the better carrier for exactly these environments.

Carrying, however, is much harder than walking. Footstrike is the dominant source of torso and base oscillation, and that disturbance propagates up the kinematic chain, so the upper body has to cancel it adaptively at the same time the legs are producing it. Prior work such as SoFTA suppresses oscillation of a single object that is suspended from, or rigidly and elastically attached to, the hand during locomotion. A tray breaks that assumption: the payload has no connection to the robot at all. It is held purely by friction and gravity, so turning, accelerating, decelerating, or absorbing a push can make it slide, tilt, or topple. The authors state plainly that, to their knowledge, no prior method had balanced an unattached object on a tray through common maneuvers such as turning, speed changes, or external pushes.

The physical root of the difficulty is that the tray inserts a non-rigid intermediate coupling. The end effector regulates the tray's motion, while the object is only passively constrained by it. The controller therefore faces a genuine conflict: generate agile whole-body locomotion, yet keep the end-effector attitude close to level. Cramming both into one end-to-end policy makes the gradients fight, and the experiments section turns that intuition into a measurement. The end-to-end baseline's total reward is not bad at all; most of it comes from the locomotion terms, while its object-upright reward stays far behind for the whole run.

The paper's answer is structural rather than reward-shaped. Instead of designing a larger monolithic policy, the authors concede that walking and stabilizing are skills with different time scales and different information sources, and separate them at the parameter level with a freeze-plus-residual construction. The base policy is responsible for not falling over; the residual module is responsible for actively cancelling what the gait transmits to the hands. This split is also convenient engineering: the base policy can reuse a community-validated locomotion recipe, and the residual module is small, trains fast, and cannot degrade an existing gait.

A third motivation is deployment. Privileged information, such as object linear velocity in the tray frame and the tray's projected gravity, is free in simulation and does not exist on hardware. The paper therefore separates "learn with privileged information" from "run with deployable information": a teacher stage trains the residual module on privileged observations, and a student stage distills the encoder alone while the entire adapter stays frozen, shrinking the real-robot input to proprioception plus a head-camera estimate of the object pose. That teacher-student link is the "Student-Teacher" in ReST-RL.

Preliminaries: Residual RL and Privileged Distillation

Residual reinforcement learning keeps an already-trained policy and learns a correction on top of its output or its intermediate features, updating only the correction. It originated as a way to compensate model error in sim-to-real transfer and has since been extended to motion tracking and cooperative transport. Its value is that exploration is confined to deviations from behavior that already works: when the correction is zero-initialized, the closed-loop system is bit-for-bit identical to the base policy at step zero, so training never has to destroy a skill in order to rebuild it.

Privileged teacher-student distillation is standard practice in legged robotics. The teacher trains on simulator-only state (linear velocity, contact forces, true object pose) and the student imitates it from deployable observations. SteadyTray's twist is what gets distilled: not the whole policy, but only the encoder inside the residual module. The adapter has already learned "how to correct, given the right features," so the student only has to learn "how to estimate those features from a camera." That compresses the imitation problem from action space into a 64-dimensional latent space and makes it dramatically easier.

Method

Figure 2: the ReST-RL framework. Top left is base-policy training; top right is residual-module training with privileged observations feeding the encoder; the bottom row shows the two adapter wirings — the action adapter superimposes a correction in action space, while the FiLM adapter modulates the frozen base policy's activations layer by layer.

Problem formalization: goal-conditioned policy, three observation streams

The task is written as goal-conditioned reinforcement learning with a policy $\pi:\mathcal{G}\times\mathcal{S}\to\mathcal{A}$, where $\mathcal{A}\subseteq\mathbb{R}^{n}$ is the vector of target joint positions for all humanoid joints. At time $t$ the goal $\mathbf{g}_{t}=[\hat{\mathbf{v}}^{r_{x}}_{t},\hat{\mathbf{v}}^{r_{y}}_{t},\hat{\boldsymbol{\omega}}^{r_{\text{yaw}}}_{t}]$ is the commanded robot linear and yaw velocity. Observations come in two streams. Proprioception $\mathbf{s}^{\text{prop}}_{t}=[\mathbf{q}_{t-H:t},\dot{\mathbf{q}}_{t-H:t},\boldsymbol{\omega}^{\text{r}}_{t-H:t},\mathbf{g}^{\text{r}}_{t-H:t},\mathbf{a}_{t-H:t-1}]$ holds joint positions, joint velocities, robot angular velocity, projected gravity, and previous actions over a history window. Object observations $\mathbf{s}^{\text{obj}}_{t}=[\mathbf{p}^{c}_{t-H:t},\mathbf{q}^{c}_{t-H:t}]$ are the object position and quaternion attitude in the camera frame. The base policy uses $H=5$ steps of history; the residual module uses $H=32$. The long window matters because a tray payload destabilizes gradually, and a five-step window simply cannot see the trend.

Stage one: the frozen locomotion base policy

The base policy $\pi^{\text{base}}:\mathcal{G}\times\mathcal{S}^{P}\to\mathcal{A}$ is first trained to walk stably while holding an empty tray. It maps proprioceptive history to target joint positions and is optimized with PPO to maximize the discounted return

$$\mathbb{E}\left[\sum_{t=1}^{T}\gamma^{t-1}r^{\text{base}}_{t}\right]$$

where $r^{\text{base}}_{t}=\mathcal{R}^{\text{base}}(\mathbf{g}_{t},\mathbf{s}^{\text{prop}}_{t})$ covers velocity tracking, torso stabilization, foot-impact penalization, and tray-to-end-effector contact and orientation terms. Once trained, its parameters are frozen completely: no gradient from any later stage is allowed to flow back into it. That is the single most important engineering constraint in the paper, because it guarantees that whatever the residual module learns, nominal gait performance cannot regress.

The residual module: privileged encoder plus two adapter flavors

The residual module is an encoder plus an adapter. In the teacher stage the encoder consumes privileged observations $\mathbf{s}^{\text{priv}}_{t}=[\mathbf{v}^{\text{r}}_{t-H:t},\mathbf{p}^{\text{tray}}_{t-H:t},\mathbf{g}^{\text{tray}}_{t-H:t},\mathbf{p}^{\text{obj}}_{t-H:t},\mathbf{v}^{\text{obj}}_{t-H:t},\boldsymbol{\omega}^{\text{obj}}_{t-H:t},\mathbf{g}^{\text{obj}}_{t-H:t}]$ — robot linear velocity, tray position and projected gravity, and the object's position, linear velocity, angular velocity, and projected gravity — concatenated with proprioception and the goal, and emits a 64-dimensional context feature $\hat{\mathbf{z}}_{t}=\phi(\mathbf{s}^{\text{priv}}_{t},\mathbf{s}^{\text{prop}}_{t},\mathbf{g}_{t})$. Two adapter instantiations consume it.

The first is the residual action adapter, which regresses a correction action $\tilde{\mathbf{a}}_{t}$ directly from $[\hat{\mathbf{z}}_{t};\mathbf{s}^{\text{prop}}_{t};\mathbf{g}_{t}]$ and superimposes it on the base action before the low-level controller:

$$\hat{\mathbf{a}}_{t}=\alpha_{\text{base}}\,\mathbf{a}_{t}+\alpha_{\text{residual}}\,\tilde{\mathbf{a}}_{t}+\mathbf{q}_{\text{default}} \quad (1)$$

Here $\alpha_{\text{base}}$ and $\alpha_{\text{residual}}$ scale the two action paths and $\mathbf{q}_{\text{default}}$ is the default joint configuration. The second is the residual FiLM adapter, which leaves the action alone and instead applies a feature-wise affine modulation to every linear layer inside the frozen base policy. For a frozen layer with output $\mathbf{y}=f(\mathbf{x})$, the adapter predicts $(\boldsymbol{\gamma}_{t},\boldsymbol{\beta}_{t})$ from $\hat{\mathbf{z}}_{t}$ and computes

$$y^{\prime}_{i}=y_{i}\bigl(1+\gamma_{t,i}\bigr)+\beta_{t,i},\quad i=1,\dots,d \quad (2)$$

Both adapters are zero-initialized: at the start of training the correction is identically zero, closed-loop behavior matches the base policy exactly, and the gait skill is fully preserved. Task-specific behavior grows in only as training proceeds. PPO jointly optimizes the encoder and adapter parameters to maximize $\mathbb{E}[\sum_{t}\gamma^{t-1}r^{\text{residual}}_{t}]$. Implementing two structurally different adapters is deliberate — it lets the authors answer whether the gain comes from the residual construction itself or from one particular way of wiring it. The experiments say the former.

Privileged distillation: encoder only, adapter frozen

Privileged observations do not exist on hardware, so a student version is required. Distillation runs as DAgger and touches the encoder alone. The student encoder $\phi^{\prime}$ sees only object observations over a 32-step history plus proprioception and the goal, producing $\mathbf{z}_{t}=\phi^{\prime}(\mathbf{s}^{\text{obj}}_{t},\mathbf{s}^{\text{prop}}_{t},\mathbf{g}_{t})$; the frozen adapter maps $\mathbf{z}_{t}$ to the student action $\hat{\mathbf{a}}^{\text{student}}_{t}$. The teacher encoder runs on the same rollout to produce $\hat{\mathbf{z}}_{t}$, which the same frozen adapter turns into $\hat{\mathbf{a}}^{\text{teacher}}_{t}$. The student is trained with the joint loss

$$\mathcal{L}=\mathcal{L}_{z}+\mathcal{L}_{a}=\|\mathbf{z}_{t}-\hat{\mathbf{z}}_{t}\|_{2}^{2}+\|\hat{\mathbf{a}}^{\text{student}}_{t}-\hat{\mathbf{a}}^{\text{teacher}}_{t}\|_{2}^{2} \quad (3)$$

Latent alignment keeps the student's features on the teacher's distribution; action alignment makes sure that even a biased feature estimate still produces teacher-like behavior. Freezing the adapter means the "features to correction" mapping learned in the teacher stage transfers to hardware untouched, so distillation error can only originate in the encoder. The problem is made small on purpose.

Figure 3: distillation structure. The teacher encoder (frozen, purple lock) consumes privileged observations, the student encoder consumes object observations, and both feed the same frozen adapter to produce actions; the loss is latent L2 plus action L2.

Domain randomization, control delay, and observation delay

Training randomizes mass, friction, and restitution for robot, tray, and object, along with torso center of mass, and injects control delay to cover actuation and communication latency. Objects are sampled as cylinders of varying dimensions with a random horizontal offset and yaw around the tray center, which prevents overfitting to one initial configuration; velocity commands are zeroed during the first second of each episode to give the payload a stable initialization window; and random pushes on the object during training force the residual module to learn recovery rather than just prevention.

The most interesting detail is observation delay. A latency $\ell_{t}$ is applied only to the object-related observations $(\mathbf{s}^{\text{obj}}_{t},\mathbf{s}^{\text{priv}}_{t})$, so at time $t$ the residual module reads measurements from $t-\ell_{t}$, and each sampled latency is held for several consecutive steps to mimic time-correlated perception lag. The rationale is that object perception is inherently laggier than proprioception. The ablation shows that this ostensibly sim-to-real trick also improves the zero-latency case, because the module is forced to estimate from history instead of instantaneous values — which is itself a stabilizing regularizer.

Reward design: a deliberately sparse object reward and a three-stage curriculum

The reward has two layers. The base reward $r^{\text{base}}_{t}$ reuses established locomotion terms (velocity tracking, torso linear-velocity and roll/pitch rate suppression, foot impact, joint limits, and action-rate regularization) and adds tray terms: a tray-to-end-effector contact indicator, a tray-to-end-effector quaternion error, tray uprightness, and tray velocity suppression. Waist deviation is penalized only lightly, because a stiff waist constraint would cripple the policy's ability to use its upper body to actively absorb disturbances. The object reward $r^{\text{obj}}_{t}$ is intentionally sparse, with just two terms:

$$r^{\text{obj}}_{t}\supset \exp\!\big(-\lambda_{\text{upright}}\big\|\mathbf{P}_{xy}\big(\mathbf{g}^{\text{obj}}_{t}\big)\big\|_{2}^{2}\big)+\mathbf{1}_{\{\text{object\_tray\_contact}\}}$$

That is, the lateral component of the object's own projected gravity should be small (the more upright, the better), and the object should stay in contact with the tray. No position or velocity target is given, so under a push the adapter has to discover its own recovery strategy instead of imitating a prescribed trajectory. The residual module's total objective is $r^{\text{residual}}_{t}=r^{\text{base}}_{t}+r^{\text{obj}}_{t}$.

Training proceeds through three inherited stages: stage one pretrains locomotion, stage two adds tray-holding rewards and fine-tunes, stage three adds the object-stabilization reward and trains the residual module. Each stage inherits all previous reward terms and modifies or disables individual ones as needed. Failure conditions — tray or object dropped, tilt angle too large — do not terminate the episode immediately. Instead a delayed timeout is used: the episode is truncated only after the condition has persisted for 1.0 second. This preserves sample efficiency and, more importantly, exposes the policy to post-failure states so it learns to keep walking stably inside them. On hardware an unseen object state can trigger unusual behavior, which makes this directly relevant to safe deployment; truncating rather than terminating also keeps value bootstrapping intact, so the critic never learns the false lesson that post-failure states are worth zero.

Stage one also carries an acceleration trick: an upper-body curriculum loss. A whole-body policy has an action space large enough that the optimizer wastes early iterations exploring arm motions alongside gait. The authors add a term to the PPO objective that penalizes both the mean and the standard deviation of the action distribution over a designated set of upper-body joint indices $\mathcal{J}_{\mathrm{upper}}$:

$$\mathcal{L}_{\mathrm{upper}}=\alpha\left(\frac{\sum_{b,j}\mu_{b,j}^{2}}{|\mathcal{B}||\mathcal{J}_{\mathrm{upper}}|}+\frac{1}{2}\cdot\frac{\sum_{b,j}\sigma_{b,j}^{2}}{|\mathcal{B}||\mathcal{J}_{\mathrm{upper}}|}\right) \quad (4)$$

The L2 penalty on $\mu$ pulls upper-body actions toward the default posture, and the penalty on $\sigma$ suppresses exploration at those joints, so early gradient budget goes entirely to the legs. $\alpha$ can be annealed on a curriculum, and the regularizer is used in stage one only — later stages release the upper body to learn tray holding and object stabilization.

Pipeline overview

flowchart TD
  S1["Stage 1 Pretrain locomotion
Upper-body curriculum loss pins the arms
PPO optimizes the base policy"]
  S2["Stage 2 Tray-hold fine-tune
Add tray upright, contact and quaternion alignment rewards"]
  S3["Stage 3 Residual teacher training
Base policy frozen
Privileged encoder + action or FiLM adapter
Zero-initialized, PPO"]
  S4["Stage 4 DAgger distillation
Adapter frozen
Student encoder sees camera object pose only
Latent plus action alignment loss"]
  DEP["Real-robot deployment, Unitree G1
Head RealSense D435 + AprilTag
estimates object pose"]
  S1 --> S2 --> S3 --> S4 --> DEP
  S3 -.->|"frozen base policy"| S4
  OBS["Privileged obs: object velocity, tray gravity"] --> S3
  CAM["Deployable obs: object pose, proprioception"] --> S4

Experiments

Simulation runs in Isaac Lab; hardware is a 29-DoF Unitree G1 whose end effectors were designed specifically for the tray. They support the tray and partially constrain its motion, but the tray is not rigidly attached to either side — which is precisely where the task difficulty comes from. On the robot, object pose is estimated from an AprilTag on the object plus the G1's head-mounted RealSense D435. Three tasks are evaluated: Command Track, where velocity commands are resampled every 10 seconds; Push Robot, where an external force is applied to the torso during locomotion; and Push Object, where the payload itself is pushed. Episodes run 20 seconds with a perturbation every 5 seconds. Metrics are success rate (object upright and on the tray for the whole episode), linear and angular velocity tracking error, and Grav-XY, the L2 norm of the xy components of the gravity unit vector expressed in the object frame — zero is perfectly upright, near one is nearly horizontal.

Table 1 gives the headline numbers. Across all three tasks every ReST-RL variant cuts Grav-XY sharply and lifts success rate, while tracking error stays comparable to the baselines — the stability gain is not bought by degrading command following. On Command Track, FiLM WB reaches 96.9% success at Grav-XY 0.046. On Push Robot, FiLM WB reaches 84.6% and Action WB 73.4%, against 9.1% for the frozen base policy and 44.0% for end-to-end. On Push Object, FiLM WB reaches 74.6% and Action WB 71.3%, against 25.2% and 50.2%. The three residual wirings (FiLM WB, Action WB, Action JT) land close together, which is why the authors attribute the gain to structured residual adaptation conditioned on object features rather than to any one adapter shape.

Task	Method	TrackLinErr (m/s)	TrackAngErr (rad/s)	Grav-XY	Success (%)
Command Track	Base Policy (WB)	0.110	0.078	0.179	47.4
Command Track	End2End	0.116	0.096	0.046	89.1
Command Track	ReST-RL (Action WB)	0.093	0.081	0.029	95.9
Command Track	ReST-RL (Action JT)	0.160	0.135	0.031	96.7
Command Track	ReST-RL (FiLM WB)	0.106	0.069	0.046	96.9
Push Robot	Base Policy (WB)	0.162	0.110	0.190	9.1
Push Robot	End2End	0.170	0.123	0.055	44.0
Push Robot	ReST-RL (Action WB)	0.146	0.110	0.039	73.4
Push Robot	ReST-RL (FiLM WB)	0.142	0.094	0.043	84.6
Push Object	Base Policy (WB)	0.110	0.079	0.186	25.2
Push Object	End2End	0.117	0.099	0.049	50.2
Push Object	ReST-RL (Action WB)	0.096	0.084	0.023	71.3
Push Object	ReST-RL (FiLM WB)	0.107	0.073	0.040	74.6

Table 1: main simulation results in Isaac Lab (excerpt of the paper's Table II, mean columns). All three tasks use identical velocity commands; lower Grav-XY means a more upright object.

The reward curves explain exactly why the end-to-end baseline fails. Its mean total reward is on par with ReST-RL and actually climbs earlier, but its object-upright reward plateaus around 0.9 while both residual variants reach about 1.8. Locomotion terms dominate the total, so the end-to-end policy masks stabilization failure behind a healthy walking score; the structured residual concentrates optimization pressure on the object instead. Figures 4 and 5 side by side are the most persuasive pair in the paper, because they convert "end-to-end is the wrong shape for this task" from an opinion into a measurable quantity.

Figure 4: mean total reward during training. End2End (green) reaches its plateau near 150 earliest, the ReST-RL action adapter (red) overtakes it after roughly 6k iterations and finishes highest, and the FiLM adapter (blue) converges slowest before catching up late. Total reward alone hides the stabilization failure; read it against Figure 5.

Figure 5: object-upright reward during training. End2End (green) stalls near 0.9 while the ReST-RL action adapter (red) and FiLM adapter (blue) climb to roughly 1.8, showing that total reward hides the stabilization objective.

The observation-delay ablation appears in Table 2. Under the Push Robot task and evaluated at perception latencies of 0.00, 0.02, 0.04, and 0.06 seconds, the delay-trained policy wins at every setting including zero latency (90.2 versus 78.4), and the margin widens monotonically to 78.3 versus 55.2 at 0.06 seconds. Delay randomization is therefore not merely a sim-to-real tax; it teaches the policy temporally consistent estimation under incomplete information, which is regularization in its own right.

Observation delay (s)	Success with delay training (%)	Success without delay training (%)	Gap (points)
0.00	90.2	78.4	+11.8
0.02	88.4	68.1	+20.3
0.04	83.7	61.0	+22.7
0.06	78.3	55.2	+23.1

Table 2: observation-delay ablation (values read from the paper's Figure 5, Push Robot task). Delay training leads at every evaluated latency, including zero.

Figure 6: observation-delay ablation. The horizontal axis is the perception latency injected at evaluation; blue is trained with delay, orange without. The gap grows monotonically with evaluated latency.

Perturbation robustness is reported as a polar success map. For Push Robot, success is gridded over push direction (forward, backward, left, right, and intermediates) and magnitude (rings at 32, 65, 98, and 130 N), yielding an overall 77.8% with only moderate degradation under the most extreme pushes; the body text reports polar means of 74.4% for pushing the robot and 73.9% for pushing the object. The velocity-tracking and recovery traces show torso velocity returning to command quickly after both command switches and pushes (robot pushed near 10 s and 13 s, object pushed near 16 s), with object velocity following closely and no sustained oscillation.

Figure 7: polar success map for Push Robot, indexed by push direction and magnitude with rings at 32/65/98/130 N; overall success 77.8%.

Figure 8: torso and object velocity response under command switches and external pushes. Velocity spikes from perturbations are absorbed within roughly a second, and object velocity stays tightly coupled to torso velocity.

On geometric generalization, the authors sweep success rate over cylinder radius-to-height ratios: performance stays high across a wide range and degrades only moderately at the extreme slender end (ratio 0.1). The hardware results, shown in the bottom row of Figure 1, are the more striking half. The same policy zero-shot carries a standard coffee cup, a water-filled wine glass, medical and surgical instruments, and a sealed food container — payloads with different mass distributions, geometry, and contact properties — with no retraining or fine-tuning. Under both kicks to the robot and pushes to the object, recovery is coordinated between upper and lower limbs and contains no abrupt corrective motion.

Training configuration

Both the teacher and distillation stages run on two NVIDIA RTX A6000 GPUs with 4096 parallel environments each, 8192 total, at 24 steps per environment. The base policy and adapter backbone MLPs are [512, 256, 128]; the Transformer history encoder uses a 64-dimensional context with 2 layers and 4 heads; the FiLM adapter has a 128-unit hidden layer with $\gamma$ clamped to ±3.0; the action adapter's residual MLP is [512, 256, 128]. PPO uses $\gamma=0.99$, GAE $\lambda=0.95$, clip 0.2, value coefficient 1.0, entropy coefficient 0.01, target KL 0.01, 5 epochs, 4 minibatches, and a gradient-norm cap of 1.0. Distillation uses a learning rate of $5\times10^{-5}$, 1 epoch, gradient-norm cap 0.5, and MSE loss.

Code Correspondence: Where the Method Lives in the Repository

The official repository AllenHuangGit/SteadyTray publishes the full training pipeline and sim2sim deployment code, and each equation in the paper maps onto a specific file. The FiLM modulation of Eq. (2) is implemented in FiLMAdapter.forward inside scripts/rsl_rl/adapter/adapter.py, and the code carries two details beyond the paper: besides the zero-initialized final modulation layer there is a learnable gate $\alpha$ initialized at 0.1, plus the $\gamma$ clamp (±3.0 per the config table), so the effective formula is $y^{\prime}=y(1+\alpha\gamma)+\alpha\beta$. The base linear layers have requires_grad explicitly disabled, which is the paper's "frozen" in code. Eq. (1) lives in ResidualActionAdapter.forward in the same file: the residual MLP takes the concatenation of context embedding and proprioception, its last layer is zero-initialized, and its output passes through the gate $\alpha$ before being added to the base action.

The two-term loss of Eq. (3) is in update() of scripts/rsl_rl/adapter/distillation.py: embedding loss and action loss are weighted at 1.0 each (embedding_loss_coef / action_loss_coef), the teacher latent and teacher action mean are cached during rollout and detached, and the optimizer updates only student-encoder parameters. The DAgger coefficient $\beta$ anneals linearly from 1.0 to 0.0 and controls the probability of executing the teacher action during rollout. The history encoder is TransformerEncoder in scripts/rsl_rl/adapter/encoder.py, whose defaults (ctx_dim=64, num_layers=2, num_heads=4) match the paper's Table VI exactly. The four training stages correspond to the four task names in the README: G1-Steady-Tray-Pre-Locomotion, G1-Steady-Tray, G1-Steady-Object, and G1-Steady-Object-Distillation, with stages two through four each resuming from the previous checkpoint; on the deployment side, deploy/deploy_mujoco provides MuJoCo sim2sim validation.

Limitations

The authors' first stated limitation is the object encoding: the current encoder handles a single object and does not capture fine-grained geometry or physical properties, and they suggest wiring in a foundation pose-estimation model or visually-grounded RL to improve the representation. The second is a hardware-induced perception limit: the head camera's narrow, fixed field of view makes alternative carrying postures difficult, and avoiding collisions in clutter is often exactly what calls for an alternative carrying posture. The third is broader: pure sim-to-real RL leans on heavy reward shaping and careful simulator tuning, a path the authors argue becomes increasingly impractical for contact-rich loco-manipulation.

Two further gaps are visible from the reader's side. First, simulated evaluation uses randomly sized cylinders, and while the hardware payloads are varied they are all roughly axisymmetric or box-like stable geometries; behavior with a high center of gravity, a slippery surface, or a deformable payload (a stack of paper, a bowl of soup) is not quantified. Success is also binary — "never toppled, never left the tray" — and does not distinguish "wobbled but recovered" from "did not move," even though that distinction is precisely what determines spilled liquid in a meal-delivery setting. Second, the comparison set contains no model-based method. Tray stabilization has model-based coordinated-control precedents in wheeled and aerial manipulation, and comparing only against learned baselines leaves the sample-efficiency and performance-ceiling gap between residual RL and model-predictive control unmeasured.

Conclusion and Outlook

SteadyTray's contribution has three layers. It formalizes a task nobody had attacked directly before — balancing an unattached object on a tray during bipedal locomotion. It offers a residual teacher-student framework that decouples walking from stabilizing structurally, and backs the decoupling with ablations showing the gain comes from the construction rather than from one adapter flavor. And it documents the training details that make sim-to-real work here — observation delay, control delay, delayed-timeout termination, and the upper-body curriculum — of which observation delay is shown to improve in-simulation stability as well. The authors' stated extension is broader loco-manipulation: contact-rich tasks such as opening doors or pushing carts while walking, and incorporating visual or tactile feedback for whole-body adaptation without disturbing the pretrained gait.

For practitioners the reusable assets are concrete. A frozen base policy plus a zero-initialized residual adapter is a low-risk starting move for any project that already has a working gait and wants to add an upper-body skill. Distilling the encoder while freezing the adapter compresses privileged imitation down to a 64-dimensional latent. Delayed-timeout termination is an easily overlooked detail that matters for safety on hardware, because it teaches the policy to keep walking stably after a failure instead of only before one. With four-stage training scripts and checkpoints released, the recipe can be reproduced and modified directly.

Two Lines Worth Keeping

"A tray is not a container for the payload; it is a non-rigid coupling that hands every footstrike, unfiltered, to a cup that nothing is holding down."

"The end-to-end policy's total reward looks fine. What looks bad is the object-upright curve that total reward was hiding."