LEGS: Fine-Tuning Teleop-Free VLAs for Humanoid Loco-manipulation in an Embodied Gaussian Splatting World

Paper Deep Dive

Hojune Kim et al. · Stanford University · arXiv 2606.01458 · project page · code coming soon

LEGS produces humanoid loco-manipulation demonstrations without teleoperation by compositing a physically simulated mesh foreground into a photorealistic 3DGS background and executing scene-level motion primitives. Across three tasks and three VLA backbones on Unitree G1, purely synthetic data matches or beats 50 human teleoperation demonstrations and reduces the cost of adapting to a new appearance condition by roughly 15 times.

Background and Motivation

The humanoid form factor promises compatibility with human spaces: doorways, counters, stair edges, shelves, and tools already have human dimensions. That compatibility becomes expensive as soon as a policy must be adapted. Teleoperation couples the number of demonstrations to operator labor and robot uptime, while extended sessions introduce fatigue and make throughput decline over time.

Human-demo retargeting removes the robot from data collection but still requires egocentric capture for each scene. The costs recur whenever appearance changes. Synthetic simulation can remove the operator, yet conventional mesh rendering diverges from real camera imagery and has so far failed to transfer reliably to humanoid loco-manipulation.

The authors argue that SAM3D-style mesh reconstruction is not enough. Mesh pipelines inherit material-shading mismatch, simplified lighting, coarse geometry, and aliasing. Those errors concentrate in the high-frequency egocentric image regions that matter for contact, occlusion, and object boundaries. The relevant question is therefore not merely whether a scene can be simulated, but whether the renderer is supervised by real photographs of the deployment scene.

LEGS combines two different kinds of fidelity. MuJoCo supplies physical fidelity on mesh geometry. 3D Gaussian Splatting supplies appearance fidelity because its optimization minimizes rendering error against real photographs. The robot, manipulable objects, and props remain meshes, while walls, floors, tables, and other static scene content come from the reconstructed 3DGS field.

A second design goal separates motion from appearance. Once a command stream is recorded, the same motion can be rendered under a different background, object mesh, and language prompt. This makes the later augmentation experiment possible: a new visual condition does not require a new operator session.

The LEGS Framework

LEGS is organized by two decouplings. Physics is separated from rendering: a MuJoCo backend resolves dynamics at 500 Hz while a rendering frontend produces observations only when needed. A static 3DGS background is also separated from a dynamic mesh foreground: each frame is composited with a depth test, and the command stream refers to mesh-anchored robot motion rather than pixels.

Visual frontend

A scene begins with a 1 to 2 minute handheld video. COLMAP provides camera poses, and 3DGS optimization then reconstructs the background. Because the objective is photometric reconstruction of real images, novel views retain the scene's lighting, materials, and texture. Foreground meshes come from the Unitree URDF and SAM3D reconstructions of objects from a single handheld photograph.

The composite spans three color spaces: the scanning device's ISP for 3DGS, SAM3D's regressed albedo for meshes, and the deployment Intel RealSense D435 ISP. LEGS disables auto-exposure and auto-white-balance on both cameras so the mapping remains stationary, then applies two deterministic stages.

The first equation summarizes the composite as a color-corrected depth composite:

$$c_d \approx M\,[R(c_m)\sqcup c_b],$$

where $c_m$ is a mesh vertex color, $c_b$ is a background 3DGS pixel, $c_d$ is a deployment-camera pixel, $R(\cdot)$ is a per-mesh diagonal scale in linear RGB, $\sqcup$ is depth-test compositing, and $M\in\mathbb{R}^{3\times 3}$ is a global color-correction matrix.

Each channel of $R$ is obtained from a rendered mesh statistic matched to the source-photo statistic inside the SAM2 mask:

$$s^{(c)}=\mathrm{clip}\!\left(\frac{\mu_T^{(c)}}{\mu_S^{(c)}},\;0.5,\;3.0\right).$$

Here $\mu_S^{(c)}$ is the per-channel mean of linear RGB over rendered pixels with $\alpha>200$, and $\mu_T^{(c)}$ is the corresponding mean over iPhone pixels in the object mask. The 0.5 to 3.0 clip prevents one bad albedo estimate from destroying the mesh colors. The scale changes color only, not geometry.

The second stage aligns the composited iPhone space with RealSense space using a 24-patch ColorChecker. Let $P,Q\in\mathbb{R}^{24\times 3}$ contain linearized RealSense and iPhone patch colors:

$$M^\star=\arg\min_M\|P-QM\|_F^2=(Q^\top Q)^{-1}Q^\top P.$$

At render time, each frame is linearized, multiplied by $M^\star$, clipped to $[0,1]$, and re-encoded to sRGB. Applying $M^\star$ at the composite level keeps the mesh and 3DGS layers spatially consistent and avoids boundary artifacts. The result is deterministic rather than a learned compensator.

Physics backend and controller interface

The physics backend runs MuJoCo at 500 Hz. Object collision geometry is decomposed with CoACD, the static background has a collision mesh aligned to the 3DGS field, and the robot uses the Unitree G1 URDF. SONIC is the low-level whole-body controller, and the same binary is used in simulation and deployment.

SONIC consumes an 18-D command shared by both sides. Each arm contributes a 6-D wrist $\mathrm{SE}(3)$ pose and one continuous grip scalar, for 14 upper-body dimensions. Four more dimensions command the base:

$$u_t=\big[q_t^L,q_t^R,g_t^L,g_t^R,v_{x,t},v_{y,t},\omega_{z,t},h_t\big]\in\mathbb{R}^{18}.$$

Because the pretrained VLA backbones all fine-tune into this same interface, the experiment isolates data source within each backbone. The recipe is fixed within a backbone: full fine-tuning for $\pi_{0.5}$ and action-head-only fine-tuning for $\psi_0$ and GR00T N1.6.

Procedural episode generation

Each pick-and-place task decomposes into high-level Walk, Pick, and Place motions. Those motions are composed from primitives such as Align, Approach, Reach, Close, Adapt, Stand/Squat, VerifyHold, and VerifyPlace. Primitive arguments are scene-level target pose, object identity, and arm choice rather than joint targets, so one task plan unfolds into different valid trajectories under randomized initial states.

Pick adjusts pelvis height with Stand/Squat, interpolates the end effector with Reach, and then uses Adapt to correct the gripper from the difference between commanded and actual wrist pose. Adapt is a privileged simulator signal used only during data generation. On closure failure, the arm retracts and retries Reach and Adapt.

Every episode must pass VerifyHold or VerifyPlace before being stored. Randomization includes object position within $\pm5$ cm, object yaw within $\pm180^\circ$, robot heading within $\pm10^\circ$, and, for locomotion tasks, robot base position within $\pm10$ cm. Failed episodes are discarded and do not consume the 200-demonstration budget.

The stored stream is the 18-D command sequence, not pixels tied to a renderer. Consequently, a background, object mesh, or prompt can be changed and the same motion re-rendered. This is what makes LEGS-aug a controlled appearance augmentation rather than a new data collection.

The following diagram follows the pipeline in Figure 2:

flowchart LR
  A[Handheld scene video] --> B[COLMAP poses]
  A --> C[3DGS optimization]
  D[Single object photo] --> E[SAM3D meshes]
  E --> F[CoACD collision]
  B --> G[LEGS simulator]
  C --> G
  F --> G
  G --> H[MuJoCo physics at 500 Hz]
  G --> I[SONIC WBC]
  J[Scene-level task plan] --> K[Walk / Pick / Place primitives]
  K --> H
  H --> L[Verified 18-D command stream]
  L --> M[Mesh foreground depth composite]
  C --> M
  M --> N[Two-stage color calibration]
  N --> O[Synthetic RGB observations]
  O --> P[Fine-tune VLA]
  P --> Q[Unitree G1 deployment]
  L --> R[Swap background objects prompt]
  R --> M

Experimental Results

The robot is Unitree G1 with a head-mounted Intel RealSense D435 streaming 30 Hz RGB. Three tasks add whole-body difficulty progressively: Task 1 is manipulation-only, Task 2 adds walking to a table, and Task 3 requires picking an orange, turning right, walking to a low table, squatting, and placing it. The evaluation performs 10 real-robot trials for each data condition, backbone, and task, for 1,110 trials overall.

The table aggregates Table 2 across the three backbones:

Data condition	Task 1 mean	Task 2 mean	Task 3 mean	Main finding
Teleop (50)	4.7/10	2.7/10	0/10	0/10 across all backbones on Task 3
SAM3D (200)	4.7/10	4.0/10	1.3/10	below LEGS (200) on every experiment
LEGS (50)	7.3/10	6.3/10	3.0/10	matches or beats teleop at equal budget
LEGS (200)	8.3/10	7.3/10	4.3/10	best or tied best in all nine cells

Task 3 makes the effect concrete:

Task 3	$\psi_0$	$\pi_{0.5}$	GR00T N1.6	Interpretation
Teleop (50)	0/10	0/10	0/10	teleoperation collapses on the long horizon
SAM3D (200)	1/10	0/10	3/10	mesh rendering remains a large gap
LEGS (50)	3/10	1/10	5/10	not merely a 200-demo budget effect
LEGS (200)	5/10	2/10	6/10	synthetic-only data succeeds end to end

The equal-budget condition is important. LEGS (50) uses the same number of demonstrations as Teleop (50) yet matches or beats it in every experiment. The further improvement from LEGS (50) to LEGS (200) is consistent with ordinary imitation-learning data scaling, but it cannot explain the advantage over SAM3D (200). Stage-wise curves place the separation mainly at Pick and Place, where the egocentric camera is close to the object and rendering fidelity dominates.

Rendering-backend ablation

SAM3D (200) holds the LEGS pipeline fixed except for two variables: it removes the 3DGS background and disables color calibration. Procedural generation, MuJoCo, CoACD, SONIC, action space, randomization, and budget all remain identical. SAM3D averages 33% end-task success versus 67% for LEGS, a 33-point drop that approximately halves success.

The degradation is not uniform across stages. The four data conditions are comparable at Walk, but SAM3D separates from LEGS during Pick and Place. That pattern is more informative than a generic image-similarity claim: the visual gap becomes an action error exactly when the policy must resolve near-field geometry, contact cues, and occlusion.

Appearance re-rendering

The appearance suite has four conditions: default; scene, which swaps only to a blue table; objects, which swaps orange and plate for apple and box; and objects + scene, which swaps both. The corresponding prompt becomes “put the apple in the box” or its Task 3 variant. LEGS-aug re-renders the recorded motion under the new assets and relabels the prompt. Teleop, SAM3D (50), and LEGS (50) are trained only on default data.

All three 50-episode un-augmented baselines collapse to 0 or 1 out of 10 under objects or objects + scene. LEGS-aug (200) reaches 100%, 80%, and 40% on Tasks 1 through 3 under objects + scene, versus 60%, 50%, and 20% for SAM3D-aug. Re-rendering solves distribution shift; photorealism preserves decision quality after that shift is solved.

A scale ablation strengthens the argument. LEGS-aug (50) reaches 50%, 40%, and 30% on objects + scene, while LEGS (200) trained only on the default appearance reaches 0%, 0%, and 10%. Augmentation beats a fourfold default-data budget because the policy lacks observations of the new objects and prompt, not trajectory diversity.

Cost reinforces the practical result. Fifty Task 3 teleoperation demonstrations take 1.5 operator-hours. LEGS takes about 0.5 GPU-hours initially on one RTX 4090 and about 0.1 GPU-hours for each additional appearance condition, roughly a 15 times lower marginal cost without operator involvement.

Limitations

The authors state that disabling auto-exposure and auto-white-balance makes the policy sensitive to lighting changes outside the scanned distribution. Adapting to a new scene still requires a 1 to 2 minute handheld video, although not a new teleoperation session. 3DGS degrades on highly reflective or transparent surfaces and assumes a static background, so scenes with moving people or other dynamic content require dynamic 3DGS methods not implemented here.

The procedural generator optimizes feasible motion rather than dynamic efficiency, unlike trajectory optimization or MPC. The appearance suite varies only small graspable objects of similar size and geometry. Finally, the evaluation uses one robot platform, one camera, and one whole-body controller, so cross-hardware generalization is untested.

Three engineering boundaries follow from the paper and project page. First, discarding failed episodes makes the training set clean but hides the generator's failure distribution near difficult targets. Second, Adapt relies on privileged simulator wrist-pose error; the method is teleop-free but not privilege-free. Third, the project page marks the code release as coming soon, so COLMAP settings, SAM3D assets, randomization, and fine-tuning configurations cannot yet be independently reproduced.

Conclusion

LEGS does not introduce a new VLA. Its contribution is a data pipeline that separates physical simulation, photorealistic appearance, and procedural motion generation, then recombines them into labeled humanoid demonstrations. The result is a rare controlled evidence chain: pure synthetic data beats an equal-budget teleoperation baseline, outperforms an otherwise identical mesh-rendering baseline, and adapts to appearance shifts through re-rendering.

The most useful takeaway for practitioners is that transfer is decided by both action generation and observation distribution. Scaling default-only data helps until the test distribution changes the objects or prompt. Then matching the visual and linguistic distribution through re-rendering matters more than four times more in-domain trajectories.

“Teleoperation records a human controlling a robot; LEGS records motion that can be seen again in a different world.”