VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes

Paper: VLK: Learning Humanoid Loco-Manipulation from Synthetic Interactions in Reconstructed Scenes
Authors: Yen-Jen Wang, Jiaman Li, Sirui Chen, Takara E. Truong, Pei Xu, Pieter Abbeel, Rocky Duan, Koushil Sreenath, Angjoo Kanazawa, Carmelo Sferrarra, Carmelo Sferrazza, Guanya Shi, Karen Liu (UC Berkeley, Stanford University, Carnegie Mellon University)
Released: arXiv 2606.30645 (June 29, 2026, cs.RO)
Links: arXiv · project page
Code: not released yet. The project page reserves a Code button (placeholder link), and the vision-language-kinematics GitHub organization currently hosts only the project-site repository.

One-sentence summary: VLK removes the data bottleneck of perception-based humanoid loco-manipulation by synthesizing 48,000 paired vision-language-kinematics trajectories inside 3DGS-reconstructed real scenes, training a pi0.5-initialized policy to predict one-second whole-body kinematic chunks that a contact-aware tracker executes on a physical Unitree G1.

Figure 1: Paper teaser. Synthetic interactions synthesized in reconstructed scenes drive real-world, perception-based humanoid loco-manipulation on the Unitree G1.

1. Background and Motivation

A humanoid operating in human-centered environments must close a long loop: perceive from its own egocentric viewpoint, ground a language instruction onto task-relevant objects, navigate across cluttered space, and finally interact through coordinated whole-body motion. The learning object for such a system is a policy that maps high-dimensional visual input plus language onto robot-compatible whole-body behavior, and the obstacle the paper identifies is not the model but the data. Supervision for this mapping must come as synchronized triples: egocentric observations, task instructions, and whole-body kinematic trajectories that a real humanoid can execute.

The authors systematically rule out the existing sources of such supervision. Real-world teleoperation produces high-quality paired demonstrations, but full-body teleoperation is expensive and does not scale across scenes, objects, and interaction types. Human motion-capture datasets provide rich whole-body references that can be retargeted to humanoid morphologies, but they come with no corresponding robot egocentric observations. Egocentric video datasets supply abundant visual experience but no robot-compatible kinematics or actions. No single existing source provides the complete vision-language-kinematics tuple, which is precisely why the paper frames the contribution as a data-generation problem rather than a new architecture.

The paper's diagnosis is that two partial solutions already exist and only need to be connected. On the Real2Sim side, 3D Gaussian Splatting can reconstruct indoor environments at metric scale with photorealistic detail, supplying realistic world assets cheaply. On the Sim2Real side, whole-body motion tracking trained in simulation can turn kinematic reference trajectories into executable policies on hardware. The missing link, as the paper puts it, is a system that synthesizes paired visual observations inside 3DGS worlds together with robot kinematic behaviors suitable for whole-body tracking. Once that system exists, the three modalities decouple: language commands can be prescribed, privileged scene information (object poses, collision geometry, walkable regions) can be exploited to make behavior generation tractable, and egocentric observations can be rendered after the fact by hindsight rendering.

This decoupling is what makes scale possible. The pipeline produces 48,000 paired trajectories within 600 GPU-hours and without any human demonstration effort. With that dataset, the authors train a Vision-Language-Kinematics (VLK) policy that takes an egocentric image, a task instruction, and the current robot state and predicts a short-horizon whole-body kinematic trajectory plus binary wrist-object contact labels. A contact-aware whole-body tracker then converts these predictions into joint-level actions on the physical Unitree G1. The evaluation covers navigation and single-object transport, exercising the complete perception-to-action loop: visual grounding, locomotion, pickup, carrying, and placement.

Positioning matters here. Relative to humanoid VLA systems such as WholeBodyVLA or Psi-Zero, which train on egocentric video plus real teleoperation data, VLK generates its supervision instead of collecting it, and it predicts explicit future kinematic trajectories for tracker-based execution rather than low-level joint commands. Relative to 3DGS-for-robotics work, the novelty is not the reconstruction itself but pairing reconstructed scenes with humanoid motion synthesis so that every trajectory comes with synchronized vision and language. Relative to humanoid motion synthesis, the paper adapts human-object interaction synthesis from SMPL-based human motion to the Unitree G1 kinematic representation, which is what makes the synthesized behaviors executable by a real robot tracker.

2. Preliminaries

Three building blocks carry the system. First, 3D Gaussian Splatting (3DGS) represents a scene as a cloud of anisotropic Gaussians and supports photorealistic novel-view rendering; because the scenes are reconstructed from RGB plus LiDAR scans, they retain metric scale, which is essential for motion synthesis and for a camera whose extrinsics must match the physical robot. Second, whole-body tracking controllers trained in simulation learn to follow kinematic reference trajectories with joint-level PD targets; the paper builds on the contact-aware SceneBot tracker, which is blind to vision and language and only consumes reference targets plus proprioception. Third, the policy backbone is pi0.5, a pretrained vision-language-action model that the authors fine-tune with a flow-matching objective adapted from action-chunking VLAs.

The flow-matching formulation deserves attention because it defines the training objective. Instead of predicting a velocity field, the policy is trained with an x0-prediction objective: a ground-truth future trajectory is interpolated with Gaussian noise using a coefficient alpha, and the network is supervised to reconstruct the clean trajectory from the noisy one, conditioned on all inputs. This keeps the sampling procedure simple at deployment: the external GPU server runs the flow-matching sampling and returns a full one-second kinematic chunk per replan.

Finally, the kinematic interface between perception and control is deliberately low-dimensional and robot-native. Rather than predicting raw joint torques or even raw joint angles directly from pixels, the VLK policy predicts a canonical kinematic state that includes heading-normalized root displacements and sin/cos-encoded joint angles. This choice keeps the high-level policy invariant to global heading, avoids angle wrap-around discontinuities, and hands the physically hard part — balance, contact, torque-level execution — to the tracker.

Related Work and Positioning

The paper situates itself against three research threads. The first is vision-language and perception-based policy learning for humanoids. VLA policies that map images and instructions to actions have become a mainstream paradigm, and a wave of recent work pushes this onto humanoid platforms where perception, language, and whole-body control must be tied together. WholeBodyVLA and Psi-Zero train humanoid loco-manipulation policies from egocentric video plus real teleoperation data; VLK is deliberately complementary: instead of consuming existing robot demonstrations or human videos as the primary supervision, it manufactures paired vision-language-kinematics supervision inside reconstructed scenes, and the resulting policy predicts explicit future G1 kinematics for tracker-based execution rather than emitting low-level actions directly.

The second thread is synthetic data and reconstructed scenes for robot learning. Digital twins and 3DGS-reconstructed environments are increasingly used to build realistic synthetic worlds for manipulation, locomotion, demonstration synthesis, and evaluation. VLK shares the scene-representation choice but targets a harder data requirement: humanoid loco-manipulation supervision must provide not only realistic egocentric observations but also valid whole-body interaction behavior inside those scenes, which is why the paper couples reconstructed scenes with humanoid motion synthesis rather than using scenes for rendering alone.

The third thread is humanoid motion synthesis and whole-body tracking. Retargeting pipelines map human motion onto humanoid embodiments, and whole-body tracking controllers execute such references on hardware; text-conditioned motion generation has also been connected to whole-body control. For loco-manipulation, however, synthetic behavior must include object interaction, so VLK adapts human-object interaction synthesis methods from SMPL-based human motion to the Unitree G1 kinematic representation, and uses the synthesized G1-object trajectories as the supervision paired with egocentric renderings and language instructions.

3. Method

Figure 2: Method overview. Top: reconstruct 3D scenes, generate task waypoints, synthesize G1 motions, render egocentric observations to produce paired VLK supervision. Bottom: the paired data train the VLK policy; at deployment a whole-body tracker converts predicted trajectories into robot actions.

flowchart TD
    A["iPhone 14 Pro Polycam scan
RGB + LiDAR"] --> B["3DGS scene reconstruction
metric scale"]
    B --> C["Annotation in viser
semantic 3D boxes + walkable regions"]
    C --> D["Waypoint sampling + visibility check
template language generation"]
    D --> E1["Navigation diffusion model
trained on BONES-SEED G1 motions"]
    D --> E2["Interaction diffusion model
OMOMO retargeted to G1 via OmniRetarget"]
    E1 --> F["Post-processing
foot-contact IK + wrist-pose matching"]
    E2 --> F
    F --> G["Isaac Sim egocentric rendering
virtual ZED 2i + domain randomization"]
    G --> H["48k paired VLK trajectories
image + language + kinematics"]
    H --> I["VLK policy fine-tuned from pi0.5
flow-matching x0 prediction"]
    I --> J["Real-time chunked replanning
~1.8 Hz, 10-frame overlap"]
    J --> K["SceneBot contact-aware tracker
50 Hz joint PD targets"]
    K --> L["Physical Unitree G1"]

The complete real-to-sim-to-real pipeline, from Polycam scan to physical execution.

3.1 Scene Reconstruction and Annotation

The physical basis of the dataset is two real environments — a lab-style scene and an apartment-style scene — each re-scanned under four different furniture layouts, giving eight reconstructed scenes in total. Scans are captured with the Polycam app on an iPhone 14 Pro, combining RGB imagery with LiDAR-based depth so that the resulting 3DGS reconstructions are metric-scale. For each layout the authors optimize a 3DGS representation that preserves both visual appearance and spatial layout for egocentric rendering.

Because a raw 3DGS model provides neither meshes nor semantics, the pipeline extracts a scene point cloud and annotates it with a custom interactive tool built on viser. Annotators place oriented semantic 3D bounding boxes around task-relevant objects (chair, table, box) and mark walkable regions as polygons on the floor plane. These two annotation types define everything the motion synthesizer is allowed to use: target objects for navigation, feasible initial robot poses, and sparse waypoints. The paper also describes how task instructions are generated from templates instantiated with the semantic labels — "walk toward the [object]" for navigation and "pick up the box from the floor" or "put down the box on the [surface]" for interaction — so language labels are grounded in scene annotations without any manual labeling.

Waypoint generation is task-specific and includes one detail that matters for visual grounding: for walk-toward-object tasks, the target object must be visible from the sampled initial egocentric viewpoint. The system checks whether the target lies inside the camera field of view and is not occluded by major scene structures, and resamples the initial pose otherwise. This guarantees that every training sample contains visual evidence of its goal. For turn-around tasks, all waypoints collapse onto the initial position, producing in-place turning. For box pick and place, wrist contact frames and desired relative wrist poses are designed according to the support-surface height, which keeps contact timing realistic and avoids abrupt vertical motion.

3.2 Interaction Synthesis in Reconstructed Scenes

Motion synthesis operates directly in the Unitree G1 kinematic representation. A motion sequence of length T with J joints is represented as

$$\mathbf{x}^{\mathrm{g1}}=[\mathbf{p},\mathbf{R},\sin(\mathbf{q}),\cos(\mathbf{q})],$$

where p denotes global joint positions, R denotes global joint rotations in the 6D rotation representation, and q denotes G1 joint angles encoded as (sin(q), cos(q)) to avoid wrap-around discontinuities. For object-interaction sequences the object trajectory is represented as x_obj = [o, R obj], with the global object position and the object rotation relative to the canonical pose of the input object geometry; the full interaction trajectory is the concatenation x = [x g1, x obj].

Two diffusion models generate behavior. The navigation model is trained on G1 motions from BONES-SEED and generates object-directed walking from task instructions, initial states, and sparse scene-grounded waypoints. The interaction model is trained on OMOMO human-object interaction sequences retargeted from SMPL to G1 with OmniRetarget. Both follow the conditional DDPM formulation of CHOIS with a Transformer denoiser, but the paper adapts the representation from SMPL to G1 and adds a G1-specific differentiable forward-kinematics layer so that geometric losses can be applied to global body and end-effector positions — this is what makes wrist placement accurate enough for box interaction.

Conditioning is what turns a generic motion generator into a task-directed one. The interaction model consumes the language instruction encoded as CLIP text features, initial humanoid and object states, sparse scene-derived waypoints, object geometry encoded with a Basis Point Set representation followed by an MLP, desired relative wrist poses defined in the object coordinate frame, and wrist-object contact labels. For box lifting, for example, the two wrists are specified to approach opposite sides of the box with downward-facing palms. These conditions constrain the generated whole-body motion so that wrists reach task-relevant contact regions while the object follows a consistent trajectory.

Generated motions still need cleanup, since diffusion-based motion synthesis is prone to foot sliding and unrealistic hand-object contact. The pipeline applies lower-body post-processing that uses predicted foot-contact labels to identify contact phases and constrains each foot to remain fixed during stance by solving inverse kinematics. For hand-object contact, it adapts the optimization approach of EgoAllo with a wrist-pose matching term that drives each wrist toward the input wrist pose expressed in the object's local frame, producing more realistic two-hand grasping. Sequences with scene penetrations are filtered out before rendering.

3.3 Egocentric View Rendering and Domain Randomization

Egocentric observations are rendered by replaying synthesized G1 trajectories in Isaac Sim with the reconstructed 3DGS scene loaded. The robot carries a virtual ZED 2i camera mounted on the head whose extrinsics are calibrated to match the physical camera setup on the real G1, so the rendered viewpoint aligns with the deployment viewpoint. Domain randomization is applied during rendering to bridge the visual sim-to-real gap: camera extrinsics and focal length are perturbed, dome-light intensity and yaw are randomized, and image-space augmentations (brightness, contrast, saturation, hue, Gaussian noise, Gaussian blur) are applied. The ranges are summarized below.

Randomization	Range
Camera translation jitter	±2 cm
Camera rotation jitter	±3°
Camera focal perturbation	±2%
Dome-light intensity	[400, 1500]
Dome-light yaw rotation	[−π, π]
Brightness	[0.8, 1.2]
Contrast	[0.8, 1.3]
Saturation	[0.7, 1.4]
Hue	[−0.05, 0.05]
Gaussian noise std.	[0, 0.05]
Gaussian blur sigma	[0, 1.0]

Table 1: Domain randomization parameters used for egocentric rendering (Appendix A.5).

Figure 3: Examples of generated VLK supervision. Each sequence pairs an egocentric RGB observation and a language instruction with the corresponding G1 whole-body kinematic trajectory.

The throughput numbers are part of the contribution's claim. Each environment contains 4 layouts and 12 data-generation modes — the six evaluated modes (Walk To, Turn Around, Pick Floor, Put Floor, Pick Surface, Put Surface) plus auxiliary modes for random walking, turning, carrying, and carry-turning that improve state coverage. The pipeline synthesizes 1000 trajectories per layout and mode, yielding 48,000 trajectories per environment. On a single NVIDIA L40S GPU, synthesizing 1000 trajectories for one mode in one layout takes about 4 hours, and rendering the corresponding egocentric observations takes about 8.3 hours; generation parallelizes across layouts and modes, and the whole dataset costs 600 GPU-hours with no human intervention after scene annotation.

3.4 The VLK Policy

The per-frame kinematic state at frame τ is

$$\mathbf{x}_{\tau}=[\Delta x_{\tau},\Delta y_{\tau},\Delta\psi_{\tau},h_{\tau},\mathbf{R}^{\mathrm{root}}_{\tau},\sin(\mathbf{q}_{\tau}),\cos(\mathbf{q}_{\tau}),\mathbf{c}_{\tau}],$$

where (Δx, Δy) is the heading-normalized root planar displacement, Δψ the yaw change, h the root height, R root the heading-normalized 6D root orientation, q the G1 joint angles, and c the contact vector

$$\mathbf{c}_{\tau}=[c_{\tau}^{L},c_{\tau}^{R}],\quad c_{\tau}^{L},c_{\tau}^{R}\in\{0,1\},$$

with binary left/right wrist-object contact labels. At timestep t the policy takes the current egocentric RGB observation o_t, the task instruction ℓ, and the current state x_t, and predicts an H-frame future trajectory

$$\hat{\mathbf{x}}_{t+1:t+H}=\pi_{\theta}(o_{t},\ell,\mathbf{x}_{t}),$$

with H = 30, i.e. one second at 30 Hz. The instruction is fixed for a task execution, and during deployment the contact label inside x_t is auto-regressively predicted rather than observed. The policy is initialized from pretrained pi0.5 and fine-tuned end to end with the action space adapted to the G1 kinematic representation.

Training uses an x0-prediction objective in a flow-matching formulation. Given the ground-truth future trajectory, Gaussian noise ε, and interpolation coefficient α, the noisy trajectory is

$$\mathbf{x}^{\alpha}_{t+1:t+H}=\alpha\bm{\epsilon}+(1-\alpha)\mathbf{x}_{t+1:t+H},$$

and the policy predicts the clean trajectory conditioned on everything:

$$\hat{\mathbf{x}}_{t+1:t+H}=\pi_{\theta}(\mathbf{x}^{\alpha}_{t+1:t+H},\alpha,o_{t},\ell,\mathbf{x}_{t}),$$

with the primary reconstruction loss

$$\mathcal{L}_{\mathrm{traj}}=\left\|\hat{\mathbf{x}}_{t+1:t+H}-\mathbf{x}_{t+1:t+H}\right\|_{2}^{2}.$$

Four auxiliary losses improve physical consistency. Foot-floor contact labels are supervised with a focal loss because contact events are sparse. An accumulated root-trajectory loss integrates the predicted relative root displacements and penalizes drift:

$$\mathcal{L}_{\mathrm{acc\text{-}root}}=\frac{1}{H}\sum_{k=1}^{H}\left\|\hat{\mathbf{p}}^{\mathrm{root}}_{t+k}-\mathbf{p}^{\mathrm{root}}_{t+k}\right\|_{2}^{2}.$$

A forward-kinematics loss supervises ankle and wrist positions obtained by applying FK to the predicted joint angles:

$$\mathcal{L}_{\mathrm{fk}}^{\mathrm{ankle}}=\frac{1}{H|\mathcal{A}|}\sum_{k=1}^{H}\sum_{j\in\mathcal{A}}\left\|\hat{\mathbf{p}}^{j}_{t+k}-\mathbf{p}^{j}_{t+k}\right\|_{2}^{2},$$

with the analogous wrist term over the wrist end-effectors. Finally, a foot-skating regularizer penalizes horizontal foot velocity whenever a foot is predicted to be in contact:

$$\mathcal{L}_{\mathrm{foot}}=\frac{1}{H|\mathcal{F}|}\sum_{k=1}^{H}\sum_{j\in\mathcal{F}}\hat{m}^{j}_{t+k}\left\|\mathbf{v}^{j,xy}_{t+k}\right\|_{2}^{2}.$$

The complete objective, with the paper's loss weights, is

$$\mathcal{L}_{\mathrm{total}}=1.0\,\mathcal{L}_{\mathrm{traj}}+0.5\,\mathcal{L}_{\mathrm{foot\text{-}contact}}+0.2\,\mathcal{L}_{\mathrm{acc\text{-}root}}+1.0\,\mathcal{L}_{\mathrm{fk}}^{\mathrm{ankle}}+1.0\,\mathcal{L}_{\mathrm{fk}}^{\mathrm{wrist}}+0.05\,\mathcal{L}_{\mathrm{foot}}.$$

3.5 Whole-Body Tracking and Deployment

The predicted kinematics are executed by a contact-aware whole-body tracker based on SceneBot. The tracker is deliberately blind to egocentric observations and language: it receives the converted reference — lower-body joint targets, head and wrist target 6D poses, root target pose, and wrist-object contact labels — together with the current low-level robot state s_t, and outputs joint-level PD targets

$$\mathbf{u}_{t}=\pi_{\mathrm{track}}(\bar{\mathbf{x}}_{t},\mathbf{s}_{t}).$$

When wrist contact is active, the tracker engages contact-aware wrist behavior to maintain contact with the manipulated object, which is what stabilizes bimanual box transport. This division of labor — VLK handles perception-conditioned replanning, the tracker handles balance and contact — is the architectural reason the system can be trained purely on kinematic supervision.

Deployment is engineered as a real-time system. The VLK model runs through a Triton-optimized backend on an RTX 5090 with stable inference latency of 31 ms. To reduce discontinuities between predicted chunks, the system adapts the client-side real-time chunking idea from Psi-Zero: adjacent action chunks overlap by 10 frames. A multi-process runtime on a tethered laptop runs a state estimator, the whole-body tracker at 50 Hz on an RTX 5000 Ada (4.3 ms average per tick, no missed deadlines), and a VLK inference client that streams observations to the external GPU server over a websocket and merges returned chunks into the tracker's reference stream. Each replan takes about 63 ms end to end against a roughly 555 ms replan period (~1.8 Hz), leaving about 8.8× headroom against backlog formation.

Stage	Latency (ms)	Share
Image fetch from camera buffer	5.4	9%
Observation packing (state norm. + image enc.)	7.5	12%
Server roundtrip + GPU flow-matching sampling	37.0	59%
Output denormalization	4.2	7%
World-frame transform + forward kinematics	7.2	11%
Reference merge into tracker stream	3.4	5%
Total	63.0	100%

Table 2: End-to-end VLK replan latency breakdown (Appendix B.4).

One practical failure mode gets an explicit solution: motion blur during fast bending degrades prediction quality. During training, a subset of RGB observations is augmented with synthetic motion blur, I′ = B_σ(I) with σ drawn from a small range that preserves scene semantics. During deployment, the system keeps a 0.3 s image buffer and selects the sharpest frame by the variance of the Laplacian,

$$S(I)=\mathrm{Var}\!\left(\nabla^{2}I\right),\qquad I_{t}^{\ast}=\arg\max_{I_{t}^{(i)}}S\!\left(I_{t}^{(i)}\right),$$

so the policy always consumes the least blurred frame available in the recent window.

4. Experiments

The evaluation is built around four questions: whether the pipeline can generate diverse paired VLK data efficiently, whether a policy trained on synthesized data transfers to the physical humanoid, how data volume affects performance, and whether visual domain randomization bridges the visual sim-to-real gap. Hardware is a Unitree G1 with a custom 3D-printed head bracket for a wider field of view, using the left ZED 2i RGB image at 672×376 resized to the 224×224 VLK input. Real-world testing happens in the same two physical environments used for data generation, with furniture layouts manually varied across trials; six task modes are evaluated with 20 trials each, for 120 real-world trials. Simulation evaluation renders egocentric observations from the 3DGS scenes in Isaac Sim and executes the tracker in MuJoCo, with a held-out 10% validation set providing initial states for 1000 rollouts per mode.

Figure 4: Full-system evaluation results across simulation and real-world settings (results figure accompanying Table 1 in the paper).

Setting	Scene	Walk To	Turn Around	Pick (Floor)	Put (Floor)	Pick (Surface)	Put (Surface)
Real-World	Lab	20/20	20/20	16/20	20/20	11/20	8/20
Real-World	Apartment	19/20	18/20	18/20	20/20	13/20	15/20
Simulation	Lab	994/1000	843/1000	731/1000	991/1000	458/1000	569/1000
Simulation	Apartment	948/1000	813/1000	749/1000	987/1000	521/1000	722/1000

Table 3: Full-system evaluation in MuJoCo simulation (with Isaac Sim rendering) and real-world deployment (successes / trials per mode).

The headline result is that synthesized data supports genuine sim-to-real transfer for perception-based loco-manipulation. Object-directed navigation is essentially solved in this setup — Walk To and Put (Floor) hit 100% real-world success in the lab, and all navigation-style modes stay at or above 18/20 in both scenes. Floor-level box manipulation is strong as well (Pick Floor 16/20 lab, 18/20 apartment). Surface-level interaction is the clear weak point: Pick (Surface) manages 11/20 and 13/20 in the real world, and Put (Surface) drops to 8/20 in the lab. The paper attributes this to coverage: the retargeted OMOMO interaction data provides limited coverage of box manipulation at different support-surface heights, producing less reliable grasps and placements. The same hierarchy appears in simulation, where surface modes fall to 45.8%–72.2% while navigation approaches 100%, confirming that the difficulty is in the learned behavior rather than hardware execution alone.

One ablation deserves emphasis because it validates a design decision: removing the wrist-object contact label from Pick (Floor) yields 0/5 real-world successes in both scenes. The binary contact signal — freely available from the synthetic pipeline and predicted auto-regressively at deployment — is what lets the tracker engage contact-aware wrist behavior at the right time. It is a small piece of state with outsized functional importance.

Figure 5: Real-world deployment examples. The VLK system executes language-conditioned navigation and box manipulation under different layouts and visual variations, including lighting changes in the apartment scene.

The data-volume ablation trains policies on different fractions of the synthesized dataset and evaluates closed-loop in simulation. Increasing data consistently improves every mode, but the shape differs by task family: navigation reaches high success with limited data, while manipulation keeps benefiting — Pick (Surface) climbs from 0% success at 10% of the data to 46% with the full dataset. The reading is that contact-rich, perception-conditioned manipulation requires more scene-grounded supervision than navigation, which strengthens the paper's central argument that scalable synthesis, not a cleverer policy, is the binding constraint for manipulation performance.

The domain-randomization ablation quantifies the visual sim-to-real gap on walking-mode success under visual perturbations. With no randomization the policy achieves 41%; camera randomization alone lifts it to 48%; lighting randomization alone to 87%; and the full combination to 90%. Lighting dominates the gap in this environment, camera calibration mismatch contributes modestly, and the two combine nearly additively. This is a useful, concrete decomposition of "the visual gap" for anyone building rendered-to-real pipelines.

5. Limitations

The authors are explicit about the boundary of the current system. Interaction synthesis is bounded by OMOMO's coverage, which contains interactions with a limited set of large objects; the generated behaviors are therefore better suited to bimanual transport of box-like objects than to grasping small objects such as cups or tools. Likewise, the contact-aware tracker stabilizes large-object manipulation through wrist-object contact but does not address precise object grasping. Extending VLK to small-object manipulation would require both richer interaction data and a low-level controller designed for precise grasping.

Beyond the self-reported limits, the results themselves expose others. Surface-level manipulation remains the fragile regime in both simulation and the real world, a direct consequence of retargeted-data coverage at varied support heights. The language interface is template-based rather than open-vocabulary, so the system's "language conditioning" currently spans a handful of instruction families tied to the six task modes. And the data story, while efficient, is anchored to two physical environments with eight scanned layouts; generalization to unseen environments would require scaling the scan-reconstruct-annotate loop itself, of which the annotation step (semantic boxes and walkable regions per scene) is still manual.

6. Summary and Outlook

VLK's contribution is a complete real-to-sim-to-real data engine for perception-based humanoid loco-manipulation, demonstrated end to end on a physical Unitree G1. The paper connects three maturing pieces — metric-scale 3DGS scene reconstruction, conditional diffusion-based humanoid motion synthesis, and whole-body tracking — with a decoupled generation scheme in which language is prescribed, privileged scene information guides behavior synthesis, and egocentric observations are rendered in hindsight. The payoff is 48,000 paired vision-language-kinematics trajectories within 600 GPU-hours and zero human demonstration effort, which train a pi0.5-initialized policy that predicts one-second whole-body kinematic chunks executed by a contact-aware SceneBot tracker at 50 Hz.

The evidence is strongest where the data coverage is strongest: navigation and floor-level transport reach near-perfect real-world success, while surface manipulation degrades in proportion to the retargeted data's coverage gap — and the data-volume ablation shows exactly that more synthesized interaction data is what closes such gaps. That is the paper's real claim on the field's direction: for contact-rich loco-manipulation, the bottleneck is supervision, and reconstructed-scene synthesis is a scalable way to manufacture it. Natural next steps the paper points toward include richer interaction datasets for small-object grasping, precision-capable low-level controllers, and broader scene diversity from scaling the reconstruction pipeline.

7. Golden Quote

"We can prescribe the language commands, leverage privileged information in the simulated world to ease the challenge of behavior generation, and render the corresponding egocentric observations as RGB images after the fact." — this decoupling of language, behavior, and vision is the single reason the data pipeline scales without human demonstrations, and it is the idea worth stealing.