Scaling Bimanual Household Manipulation from 1,500 hours of Demonstrations to On-Policy Corrections

Paper Metadata

Title: Scaling Bimanual Household Manipulation from 1,500 hours of Demonstrations to On-Policy Corrections

Authors: Jiafeng Xu, Qi Li, Yan Shen, Yiyu Ren, Travis Davies, Shaowen He, Ze Wang, Yifan Yang, Ran Cheng, Hao Dong (PrimeBot Research Institute, School of Computer Science at Peking University, Crobotia)

Links: arXiv:2609.03591 (https://arxiv.org/abs/2609.03591 , v1, 3 Sep 2026); dataset at https://huggingface.co/datasets/challenge-2026/challenge_data

Code and data status: there is no standalone code repository; the full corpus and the X2W URDF are released on HuggingFace in LeRobot format under CC-BY-SA-4.0, together with the IROS 2026 Household Bimanual Manipulation Challenge. The dataset page reports more than two hundred thousand downloads.

Fig. 1: X2W hardware configuration. The platform has 25 degrees of freedom with RealSense D435i, ZED X Mini and ZED-XONE GS cameras, an IMU and heterogeneous compute units for household bimanual manipulation.

One-Sentence Summary

The paper releases 1,500 hours of bimanual household manipulation data (531.7 hours of real-robot teleoperation plus roughly 1,000 hours of in-the-wild UMI capture), trains a 5B-parameter VLA called XR-2 on it, and reports two connected scaling curves on clothes folding: expert data lifts success from 34% to 84% before saturating, and three rounds of on-policy DAgger corrections then push the same checkpoint from 58% to 93%.

Background and Motivation

Bimanual manipulation sits among the hardest problems in robot learning. Two arms must divide labor and coordinate under rich contact, tasks stretch across dozens of sub-steps, and every change in object pose or fabric configuration forces the policy to replan online. Recent VLA progress has made data scaling the main axis of generalist manipulation, with corpora such as Open X-Embodiment, DROID, AgiBot World and RoboMIND pushing scale from hundreds to thousands of hours. As datasets grow, however, the community question shifts from how much data to collect to which data are most worth collecting.

Offline expert demonstrations and on-policy corrective data are complementary information sources. The former shows what a successful trajectory looks like, with a clean, controllable and reusable state distribution. The latter is generated exactly when the deployed policy rolls out and fails, covering the failure states that the expert distribution never contains. Without the first, the policy has no stable behavioral base; without the second, the policy collapses once it drifts out of the expert distribution, and no amount of offline data supplies those states.

For bimanual manipulation the gap is sharper. Public large-scale manipulation corpora have long been dominated by single-arm or fixed-base settings, and in-the-wild two-handed household data remains scarce, so the scaling behavior of bimanual policies lacks an empirical basis. Nobody could say at what scale expert data saturates, nor quantify what on-policy corrections can still buy after saturation.

This paper answers both questions with an engineered data system. The authors assemble a 1,500-hour bimanual household corpus: 32,518 teleoperated real-robot trajectories, 57.4 million frames and 531.7 hours of interaction across four household settings and more than twelve everyday tasks, complemented by about 1,000 hours of in-the-wild UMI bimanual garment manipulation from more than two hundred households. On top of it they train XR-2, a 5B-parameter VLA, and study two scaling axes: the amount of expert demonstration data and the amount of on-policy DAgger correction data.

The experimental story is consistent. On clothes folding, success rises monotonically as expert data grows from 10 to 120 hours and then saturates; past that point three rounds of DAgger corrections raise success from 58% to 93%. Expert demonstrations and corrective on-policy data are therefore complementary levers of scaling rather than competing routes. The corpus, with temporally aligned sub-task language annotations, is released to support reproducible research on data-centric scaling for bimanual manipulation.

Preliminaries

Conditional flow matching. The action head of XR-2 neither outputs a discrete distribution nor regresses a single target; it learns a velocity field. A linear interpolation path $\mathbf{a}_{t}=(1-t)\epsilon+t\mathbf{a}$ is defined between noise and the real action, with $\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})$, and the model regresses the target flow $(\mathbf{a}-\epsilon)$. At inference a few Euler steps from pure noise sample an entire action chunk, combining multimodal action distributions with real-time control.

DAgger. Dataset Aggregation asks the expert to keep labeling the states that the policy itself visits, so the training distribution approaches the deployment distribution. On a real robot, expert labeling becomes human takeover: when the policy errs, an operator takes over, corrects or completes the current sub-task, and the takeover segment is a corrective sample with state and action supervision.

Action chunks and temporal ensembling. The policy emits a $K$-step action sequence per call instead of a single action, amortizing inference latency; overlapping chunks are fused by temporal ensembling into smooth commands, avoiding jitter at chunk boundaries.

Method

The X2W mobile bimanual platform. The robot carries two 7-DoF arms, each terminated by a 1-DoF gripper, plus a 4-DoF waist and a 2-DoF head, for 22 controllable joints mounted on a three-wheel omnidirectional base; the three independently driven wheels add three velocity commands, so the machine has 25 control variables in total. Main-body joints use quasi-direct-drive actuators for responsive, compliant motion under rich contact. Vision comes from one head-mounted camera for global scene context and two wrist-mounted cameras for close-range manipulation detail, and all proprioception, actions and visual observations are recorded synchronously at 30 Hz.

State and action spaces. Proprioception is an 89-dimensional vector:

$$\mathbf{s}_{t}=\left[\mathbf{q}_{t},\dot{\mathbf{q}}_{t},\boldsymbol{\tau}_{t},\mathbf{p}_{t}^{L},\mathbf{p}_{t}^{R},\mathbf{q}_{t}^{w},\dot{\mathbf{q}}_{t}^{w},\boldsymbol{\tau}_{t}^{w}\right]\in\mathbb{R}^{89}$$

concatenating positions, velocities and torques of the 22 controllable joints, the two end-effector poses $\mathbf{p}_{t}^{L},\mathbf{p}_{t}^{R}$, and positions, velocities and torques of the three drive wheels. The action is 22 target joint positions plus 3 target wheel velocities:

$$\mathbf{a}_{t}=\left[\mathbf{q}_{t}^{\mathrm{cmd}},\dot{\mathbf{q}}_{t}^{w,\mathrm{cmd}}\right]\in\mathbb{R}^{25}$$

Putting the mobile base inside the policy action space lets whole-body manipulation, including operating while driving, be learned end to end in one policy instead of splitting navigation and manipulation into separate modules.

Expert demonstration pipeline. Offline collection decomposes long-horizon tasks into sub-tasks along their inherent structure and writes a Standard Operating Procedure per sub-task, fixing scene setup, object arrangement, robot initialization and manipulation workflow; remote operators teleoperate the robot through VR devices. Raw multimodal trajectories go through automatic post-processing and format conversion, then cloud-side automatic quality validation, VLM-powered task and language annotation, and a final manual review. This standardized pipeline keeps data consistent across robots, scenes and operators, which is what makes 531.7 hours of data usable at quality.

Online DAgger pipeline. Multiple robots execute complete long-horizon tasks with a shared policy. Whenever the policy reaches a failure state or needs correction, a human operator temporarily takes control, corrects or completes the current sub-task, and immediately returns control to the policy, so a single rollout can contain several alternating policy- and human-controlled segments without restarting the task. The system records every intervention segment as a corrective demonstration, binds it to the corresponding sub-task prompt, and explicitly marks policy-generated versus human-controlled segments inside the same trajectory, so DAgger data needs no re-annotation in the cloud. As deployed robots multiply, execution failures keep converting into new corrective data, forming a scalable closed-loop pipeline.

Fig. 2: Dataset statistics. (a) Scene coverage across four household interaction settings; (b) episode duration distribution with a long tail near 150 s; (c) distribution of the 11 atomic skills.

Real-robot corpus. The 32,518 teleoperated trajectories span four household settings: folding station (about 14.6k), laundry washer (about 8.4k), living-room sofa (about 6.1k) and laundry basket (about 3.4k). Durations range from short atomic primitives to long sequences beyond 100 s, with some past 150 s. Merging semantically equivalent instructions yields 11 atomic skills, from high-frequency garment folding (14.6k) to less frequent but task-critical primitives such as loading clothes into the washer (1.2k) and putting a plush toy into the washer (0.8k); because one long-horizon trajectory may contain several skills, skill counts are not mutually exclusive. Multi-stage tasks carry time-aligned sub-step language annotations, providing supervision at both task and skill level.

UMI corpus. Complementing the teleoperated set, the authors collect about 1,000 hours of in-the-wild bimanual UMI household garment demonstrations (folding and organizing), matched to the folding-station task distribution. Each demonstration provides synchronized dual egocentric video at $960\times 960$ and 30 Hz, metric 6-DoF end-effector trajectories recovered by visual-inertial SLAM, continuous gripper aperture, and frame-level language annotations at two granularities: item-level segments and fine-grained action steps inside each segment. Data come from more than 200 households and over 100 collectors across 5,000 distinct garments, on beds, tabletops and drying racks, from morning to late night, so coverage of scenes, lighting and manipulation styles exceeds what a fixed robot cell can offer.

The encoder-free bimanual gripper. Each UMI gripper is fully mechanical and self-contained: the wrist camera is its only sensor, aperture is recovered visually rather than from encoders, and no head-mounted or third-person camera is involved. The authors support this design with three arguments: wrist-view-only observation suffices for bimanual coordination in manipulation tasks; hand-centric views improve training efficiency and out-of-distribution generalization over third-person views; and recent in-the-wild systems with hand-centric-only sensing deploy zero-shot in unseen homes. Dropping encoders, motors and actuation electronics also makes the gripper lighter and lower power, enabling hours-long collection and reset-free sessions of up to 90 minutes, while the directly hand-driven jaws still deliver grasp forces strong and modulated enough to tension and pin fabric during folding.

Fig. 3: Two-stage full-parameter training of XR-2. Both stages share the VLM plus action-expert architecture with all parameters trainable; stage 1 fits offline teleoperated demonstrations, stage 2 rolls out the policy, collects expert corrections on visited states and retrains on the aggregated data.

XR-2 architecture. XR-2 is a 5B-parameter Mixture-of-Transformers VLA: a pretrained Qwen3-VL-4B-Instruct backbone handles multimodal perception, and an action expert, a diffusion transformer trained with flow matching, produces control. The expert stacks 18 transformer blocks with hidden dimension $D_{\text{hidden}}=1024$, 8 attention heads and grouped key-value attention with 4 KV heads. Inside each block, noisy action tokens first undergo bidirectional self-attention, then cross-attend to backbone hidden states, then pass a SwiGLU feed-forward layer. The cross-attention is depth-aligned: block $i$ attends to the $i$-th of the 18 exposed backbone layers, so the expert grounds its predictions in progressively more abstract visual-language features instead of collapsing all conditioning onto the final layer; the denoising timestep enters through adaLN-zero modulation. The policy reads $\mathbf{a}_{t}=\pi_{\theta}(\mathbf{o}_{t},\mathbf{s}_{t},l)$, generating a $K$-length action chunk conditioned on vision, language and proprioception.

Action prefix conditioning for asynchronous execution. To make inference and execution seamlessly asynchronous, the expert is additionally trained with action prefix conditioning: a delay $d$ marks a leading action segment as already committed, its tokens anchored to ground truth with timesteps fixed at $t=1$, while only the subsequent suffix receives noise injection, denoising and loss supervision. At inference $d$ is computed dynamically as the average execution duration of the most recent $R$ inference rounds, so compute latency is modeled explicitly inside generation; full action sequences are synthesized with five Euler forward integration steps for low-latency smooth control.

Stage 1: conditional flow matching on expert data. With expert demonstrations collected, XR-2 is trained with the conditional flow matching objective so a single policy learns multiple sub-tasks jointly:

$$\mathcal{L}_{\mathrm{CFM}}(\theta)=\mathbb{E}_{\{\mathbf{a},\mathbf{o},\mathbf{s},l\}\sim\mathcal{D},\,\epsilon,\,t}\left\|\pi_{\theta}(\mathbf{a}_{t},\mathbf{o},\mathbf{s},l)-(\mathbf{a}-\epsilon)\right\|^{2}$$

with $\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})$, $t\sim\mathrm{Beta}(1.0,1.5)$ and noisy chunk $\mathbf{a}_{t}=(1-t)\epsilon+t\mathbf{a}$; the policy predicts the target flow $(\mathbf{a}-\epsilon)$ conditioned on the noisy action, visual observations, robot state and language instruction. The Beta sampling concentrates training mass on the middle and late part of the interpolation path, and the same full-parameter objective serves both stages.

Stage 2: post-training on DAgger data. Corrective trajectories stream to the cloud, where the policy is further optimized for failure cases in the test environment. Two challenges arise: the policy must acquire failure recovery from online corrections without forgetting existing skills, and sub-tasks with different learning dynamics must be trained in balance. The authors address both through data allocation. First, DAgger and expert data are mixed 1:1 by number of trajectories rather than frames: sub-task trajectory lengths differ, so frame-level mixing lets random sampling distort skill coverage, while matching trajectory counts preserves the expert skill distribution. Second, the DAgger budget is biased toward sub-tasks the policy handles poorly. With $N$ sub-tasks and $s_{i}$ the held-out success rate of the current policy on sub-task $i$, the fraction of DAgger trajectories allocated to sub-task $i$ is

$$w_{i}=w_{\min}+\bigl(1-Nw_{\min}\bigr)\,\frac{\bigl(1-s_{i}+\epsilon\bigr)^{\alpha}}{\sum_{j=1}^{N}\bigl(1-s_{j}+\epsilon\bigr)^{\alpha}}$$

where $1-s_{i}$ is the demand signal, $\alpha\geq 0$ controls how sharply the budget concentrates on weak sub-tasks ($\alpha=0$ recovers uniform allocation), $\epsilon>0$ keeps shares positive once a sub-task is solved, and $w_{\min}$ floors every share; the authors set $w_{\min}=0.05$, $\epsilon=0.05$ and $\alpha=1$. Reserving $Nw_{\min}$ before the proportional split keeps the allocation closed-form with $\sum_{i=1}^{N}w_{i}=1$ and $w_{i}\geq w_{\min}$, and the floor guards against forgetting because strong sub-tasks retain data while the budget shifts toward weak ones.

flowchart TD
  A[VR teleoperation with SOP expert demos] --> B[cloud QC plus VLM annotation plus manual review]
  C[multi-robot on-policy rollouts with shared policy] --> D[human takeover on failure segments]
  D --> E[DAgger corrective data with sub-task prompts]
  B --> F[Stage 1 CFM full-parameter training on expert data]
  F --> G[deploy 10 Hz async inference 30 Hz commands 1000 Hz whole-body control]
  G --> C
  E --> H[Stage 2 failure-weighted DAgger post-training Eq 4]
  F --> H
  H --> G

The diagram closes the loop of the method: the expert pipeline feeds stage 1, deployment failures feed stage 2, and the stage-2 checkpoint returns to deployment, a flywheel in which data and policy grow together.

Experiments

Deployment stack and evaluation protocol. The policy runs on a single NVIDIA GeForce RTX 4090, performing asynchronous inference at 10 Hz and emitting action chunks $\mathbf{a}_{t}\in\mathbb{R}^{50\times 25}$; after temporal ensembling, whole-body joint commands are issued synchronously at 30 Hz and then optimized in real time at 1000 Hz by a whole-body motion controller inherited from the teleoperation phase. All test environments include slight generalizations in material size, material color and initial robot position; the authors report that a model reaching 85% success under generalized conditions exceeds 95% in-distribution.

Dimension	Real-robot teleoperation corpus	In-the-wild UMI corpus
Scale	531.7 h, 32,518 trajectories, 57.4M frames	about 1,000 h of bimanual garment folding and organizing
Scenes	folding station 14.6k, washer 8.4k, sofa 6.1k, basket 3.4k	200+ households, 100+ collectors, 5,000 garments
Capture	VR teleoperation of X2W under SOPs	encoder-free handheld grippers, dual egocentric 960x960 at 30 Hz
State supervision	89-D proprioception plus 25-D actions at 30 Hz	SLAM-recovered 6-DoF end-effector trajectories plus visual aperture
Language	frame-level plus sub-task level, 11 atomic skills	item-level segments plus within-segment action steps

Scaling of expert data. On clothes folding the training set grows from 10 to 160 hours (8,000 trajectories). Success rises monotonically from 34% at 30 hours to 84% at 120 hours, then saturates: the final 40 hours bring no further improvement, ending at 82% for 160 hours. The scale of the released corpus is what makes this transition measurable, because it locates the point where expert demonstrations already cover the task distribution densely. Beyond it, residual failures come from states the policy reaches only at test time, outside the expert distribution and unreachable by more expert data; closing that gap requires human-in-the-loop intervention data.

Effective expert data	Folding success	Reading
30 h	34%	coverage still thin, failures mostly in-distribution
120 h	84%	expert distribution densely covers the task, near saturation
160 h	82%	last 40 h add nothing, saturation confirmed

Fig. 4: Data scaling on clothes folding. (a) Success versus effective expert-data duration, saturating after 120 h; (b) three DAgger rounds from the 58% checkpoint reaching 93%.

DAgger experiment. Given collection time overhead and compute constraints, the DAgger experiments initialize from a checkpoint trained on 18,695 expert trajectories spanning 10 sub-tasks. Per-round DAgger trajectory counts for folding follow the failure-weighted allocation, each iteration runs 2.5 epochs of post-training, and success climbs from 58% to 74%, 82% and 93% after the first, second and third iterations, a 35-point gain over the expert-only baseline.

Training round	Folding success	Delta
init (18,695 expert trajectories)	58%	-
DAgger round 1	74%	+16
DAgger round 2	82%	+8
DAgger round 3	93%	+11

How the two curves corroborate each other. The expert curve answers how far in-distribution coverage can go; the DAgger curve answers how out-of-distribution failures get repaired. The point where the first saturates is exactly where the second starts, and the two meet at the 58% checkpoint. Read together, the strongest evidence in the paper is not the single 93% number but the joint fact that a saturation point exists and that corrective data keeps paying after it, moving the data-scaling discussion from how much to which kind.

Fig. 5: Representative trajectories from the released real-robot corpus, head-camera view. Each row is one episode with the active sub-task instruction above the strip, covering washer interaction, garment loading and unloading, object placement, bimanual basket handling and long-horizon unfolding and folding.

Limitations

Author-stated: expert data stops paying after 120 hours. The paper explicitly reports that the last 40 hours bring no improvement and attributes residual failures to states reached only at test time; the scaling claim is therefore stated only for the probed data range, as a clear consistent trend at the current data scale.

Author-stated: DAgger experiments are resource-bounded. Because of collection time overhead and compute constraints, DAgger does not start from a checkpoint trained on the full 1,500-hour corpus but from 18,695 trajectories over 10 sub-tasks; per-round corrective volume is bounded by human takeover cost, and behavior beyond three rounds is not measured.

Our assessment: scaling evidence concentrates on one task. Both scaling curves are measured on clothes folding only. Whether the saturation point and the 35-point DAgger gain transfer to the washer, basket and other sub-tasks, or to longer compositional tasks, is not evidenced in the paper.

Our assessment: no cross-baseline comparison. The paper does not compare XR-2 head-to-head with other public VLAs of similar scale or with cross-dataset training recipes, so the claim of strong manipulation performance rests mainly on the authors' own evaluation protocol; the UMI subset is also limited to garment folding and organizing, and its independent contribution to downstream policy quality is not ablated.

Conclusion and Outlook

The contribution has three layers. The data layer releases 1,500 hours of bimanual household manipulation with time-aligned sub-task annotations, filling a gap in in-the-wild two-handed data. The model layer ships XR-2, a 5B VLA whose depth-aligned cross-attention and action prefix conditioning build asynchronous real-time control into the architecture. The scientific layer states that expert data and on-policy corrective data are complementary scaling levers, the first densely covering the task distribution and the second repairing out-of-distribution failures. For teams running real robots, the reusable asset is not only the corpus but the pipeline itself, SOP-standardized collection plus takeover-as-annotation during deployment, which turns every deployment failure into budget for the next training round. The authors position the corpus as a shared basis for studying data-centric scaling in bimanual manipulation, with the IROS 2026 challenge carrying the question forward on a common benchmark.

Golden Quote

"Scaling data and structuring it around on-policy failures are complementary levers rather than competing ones." Expert demonstrations teach the policy how to succeed; on-policy corrections teach it how to recover, and the two curves hand over at the 58% checkpoint to carry folding success to 93%.