PAPER DEEP DIVE
Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization
Zero-shot cross-task generalization, where a policy must execute manipulation tasks never seen during training, remains a central challenge in robot learning. In large language models, a novel task can be performed simply by specifying it in the context, without any parameter update. This form of in-context learning (ICL) turns generalization into a problem of task specification. To achieve cross-task generalization, we bring this paradigm to robotic manipulation, and argue that the natural task specification for manipulation is a human video: unlike language, it provides rich visual cues about the intended task evolution. We present Zero-WAM, a causal video-action model that executes unseen tasks by following in-context human video guidance. To address the scarcity of task-rich paired human-robot data, we propose an automatic pipeline that converts task-sampled robot trajectories into semantically matched human videos, yielding HumanGen, a dataset of 74.2K human-robot ICL pairs across 8.6K tasks. For model training, we further introduce an in-context future chunk prediction (IFP) objective that suppresses shortcuts learned from seen tasks and forces the policy to draw task information from the video prompt. On seven unseen tasks in RoboTwin 2.0 simulation, Zero-WAM achieves a 47.0% average success rate, an absolute improvement of 29.5 percentage points over the strongest video-action baseline. In real-world evaluations, it follows human video guidance to generalize to unseen task configurations involving multi-object scenes, long-horizon manipulation, and fine-grained insertion.
Paper: Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization
Authors: Jiaming Zhou, Qihang Zhang, Gangwei Xu, Cunxin Fan, Yujie Zhao, Ruilin Wang, Yiming Luo, Shuai Yang, Xing Zhu, Yujun Shen, Junwei Liang, Yinghao Xu
Affiliations: Robbyant · HKUST (GZ) · HKUST
Links: arXiv:2608.26103 · Project Page · GitHub
Code status: repository opened; code, model, and data releases all planned before Sep 15, 2026 (see the README release plan). The repository currently contains only the README; this article is based on the paper's full text.
One-Sentence Summary
Zero-WAM turns human demonstration videos into an in-context task instruction interface for robots: with a HumanGen dataset of 74.2K human-robot pairs synthesized by a generative pipeline and a shortcut-suppressing future-chunk-prediction objective (IFP), a causal video-action model executes seven unseen RoboTwin 2.0 tasks at 46.95% average success — 29.5 points above the strongest baseline — without any task-specific robot data or parameter updates.
Figure 1: Zero-WAM overview. From the current robot observation, the model conditions on either a human demonstration video or a language instruction, predicts the next robot video chunk, and decodes the temporally aligned executable action chunk.
Background and Motivation
Zero-shot cross-task generalization is the task-level exam of robot manipulation: the policy must complete a manipulation task it has never practiced, with no robot demonstrations collected for it and no parameter updates. Note that this differs from the easier "visual generalization" setting — changing backgrounds, object colors, or viewpoints around a trained task. Cross-task generalization requires the model to infer task dynamics it has never seen, purely from the task specification and the current observation.
Vision-language-action (VLA) models have pushed language-conditioned manipulation policies to large scale, yet the paper points out that VLA-style scaling has not closed the cross-task gap: there remains a large mismatch between the vision-language representation space and the robot action space. A text instruction such as "wipe the wine glass" carries very little information about the trajectory's fine dynamics — where to grasp, in what order to move, how hard to close the gripper. Representative VLA policies evaluated under zero-shot cross-task protocols largely fail; workarounds like AGNOSTOS require handing the policy a robot trajectory from the unseen task at test time, which defeats the point of zero-shot.
World action models (WAMs) offer a different route: jointly predict future visual states and executable actions, shifting the challenge from direct action generalization to unseen video generation. Since accurate robot actions can be decoded from correctly predicted future videos, the world-model component carries the task dynamics. But WAMs still face a task-specification problem at test time: how do you convey an unseen task to the model? Prior test-time-training approaches (WAM-TTT, RoboTTT) adapt memory or fast weights at deployment — costly and not truly zero-shot.
Zero-WAM's core argument: the natural task specification for manipulation is a human video, not text. Large language models showed that in-context learning converts generalization into a task-specification problem — you write the new task into the context and no weights change. Bring this to robotics and you need a task interface that carries rich dynamics: a human demonstration video directly shows the intended evolution of object states — grasp position, motion path, force application, order of steps — bypassing the embodiment gap at the specification level (the human need not be operating the same robot as long as the task semantics align). One policy, two instruction modalities: language instructions and human video instructions.
The bottleneck is data. Scaling in-context learning requires task-rich, semantically aligned human-robot paired data, and manually collecting human and robot demonstrations of the same task is costly and grows linearly with the number of tasks. Table 1 compares existing datasets: prior paired datasets (EgoDex, UMI, Humanoid-UMI, EgoMimic, HumanGen) are either small-scale, restricted to a handful of tasks, or require synchronized human-robot capture devices. Zero-WAM's answer is to synthesize the pairs: an automatic pipeline converts task-sampled robot trajectories into semantically matched human videos, producing 74.2K human-robot ICL pairs over 8.6K tasks. The paper's contributions are: (1) formulating zero-shot task generalization as in-context world-action modeling; (2) the HumanGen generation pipeline and Task-diverse VA task-balanced corpus; (3) the in-context future chunk prediction (IFP) objective that suppresses shortcut learning; (4) strong zero-shot results in RoboTwin 2.0 simulation and on real unseen task configurations.
Data: Task-Balanced Curation and Human Video Generation
Zero-WAM's training corpus has two parts, both built on top of task-level sampling. The first is Task-diverse VA: the authors re-curate five public robotic video-action pre-training datasets (AgiBot, InternData-A1, Open-X-Embodiment, RoboCOIN, RoboMIND — the same sources used by LingBot-VA). The problem with these corpora is redundancy: many trajectories are repeated tele-operation of the same task, and training on the raw distribution lets a few heavily duplicated tasks dominate. The authors re-partition each dataset into tasks — each task defined by the combination of manipulation action and object, using original metadata where available or parsing trajectories where not — then sample a bounded number of trajectories per task (the bound adjusted to each dataset's intra-task diversity). The result: more than 6,000 tasks and roughly 400K robot trajectories per training epoch.
The second part is HumanGen, built by the in-context human video generation pipeline. Starting from the same task-level sampling of the robotic corpus (so the source robot videos retain executable action annotations), the pipeline converts each sampled robot video into a human manipulation video with identical task semantics through four stages:
- Task analysis (VLM). Gemini 3.1 Pro or Qwen3.6-Plus extracts the task name, initial object states, object state changes, and final object states, and produces an image-editing prompt that converts the first robot frame into the initial observation of a human manipulation scene. Visual-alignment diversity — backgrounds, viewpoints, environment styles, object instances, placements — is injected through this prompt while preserving task semantics.
- Human first-frame synthesis (image editing). Nano Banana 2 or Qwen-Image-2.0 edits the first robot frame into the initial human observation image.
- Human video synthesis (video generation). The VLM combines the edited human observation with the extracted object-state information into a video-generation prompt describing how human hands should manipulate the objects; Wan 2.7 or Kling AI 3.0 synthesizes the human manipulation video.
- Quality control (VLM). The VLM evaluates each generated video for task-semantic preservation and physical plausibility; only qualified videos are paired back with the original robot trajectory as ICL samples.
Figure 2: Data construction and in-context human video generation. Top: Task-diverse VA supplies task-balanced robotic video-action pre-training data; HumanGen contains Pre-train ICL (External), Pre-train ICL (In-house), Simulation ICL, and Real-world ICL pairs. Bottom: the four-stage generation pipeline.
HumanGen contains 74.2K human-robot ICL pairs spanning 8.6K tasks, in four subsets: Pre-train ICL (External) with 5,062 tasks / 41,188 pairs from public data, Pre-train ICL (In-house) with 3,522 tasks / 30,247 pairs, Simulation ICL with 50 RoboTwin tasks / 2,500 pairs (43 tasks for training, 7 held out for evaluation), and Real-world ICL with 252 pairs. Table 1 contrasts this with prior task-level paired datasets: existing efforts either stay in the hundreds of tasks with synchronized capture hardware, or are small-scale; HumanGen is the first to scale paired human-robot task data into the thousands of tasks through generation rather than collection.
Method Deep Dive
1. Foundations: Flow Matching and Causal Video-Action Modeling
Zero-WAM is built on flow matching for video generation. Given a clean video $\mathbf{x}_{0}$, Gaussian noise $\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$, and flow time $t\in[0,1]$, the noised video and target velocity are
$$\mathbf{x}_{t}=(1-t)\mathbf{x}_{0}+t\bm{\epsilon},\qquad\mathbf{v}^{\star}_{t}=\bm{\epsilon}-\mathbf{x}_{0}$$and, with condition $c$, the flow-matching loss is
$$\mathcal{L}_{\mathrm{fm}}=\mathbb{E}_{\mathbf{x}_{0},\bm{\epsilon},t}\left[\left\|\mathbf{v}_{\theta}(\mathbf{x}_{t},t,c)-\mathbf{v}^{\star}_{t}\right\|_{2}^{2}\right]$$At inference, generation starts from noise and integrates the learned velocity field back to $t=0$. On top of this, the paper follows the causal video-action framework of the LingBot-VA series: each robot trajectory is chunked into $\tau=\{(\mathbf{x}^{i},\mathbf{a}^{i})\}_{i=1}^{N}$ (video chunk $\mathbf{x}^{i}$ with its temporally aligned action chunk $\mathbf{a}^{i}$), and at chunk index $i$ the model predicts the next video-action chunk causally:
$$p_{\theta}\!\left(\mathbf{x}^{i+1},\mathbf{a}^{i+1}\mid\mathbf{x}^{\leq i},\mathbf{a}^{\leq i},c\right)$$This joint prediction factorizes into video prediction followed by action decoding:
$$p_{\theta}\!\left(\mathbf{x}^{i+1},\mathbf{a}^{i+1}\mid\mathbf{x}^{\leq i},\mathbf{a}^{i\leq i},c\right)=p_{\theta}^{\mathrm{vid}}\!\left(\mathbf{x}^{i+1}\mid\mathbf{x}^{\leq i},\mathbf{a}^{\leq i},c\right)\cdot p_{\theta}^{\mathrm{act}}\!\left(\mathbf{a}^{i+1}\mid\mathbf{x}^{\leq i},\mathbf{a}^{\leq i},\mathbf{x}^{i+1},c\right)$$where $p_{\theta}^{\mathrm{vid}}$ predicts the next video chunk and $p_{\theta}^{\mathrm{act}}$ acts as an inverse dynamics model decoding executable actions from the predicted next robot video. The two branches are implemented as a video Transformer and an action Transformer under a Mixture-of-Transformers (MoT) design: each modality keeps its own QKV projections, FFNs, and output heads, while both streams stay in a single sequence and interact only through shared attention layers. Action chunk representations are placed after the future video representations so that action decoding can attend to the predicted future robot video, as the factorization requires. The whole system is instantiated by converting Wan-2.2-TI2V-5B, a bidirectional text/image-to-video generation model, into this causal video-action policy.
2. Human Video as In-Context Task Specification
For HumanGen samples, the condition is $c=\{\mathbf{h},\ell\}$ — the same-task human video $\mathbf{h}$ plus the language instruction $\ell$; for Task-diverse VA samples the condition is language-only, $c=\ell$. During training, a teacher-forcing attention mask is applied over the tokenized human-video, robot-video, and action chunks. The human video $\mathbf{h}$ is prepended before the robot trajectory and serves as prefix memory. The video branch predicts the next robot video chunk from
$$\mathcal{C}^{\mathrm{vid},i}=[\,[\mathbf{h},\mathbf{x}^{\leq i}],\mathbf{a}^{\leq i},\ell\,],\qquad p_{\theta}^{\mathrm{vid}}(\mathbf{x}^{i+1}\mid\mathcal{C}^{\mathrm{vid},i})$$The action branch deliberately does not attend to the human video: the task semantics carried by $\mathbf{h}$ are meant to be absorbed into the predicted next robot video chunk, so action decoding reduces to standard inverse dynamics. Its condition stays unchanged from ordinary robot video-action training:
$$\mathcal{C}^{\mathrm{act},i}=[\,\mathbf{x}^{\leq i},\mathbf{a}^{\leq i},\mathbf{x}^{i+1},\ell\,],\qquad p_{\theta}^{\mathrm{act}}(\mathbf{a}^{i+1}\mid\mathcal{C}^{\mathrm{act},i})$$During training $\mathbf{x}^{i+1}$ in the action context is the ground-truth next chunk under teacher forcing; at inference it is replaced by the video Transformer's generation. Since the human video and the paired robot trajectory differ in embodiment, viewpoint, background, and object placement by construction, the model cannot succeed by copying motions — it must learn task-level correspondence.
A clean implementation detail separates the human video from robot videos inside the token sequence. Both are encoded by the same Wan-2.2 VAE and share the same visual latent space, so they are distinguished by height-axis RoPE coordinates: robot video latents keep their original coordinates while human video latents are shifted along the height axis,
$$\mathrm{pos}_{\mathrm{robot}}(q,y,x)=(q,y,x),\qquad\mathrm{pos}_{\mathrm{human}}(q,y,x)=(q,y+\Delta_{H},x),\qquad\Delta_{H}>H_{\mathrm{mv}}$$where $(q,y,x)$ are the temporal, vertical, and horizontal coordinates of a visual latent and $H_{\mathrm{mv}}$ is the height of the multi-view robot video latent layout. The offset (set to $\Delta_{H}=32$ in practice) places human-video latents outside the coordinate range of robot-video latents, preventing representation confusion when the two interact in one sequence.
3. In-Context Future Chunk Prediction (IFP)
This is the paper's key training-side insight. Under teacher-forcing on seen tasks, the immediate next robot video chunk can often be predicted by extrapolating recent robot history $(\mathbf{x}^{\leq i})$ alone — a shortcut that lets the model reduce training loss without ever learning to use the human video. At test time on unseen tasks, exactly when the ICL signal is needed, the model keeps leaning on robot history and underuses the prompt. The fix, inspired by Next Forcing's multi-chunk prediction: a training-only auxiliary objective that forces the model to predict multiple strided future robot video chunks from the current robot-video representation. With temporal stride $s\geq1$ and $K$ future chunks, the $k$-th target index is
$$j_{k}=(i+1)+1+(k-1)s,\qquad k=1,\ldots,K$$The model adds $K$ IFP modules $\{G_{k}\}_{k=1}^{K}$, each with the architecture of a single video-branch Transformer layer, initialized from the last video Transformer layer. To predict the targets, robot-video hidden representations $\{\mathbf{r}^{i+1}_{m}\}_{m=1}^{M}$ are collected from $M$ intermediate layers of the main video Transformer (representations of the noised chunk $\mathbf{x}^{i+1}_{t}$ at flow time $t$), concatenated, and projected back to single-layer hidden dimension by a lightweight MLP $P_{\mathrm{fuse}}$:
$$\bm{\phi}^{i+1}=P_{\mathrm{fuse}}\left(\mathrm{Concat}\left(\{\mathbf{r}^{i+1}_{m}\}_{m=1}^{M}\right)\right)$$Each IFP module then denoises its strided future chunk under the flow-matching objective, conditioned on the fused current representation, the clean robot history, and the language instruction:
$$p_{\theta,k}^{\mathrm{ifp}}\left(\mathbf{x}^{j_{k}}\mid\bm{\phi}^{i+1},\mathbf{x}^{\leq i},\mathbf{a}^{\leq i},\ell\right),\qquad\mathcal{L}_{\mathrm{ifp}}=\sum_{k=1}^{K}w_{k}\mathcal{L}_{\mathrm{fm}}\left(\mathbf{x}^{j_{k}};\bm{\phi}^{i+1},\mathbf{x}^{\leq i},\mathbf{a}^{\leq i},\ell\right)$$One design choice deserves emphasis: the IFP modules are not directly conditioned on the human video $\mathbf{h}$. Their only access to its task information is through $\bm{\phi}^{i+1}$, which is extracted after the main robot-video Transformer has interacted with the in-context human video. If the IFP modules could attend to $\mathbf{h}$ directly, the auxiliary branch could learn an independent human-video-conditioned future predictor without forcing the main branch to encode the task information — and since IFP modules are removed at inference, the deployed policy would gain nothing. Conditioning only on $\bm{\phi}^{i+1}$ makes the future loss supervise the main branch's current robot-video representation. Predicting multiple future chunks in parallel with temporal stride raises the prediction difficulty without introducing temporal dependence among future predictions during training. The following diagram traces the paper's actual data flow — the main causal prediction path, the IFP auxiliary path, and how the human-video signal is forced through the main branch:
flowchart TD H["Human video h
(ICL task prompt)"] -->|RoPE height offset| V L["Language instruction l"] --> V XR["Robot history x^<=i, a^<=i"] --> V V["Video Transformer (30 layers, MoT)
predicts x^(i+1) by flow matching"] --> A A["Action Transformer
inverse dynamics, no direct h attention"] --> ACT["Action chunk a^(i+1)"] V -->|"multi-layer features r_m"| FUSE["Concat + MLP P_fuse"] FUSE --> IFP["IFP modules G_1..G_4
strided future chunks x^j_k
(training only, removed at inference)"] IFP -.->|"gradient forces main branch
to encode task evolution from h"| V
4. Training Objectives and Inference
For each target next video chunk $\mathbf{x}^{i+1}$, noise $\bm{\epsilon}^{i+1}$ and flow time $t$ are sampled to form $\mathbf{x}^{i+1}_{t}=(1-t)\mathbf{x}^{i+1}+t\bm{\epsilon}^{i+1}$ with target velocity $\mathbf{v}^{\star,i+1}_{t}=\bm{\epsilon}^{i+1}-\mathbf{x}^{i+1}$; actions are treated analogously with noise $\bm{\epsilon}^{i+1}_{a}$ and flow time $r\in[0,1]$. The two data families get different losses. Task-diverse VA samples (language-only condition):
$$\mathcal{L}_{\mathrm{VA}}=\mathbb{E}\left[\mathcal{L}_{\mathrm{fm}}^{i+1}(\ell)+\lambda_{a}\mathcal{L}_{a}^{i+1}(\ell)\right]$$HumanGen samples (video conditioned on $c=\{\mathbf{h},\ell\}$, action loss still conditioned on $\ell$ since the action branch never sees the human video), augmented with IFP:
$$\mathcal{L}_{\mathrm{ICL}}=\mathbb{E}\left[\mathcal{L}_{\mathrm{fm}}^{i+1}(c)+\lambda_{a}\mathcal{L}_{a}^{i+1}(\ell)+\lambda_{\mathrm{ifp}}\mathcal{L}_{\mathrm{ifp}}\right]$$Implementation follows LingBot-VA: the video Transformer keeps the Wan-2.2 backbone (hidden dimension $d_{v}=3072$, 30 layers); the action Transformer has $d_{a}=3072$ and is initialized from the video branch. The IFP module predicts $K=4$ future chunks with stride $s=2$ and loss weights $(w_{1},\dots,w_{4})=(0.5,0.25,0.15,0.15)$. AdamW with peak learning rate $1\times10^{-4}$ and weight decay 0.01. Pre-training draws Task-diverse VA and HumanGen at a sampling ratio of 1:5; for ICL samples the human-video latent is dropped with probability 0.1 and the language-instruction dropout is raised from Wan-2.2's default 0.1 to 0.4, reducing dependence on language when the model learns from human video instructions. Robot video chunk size is randomly sampled from 1 to 4; each GPU packs samples of different token lengths into one sequence with attention masks, keeping the maximum token length per GPU within 160K. Pre-training costs 15,360 GPU hours. At inference, IFP modules are removed. Language-only mode conditions on $c=\ell$; ICL mode encodes the human video once, caches its tokens as prefix memory, and disables the language instruction. Both modes use classifier-free guidance scale 5 for video, action CFG scale 1.0, and a fixed inference chunk size of 2.
Experiments and Results
RoboTwin 2.0 Zero-Shot Cross-Task Evaluation
The simulation benchmark is RoboTwin 2.0, which contains 50 bimanual manipulation tasks with 50 trajectories each; for every trajectory the paper's pipeline generates a corresponding human video instruction, forming the Simulation ICL subset. The evaluation splits at the task level: 43 tasks for post-training, 7 held out as unseen (place object on scale, stamp seal, open microwave, move stapler to pad, place bread in basket, place empty cup, stack blocks three). These cover unseen-object pick-and-place, unseen articulated-object manipulation, and unseen long-horizon manipulation. Post-training runs 4,000 steps on 64 GPUs, jointly sampling Task-diverse VA, HumanGen, and RoboTwin data at a ratio of 2:10:3. Two baselines under the same cross-task protocol: WAN-Action (Wan-2.2-TI2V-5B + the same MoT causal framework, trained only on the 43 seen tasks) and LingBot-VA (released pre-trained checkpoint post-trained on the same seen tasks). Evaluation uses 3 seeds × 100 closed-loop rollouts per unseen task.
Figure 3: The seven unseen RoboTwin 2.0 tasks, spanning unseen-object manipulation, articulated-object manipulation, bimanual manipulation, and long-horizon manipulation.
| Task | WAN-Action | LingBot-VA | Zero-WAM |
|---|---|---|---|
| Place object on scale | 3.00 ± 2.16 | 6.17 ± 4.87 | 24.67 ± 2.05 |
| Stamp seal | 7.33 ± 1.25 | 3.67 ± 2.49 | 47.00 ± 4.55 |
| Open microwave | 2.26 ± 1.60 | 29.33 ± 10.66 | 59.00 ± 2.83 |
| Move stapler to pad | 10.67 ± 1.70 | 23.33 ± 8.22 | 69.14 ± 2.93 |
| Place bread in basket | 15.26 ± 2.55 | 17.33 ± 6.18 | 35.00 ± 3.74 |
| Place empty cup | 38.33 ± 2.05 | 42.33 ± 7.85 | 84.87 ± 0.18 |
| Stack blocks three | 0.00 ± 0.00 | 0.00 ± 0.00 | 9.00 ± 2.16 |
| Average | 10.98 ± 1.07 | 17.45 ± 1.40 | 46.95 ± 0.72 |
Table 2 (paper): zero-shot cross-task success rates (%) on seven unseen RoboTwin tasks, macro-averaged over three seeds.
Three observations from the table. First, the gain is consistent: Zero-WAM wins on all seven tasks, with the average jumping from 17.45% (LingBot-VA) to 46.95% — an absolute gain of 29.50 points, and 35.97 points over WAN-Action. Second, the largest margins appear exactly where task dynamics must be inferred from the specification: open microwave (unseen articulated object, 2.26% → 59.00%) and stamp seal (unseen rare-object relocation, 7.33% → 47.00%). Third, stack blocks three — the hardest long-horizon task — stays at zero for both baselines, while Zero-WAM is the only method to achieve non-zero success (9.00%). The three methods form an ordered comparison of what pre-training buys you: WAN-Action adapts only the generic Wan-2.2 video prior; LingBot-VA adds robotic video-action pre-training; Zero-WAM further adds task-balanced robotic data and human-robot ICL pairs.
Real-World Evaluation on a Bimanual Franka
Three task families are evaluated on a real bimanual Franka robot, each holding out the tested configurations from training. Zero-WAM conditions on the human video instruction alone at test time; LingBot-VA receives detailed textual task descriptions. Each task is run for 30 real-robot trials.
| Task family | Train combos | Train demos | LingBot-VA | Zero-WAM |
|---|---|---|---|---|
| Object-to-container placement | 30 | 120 | 43.3 | 53.3 |
| Three-object sequential manipulation | 16 | 96 | 10.0 | 33.3 |
| Two-table-leg insertion | – | 36 | 0.0 | 16.7 |
Table 3 (paper): real-world unseen-configuration success rates (%) over 30 trials per task. LingBot-VA uses language instructions; Zero-WAM uses human video instructions.
In object-to-container placement, every test configuration contains at least one unseen object or container, and Zero-WAM reaches 53.3% against 43.3%. The three-object sequential task randomizes the order of three objects shown in the human video; Zero-WAM reaches 33.3% against 10.0%. The insertion task is the most demanding: with only 36 demonstrations, the human video specifies which colored table leg goes into which target hole — Zero-WAM achieves 16.7% while LingBot-VA gets zero, and the authors additionally observe qualitative transfer to unseen leg colors and base types, indicating the fine-grained specification in the human video is actually being followed.
Figure 4: Qualitative real-world evaluations of Zero-WAM with human video instructions.
Ablations: Disentangling ICL, IFP, and Task-Balanced Pre-Training
All ablations reuse the RoboTwin 43-seen / 7-unseen split for reliable large-scale closed-loop comparison. Three factors are isolated. (1) Human video ICL. Training all variants exclusively on the 43 seen tasks, adding human video instructions raises the average from 10.98% (WAN-Action) to 36.36% (Zero-WAM without pre-training) — a model with no robotic pre-training at all beats LingBot-VA's 17.45%, showing the human video carries task information beyond both text and generic video-action pre-training. But all three seen-task-only variants score zero on stack blocks three, whereas the full model scores 9.00%: small-scale ICL data cannot support unseen long-horizon tasks, which is precisely why large-scale task-diverse ICL pre-training matters. (2) IFP. Adding the future-chunk objective improves the seven-task average from 28.55% to 46.95%, with especially large gains on open microwave and stamp seal, and it breaks the zero barrier on stack blocks three (0.00% → 9.00%). IFP is what elicits the human-video-following capability the ICL data teaches. (3) Task-balanced robotic data. A text-only Zero-WAM variant (ICL samples included but the human-video condition masked, reducing each pair to an ordinary text-conditioned sample) reaches 39.44%, beating LingBot-VA by 21.99 points — since most of the task-sampled robotic data is already covered by LingBot-VA's full pre-training corpus, this gap is attributable to task-level re-partitioning and balanced sampling rather than more data.
Figure 5: Effect of in-context human video prompts on RoboTwin unseen tasks (success rates and macro average across variants).
Related Work and Positioning
Zero-WAM sits at the intersection of three research lines. First, cross-task manipulation: vision-language-action (VLA) models scale language-conditioned policies over heterogeneous robot data, but a persistent gap remains between the vision-language representation space and the robot action space, so cross-task transfer lags behind visual generalization. The AGNOSTOS benchmark showed representative VLA policies fail at zero-shot cross-task execution; its workaround uses a robot trajectory from the unseen task as in-context guidance, which raises test-time cost. Zero-WAM instead uses a human video, which costs nothing to obtain per task. Second, world-action models: rather than mapping language directly to actions, WAMs jointly predict future visual states and executable actions, shifting the generalization problem from action generalization to unseen-video generation. LingBot-VA developed autoregressive causal video-action policies; DreamZero evaluated WAMs under visual-domain shifts; EgoWAM extended the view to egocentric human data. A recent thread (WAM-TTT, RoboTTT) enables unseen tasks through test-time training — updating memory or fast weights at deployment. Zero-WAM deliberately avoids test-time adaptation: human-robot ICL data and task-balanced video-action data go into pre-training, so the policy follows in-context human videos directly at test time with no parameter updates. Third, human video data: prior work uses in-the-wild human videos for representation learning or mixes them with robot data, or conditions policies on human demonstrations as task prompts — but those systems rely on manually collected human demonstrations, restricting them to narrow task families. Zero-WAM targets the missing regime: it automatically generates HumanGen from task-sampled robot data, so human video instructions scale together with task-diverse robot dynamics instead of being collected task by task.
Limitations
First, the authors themselves acknowledge that the experiments focus on stationary tabletop manipulation; extending the paradigm to mobile manipulation, dynamic environments, and substantially longer-horizon tasks remains future work — and the simulation results already hint at the difficulty, with stack blocks three capped at 9.00%. Second, the absolute real-world numbers are modest (53.3% / 33.3% / 16.7%), and the precise-insertion family is explicitly limited by the physical difficulty of insertion; human-video specification improves relative performance but does not yet make fine manipulation reliable. Third, from this reader's perspective: the entire interface depends on the quality of generated human videos — the pipeline's VLM quality control is a filter, not a guarantee, and the paper does not report what fraction of generated videos is rejected or how sensitive final performance is to generation quality; HumanGen's in-house subset (30K pairs) is also not fully characterized. Fourth, inference remains expensive: each control step runs flow-matching video generation with CFG scale 5 before action decoding, and the paper reports no closed-loop control frequency — a real question for deployment.
Conclusion and Outlook
Zero-WAM's contribution is best read as a data-interface thesis: zero-shot cross-task generalization is not solved by yet another architecture trick, but by giving the policy a task specification rich enough to carry unseen dynamics (human video), plus a scalable mechanism to manufacture such specifications at task-level diversity (the HumanGen pipeline + task-balanced curation), plus a training objective that prevents the model from ignoring them (IFP). The evidence is unusually clean: a held-out task split with consistent gains across all seven unseen tasks, real-world transfer to unseen configurations with only dozens of seen-task demos, and ablations that attribute the gain to each of the three ingredients separately. If code, model, and data land as planned before mid-September 2026, the community gets both a strong baseline and — more importantly — a reproducible recipe for manufacturing human-robot ICL data at scale. The questions that will decide how far this goes: can generated human videos keep their semantic fidelity as task difficulty grows, does the video-generation inference cost fit inside a real-time control loop, and does the "video as instruction" interface extend to mobile manipulation and genuinely long-horizon tasks? What this paper has already shown is that the bottleneck was never the absence of human videos — it was the absence of a scalable way to turn them into grounded, task-aligned robot supervision, and HumanGen is a concrete, reproducible answer to that problem.
Golden Quote
"Language tells you the name of the task; a human video shows you its soul — Zero-WAM teaches the robot to act on the soul, instead of guessing the next move from memory."
