PAPER DEEP DIVE
Dynamics-Aware Meta-Imitation for Generalization to Unseen Robotic Manipulation
Imitation Learning aims to learn skills from extensive observations and demonstrations for robots, so it suffers from data scarcity and environment generalization. The existing methods predominantly focus on imitation from in-domain tasks and consequently struggle with generalization to unseen tasks. To bridge this generalization gap, we propose the \textbf{D}ynamics-\textbf{A}ware \textbf{M}eta-\textbf{I}mitation (DAMI) framework. By integrating meta-learning to construct a shared skill space, DAMI equips agents for rapid adaptation to novel tasks. We introduce the Visual-Motor Trajectory (VMT) module to capture complex spatio-temporal dynamics within the task latent space. Furthermore, we propose the Unpaired Unified Task (U2T) block to fuse unstructured multimodal observations. To coordinate these representations, we integrate a Task-Conditioned Feature Modulation (TCFM) mechanism customized for modulating low-level 3D features. By capturing intrinsic dynamics from a random complete reference demonstration, our framework learns the underlying task logic rather than memorizing static cues, ensuring effective generalization. Extensive experiments in both simulation and real-world settings demonstrate that our approach outperforms state-of-the-art baselines regarding direct inference on seen tasks and adaptation to unseen tasks via few-shot fine-tuning.
DAMI: Dynamics-Aware Meta-Imitation for Generalization to Unseen Robotic Manipulation
Paper: Dynamics-Aware Meta-Imitation for Generalization to Unseen Robotic Manipulation | Authors: Zhenduo Shang, Xiyao Liu, Bohan Li et al. | Institutions: Shenyang Institute of Automation, CAS / Shenyang Univ. of Tech. | Link: https://arxiv.org/abs/2607.15880
One-Sentence Summary
DAMI combines a 3D diffusion policy with meta-learning, using a Visual-Motor Trajectory (VMT) module to capture spatio-temporal dynamics, an Unpaired Unified Task (U2T) module to fuse multimodal observations, and Task-Conditioned Feature Modulation (TCFM) to inject low-level 3D features, enabling rapid adaptation to unseen manipulation tasks from few demonstrations, significantly outperforming baselines on Meta-World and RLBench.
Background and Motivation
Imitation Learning (IL) acquires complex robotic manipulation skills from demonstrations, but policies learned from limited demonstrations suffer from distribution shift and compounding errors. Collecting additional demonstrations is costly. Many methods train specialized policies for individual tasks or use multi-task models that memorize training patterns, leading to poor generalization to out-of-distribution tasks.
Few-shot generalization provides a more data-efficient direction using prior knowledge. However, traditional meta-learning methods struggle to infer dynamics of novel tasks under large distribution shifts. In-Context Imitation Learning (ICIL) uses demonstrations directly as inference context, but these non-parametric methods rely on visual similarity rather than task dynamics or action semantics, limiting performance when tasks have different structures.
3D policy learning advances show that explicit 3D representations are essential for capturing spatial structure in unstructured environments. Diffusion policies have become a powerful paradigm for modeling multimodal action distributions. However, when only a few demonstrations are available, directly fine-tuning high-capacity 3D diffusion policies creates a mismatch between data availability and model capacity—the policy overfits support trajectories rather than learning transferable task dynamics.
Figure 1: Generalization gap in conventional methods. Standard approaches overfit seen data and fail on unseen tasks.
Method
DAMI couples an expressive 3D diffusion policy with a data-efficient meta-learning objective. The policy is conditioned on a text instruction, a complete reference demonstration, and the current observation, enabling task-specific behavior inference from sparse data. The architecture comprises three core modules: VMT, U2T, and TCFM.
Figure 2: DAMI Architecture overview. Given current observation, language instruction, and complete reference demonstration, DAMI predicts action trajectory using a meta-learned 3D diffusion policy.
Diffusion-based Meta-Imitation Formulation
DAMI adopts the Diffusion Policy paradigm. Given clean action trajectory $\mathbf{A}_0$, the forward diffusion process gradually perturbs it into noisy trajectory $\mathbf{A}_k$. The reverse process is implemented by a conditional U-Net. The diffusion loss is:
$$\mathcal{L}_{diff} = \mathbb{E}_{k, \mathbf{A}_0, \epsilon}\left[\left\|\mathbf{A}_0 - \hat{\mathbf{A}}_{0,\theta}(\mathbf{A}_k, k, \mathbf{f}_{obs}, \mathbf{H}_{demo}, \mathbf{f}_{text})\right\|^2\right]$$
To support rapid adaptation, this diffusion objective is integrated with the full MAML procedure. The inner loop adapts the current initialization $\theta$ on the support set:
$$\theta'_i = \theta - \alpha \nabla_\theta \mathcal{L}_{diff}(\theta, B_i^{sup})$$
The outer loop updates the shared initialization from the average query-set meta-gradient:
$$\bar{\mathbf{g}}_t = \nabla_\theta \frac{1}{|\mathcal{B}|} \sum_{\mathcal{T}_i \in \mathcal{B}} \mathcal{L}_{diff}(\theta'_i, B_i^{qry})$$
$$\theta \leftarrow \text{AdamW}(\theta, \bar{\mathbf{g}}_t; \eta_t)$$
The computation graph is retained through the inner-loop update, so the meta-gradient includes second-order derivative terms of full MAML. During evaluation, base tasks use direct inference with the EMA policy, while novel tasks are fine-tuned from the same EMA policy on few-shot demonstrations.
Visual-Motor Trajectory Module (VMT)
Purely visual demonstration encoding over-emphasizes static object appearance, insufficient for distinguishing manipulation intents like pushing, pulling, and pressing. VMT jointly models visual states and motor commands, capturing "what changes under what action" rather than relying on static visual cues. The reference demonstration trajectory $\tau = \{(\mathbf{f}_{v,t}, \mathbf{a}_t)\}_{t=1}^T$ comprises visual representations $\mathbf{f}_{v,t} \in \mathbb{R}^{D_v}$ and proprioceptive actions $\mathbf{a}_t \in \mathbb{R}^{D_a}$.
A learnable linear projection bridges the semantic gap by projecting concatenated features onto a shared latent manifold:
$$\mathbf{e}_t = \phi_{\text{proj}}([\mathbf{f}_{v,t} \oplus \mathbf{a}_t])$$
After adding learnable positional embeddings $\mathbf{P}$, a Transformer encoder processes the sequence to model global temporal dependencies:
$$\mathbf{H}_{demo} = \text{Transformer}(\mathbf{E} + \mathbf{P})$$
Figure 3: VMT module illustration. Jointly encodes visual features and motor actions to capture spatio-temporal dynamics.
Unpaired Unified Task Module (U2T)
U2T harmonizes unstructured text instructions, structured reference demonstrations, and current observations. "Unpaired" means the reference demonstration and current observation need not be temporally aligned—they share task semantics but may correspond to different states or execution phases. Modality-specific affine projections align demonstration and text tokens:
$$\mathbf{e}_m = \mathbf{W}_m \cdot \text{LN}(\mathbf{f}_m) + \mathbf{b}_m, \quad m \in \{demo, text\}$$
At diffusion step $k$, the timestep embedding $\mathbf{e}_k$ is concatenated with the current observation feature and transformed by the U-Net condition MLP:
$$\mathbf{h}_k = \text{MLP}([\mathbf{e}_k \oplus \mathbf{f}_{obs}])$$
The U2T input and first downsampling block condition are:
$$\mathbf{Z}_{in} = [\mathbf{e}_{demo} + \mathbf{P}_{demo} \| \mathbf{e}_{text} + \mathbf{P}_{text} \| \mathbf{h}_k]$$
$$\mathbf{C}_{\text{Down}_1} = \text{U2T}(\mathbf{Z}_{in}) = \text{TransformerEncoder}(\mathbf{Z}_{in})$$
Task-Conditioned Feature Modulation (TCFM)
The denoising network is a modified 1D conditional U-Net. Since early downsampling layers perform geometric grounding and local contact reasoning, U2T fusion is applied only at the first downsampling block; other blocks use the lighter observation-timestep condition. FiLM modulation is applied to the $j$-th U-Net block feature:
$$\tilde{\mathbf{h}}_j = \boldsymbol{\gamma}_j(\mathbf{c}_j) \odot \mathbf{h}_j + \boldsymbol{\beta}_j(\mathbf{c}_j)$$
where $\boldsymbol{\gamma}_j$ and $\boldsymbol{\beta}_j$ are learned projections producing modulation coefficients, with layer-specific condition $\mathbf{c}_j = \mathbf{C}_{\text{Down}_1}$ (when $j = \text{Down}_1$) or $\mathbf{h}_k$ (otherwise).
flowchart TB
A["Current observation Ot
point cloud + robot state"] --> B["Visual Encoder"]
B --> C["Observation feature f_obs"]
D["Text instruction l"] --> E["Frozen CLIP encoder"]
E --> F["Text feature f_text"]
G["Reference demo tau
200-frame trajectory"] --> H["VMT Module
visual+action joint encoding"]
H --> I["Demo tokens H_demo"]
C --> J["Condition MLP
h_k = MLP(e_k + f_obs)"]
I --> K["U2T Module
Transformer fusion"]
F --> K
J --> K
K --> L["TCFM Modulation
FiLM injection into U-Net"]
L --> M["3D Diffusion U-Net
predict action trajectory"]
M --> N["H-step action sequence"]
Experimental Results
Experiments are conducted on Meta-World (ML10 and ML45 protocols) and RLBench (FS25 setting). ML10 contains 10 training and 5 testing tasks, ML45 contains 45 training and 5 testing tasks, and RLBench FS25 trains on 25 base tasks and evaluates on 5 held-out tasks.
| Method | ML10 Base Avg | ML45 Base Avg | ML10 Novel Avg | ML45 Novel Avg |
|---|---|---|---|---|
| Meta-World | 39.60 | 7.93 | 29.33 | 12.53 |
| DP3 | 82.13 | 47.24 | 48.87 | 37.53 |
| Mamba | 72.17 | 50.90 | 35.93 | 17.00 |
| FreqPolicy | 80.77 | 45.87 | 46.47 | 13.47 |
| FlowPolicy | 78.03 | 45.85 | 27.53 | 13.07 |
| DAMI | 87.23 | 79.05 | 56.80 | 37.53 |
Table 1: Meta-World base and novel task success rates (%). DAMI leads significantly on both base and novel tasks.
DAMI achieves 87.23% on ML10 base tasks (vs DP3's 82.13%) and 79.05% on ML45 base tasks (vs DP3's 47.24%), a dramatic improvement. On novel tasks, DAMI achieves 56.80% on ML10 (vs DP3's 48.87%) and 37.53% on ML45 (matching DP3). On RLBench FS25 novel tasks, DAMI averages 10.80% (vs DP3's 9.93%), outperforming all baselines.
Ablation studies show that the VMT module's spatio-temporal dynamics are crucial for generalization—removing VMT significantly reduces novel task success. U2T's Transformer fusion is more effective than simple concatenation, and TCFM's FiLM modulation better preserves low-level 3D geometric information than direct condition injection. Real-world experiments validate DAMI's effectiveness on a physical robot, with adaptation taking less than two minutes.
Figure 4: DAMI meta-training algorithm. Inner-loop adaptation + outer-loop meta-gradient update + EMA maintenance.
| Parameter | Value | Description |
|---|---|---|
| Learning rate | 1e-4 | Adam optimizer |
| Batch size | 256 | Training batch |
| Discount $\gamma$ | 0.99 | Reward discount |
| Epochs | 5000 | Training iterations |
- Requires task-specific fine-tuning: Novel tasks still require a few demonstrations for fine-tuning, not fully zero-shot generalization. The authors note in the practitioner note "current limitations include the need for task-specific fine-tuning."
- Additional computational overhead: Full MAML's second-order derivative computation increases training cost. Compared to direct-inference baselines, DAMI's meta-training requires more computational resources and time.
- Limited evaluation task scope: Experiments are conducted only on limited task sets from Meta-World and RLBench. Reliability under broader object, sensing, and operating conditions needs validation. The authors note "evaluation on a limited set of manipulation tasks."
Conclusion and Outlook
DAMI demonstrates the feasibility of combining an expressive 3D diffusion policy with a data-efficient meta-learning objective. The VMT module captures task dynamics rather than static appearance by jointly encoding visual and motor information, U2T fuses multimodal observations in an unpaired manner, and TCFM injects task conditions into low-level 3D features via FiLM modulation. These three components synergistically reduce reliance on dense demonstrations and support rapid adaptation to novel tasks.
On Meta-World and RLBench, DAMI outperforms state-of-the-art baselines in both direct inference on base tasks and few-shot adaptation to novel tasks, with the most significant advantage in the large-scale ML45 setting (79.05% vs 47.24%). Real-world experiments further validate its practicality.
Golden insight: Understanding a task is not about remembering what objects look like, but about comprehending "what happens when you take a certain action"—DAMI lets robots learn dynamics logic from demonstrations rather than memorizing static frames.
SOURCE LINKS
