Skip to content
RobotWorld
Back to Papers

PAPER DEEP DIVE

PaperSimulation仿真

KAI: A Kinematic-Aware Interface for Data-Efficient Articulated Object Manipulation

Articulated object manipulation requires an understanding of kinematic structure that is difficult and costly to learn from robot demonstrations alone. We introduce the Kinematic-Aware Articulation Interface (KAI), a structured intermediate representation that captures the kinematic structure of articulated objects. By embedding interpretable geometric and kinematic priors into policy learning, KAI provides a strong inductive bias aligned with the underlying structure of articulated motion. This design effectively improves sample efficiency, with gains particularly pronounced in low-data regimes: across six simulation tasks, our method achieves an average success rate of 82.9%, matching or surpassing baseline performance while using only half the demonstration data. Our method also exhibits robust generalization to unseen backgrounds and visual distractors, transferring from a single clean training environment to cluttered real-world scenes. KAI's action-agnostic design further enables co-training with human interaction videos to enhance real-world robustness: under diverse visual distractions, our method with video co-training achieves over 70% average success rate.

Yaping Li, Zhaxizhuoma, Qiaojun Yu, Jia Zeng, Dahua Lin, Jiangmiao PangJuly 27, 20266 min read
中文

1. Overview: Kinematic-Aware Interface for Articulated Object Manipulation

KAI (Kinematic-Aware Articulation Interface), proposed in July 2026, is a structured intermediate representation that explicitly encodes the kinematic structure of articulated objects into policy learning. By embedding interpretable geometric and kinematic priors, KAI provides a strong inductive bias aligned with the underlying structure of articulated motion, effectively improving sample efficiency—achieving an average success rate of 82.9% across six simulation tasks while using only half the demonstration data to match or surpass baselines. The method also exhibits robust generalization to unseen backgrounds and visual distractors, transferring from a single clean training environment to cluttered real-world scenes. KAI's action-agnostic design further enables co-training with human interaction videos: under diverse visual distractions, the method with video co-training achieves over 70% average success rate.

KAI overview

2. Core Problem: Missing Kinematic Structure in Articulated Object Manipulation

Articulated objects—doors, drawers, laptops, microwaves—are ubiquitous in human environments. Despite their diversity in appearance and function, they share a fundamental property: motion is governed by kinematic constraints. A door rotates about a fixed hinge; a drawer translates along a constrained rail. Reliable manipulation therefore requires understanding which part of the object moves and in what direction. End-to-end imitation learning methods learn direct perception-to-action mappings without explicitly modeling kinematic structure, forcing the policy to infer joint mechanics implicitly from data. Under limited demonstrations, this leads to reliance on superficial visual cues and brittle generalization when backgrounds or appearances change. Prior intermediate representations like affordance maps, point flow, and grasp poses describe "where to interact" or "what motion should result" rather than directly encoding the underlying kinematic structure. KAI's core insight: the most effective inductive bias for articulated object manipulation is an interface that captures the kinematic constraints of articulated joints—explicitly representing kinematic structure itself.

3. Method: KAI Framework

3.1 Kinematic-Aware Articulation Interface

KAI represents the kinematic structure through $K$ interaction keypoints on the moving part. For each keypoint $i \in \{1,\ldots,K\}$, given observation at timestep $t$, KAI predicts future locations $\mathbf{l}_{t+1}^i, \ldots, \mathbf{l}_{t+N}^i$ and displacements $\mathbf{d}_t^i, \ldots, \mathbf{d}_{t+N-1}^i$ where $\mathbf{d}_{t+\delta}^i = \mathbf{l}_{t+\delta+1}^i - \mathbf{l}_{t+\delta}^i$, $\delta \in \{0,\ldots,N-1\}$. To ensure predicted keypoints respect kinematic constraints, a geometric loss regularizes the keypoint set $\mathcal{K}$:

$$\mathcal{L}_{\text{geo}} = \lambda_s \mathcal{L}_{\text{struct}} + \lambda_m \mathcal{L}_{\text{motion}} \tag{1}$$

where $\mathcal{L}_{\text{struct}}$ enforces isometric constraints preserving pairwise distances, and $\mathcal{L}_{\text{motion}}$ regularizes displacements by joint type. The structural loss penalizes changes in relative distances, preventing non-physical deformation:

$$\mathcal{L}_{\text{struct}} = \sum_{\delta=0}^{N-1} \sum_{i,j \in \mathcal{K}} \left|\|\mathbf{l}_{t+\delta+1}^i - \mathbf{l}_{t+\delta+1}^j\|_2 - \|\mathbf{l}_t^i - \mathbf{l}_t^j\|_2\right|^2 \tag{2}$$ KAI method architecture

3.2 Joint-Type Motion Constraints

For prismatic joints, motion is constrained to pure translation, implemented via a translation invariance prior minimizing displacement variance:

$$\mathcal{L}_{\text{motion}}^{\text{pris}} = \sum_{\delta=0}^{N-1} \sum_{i \in \mathcal{K}} \|\mathbf{d}_{t+\delta}^i - \bar{\mathbf{d}}_{t+\delta}\|_2^2 \tag{3}$$

where $\bar{\mathbf{d}}_{t+\delta} = \frac{1}{|\mathcal{K}|} \sum_{j \in \mathcal{K}} \mathbf{d}_{t+\delta}^j$. For revolute joints, motion is modeled as constrained rotation, with keypoints divided into stationary anchors $\mathcal{K}_a$ on the rotation axis and moving points $\mathcal{K}_m$:

$$\mathcal{L}_{\text{motion}}^{\text{rev}} = \sum_{\delta=0}^{N-1} \left(\sum_{a \in \mathcal{K}_a} \|\mathbf{d}_{t+\delta}^a\|_2^2 + \sum_{i \in \mathcal{K}_m} \|\text{proj}(\mathbf{l}_{t+\delta+1}^i) - \text{proj}(\mathbf{l}_{t+\delta}^i)\|_2^2\right) \tag{4}$$

where $\text{proj}(\cdot)$ is the orthogonal projection onto the rotation axis. Anchor displacements are penalized to zero; moving points' projections onto the axis remain constant.

KAI interface instantiation

3.3 Phased-Reasoning Architecture and Co-Training

The model adopts a "Perception-KAI-Action" phased-reasoning architecture. The perception module fuses 2D semantic features (DINOv2 extracting RGB semantic features projected onto point clouds) with 3D geometric features (sparse 3D encoder processing point clouds) via a spatial aligner into unified visual tokens. Language instructions are tokenized by CLIP; robot states are embedded by an MLP. KAI tokens attend exclusively to perceptual tokens to predict keypoint locations and displacements without action information access, then combine with perceptual tokens for the action decoder. Unidirectional attention masks enforce information flow. The total training loss combines KAI geometric loss with action imitation loss:

$$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{action}} + \lambda_s \mathcal{L}_{\text{struct}} + \lambda_m \mathcal{L}_{\text{motion}} \tag{5}$$

The co-training strategy leverages human interaction videos (HOI4D dataset) to enhance real-world robustness—KAI's action-agnostic design enables learning perceptual and kinematic representations from human videos without robot action labels.

4. Experiments

4.1 Simulation Results

Evaluated in Isaac Sim on six tasks (open/close drawer, door, laptop), 200 trials per task. KAI without video co-training achieves 82.9% average success rate, surpassing all baselines; with co-training, 84.6%.

MethodAvg SROpen DrawerOpen DoorOpen LaptopClose DrawerClose DoorClose Laptop
ACT58.544.552.567.539.054.093.5
Seer73.484.867.267.768.163.789.2
DP337.325.012.525.52.097.062.0
KAI82.986.754.581.782.392.5100.0
KAI (w/ video)84.686.866.085.081.888.899.2
Simulation experiment results table

4.2 Data Efficiency and Generalization Analysis

Evaluated under varying training data (100/200/400/800 demonstrations). KAI consistently outperforms Seer and DP3 at all data scales, with the most significant advantage at 100 demos. In generalization tests, without video co-training the average SR is 66.7% (background distraction) and 62.2% (object distraction); co-training improves these by +6.6%/+11.1%/+13.4% (none/background/object), exceeding 70%. DP3 drops from 33.3% to 20.0% under background changes, while KAI drops only 4.5 percentage points.

Data VolumeDP3SeerKAI
100 demoshighest
200 demoslowmidhigh
400 demosmidhighhighest
800 demosmidhighhighest
Data efficiency comparison Real-world experiment setup

5. Limitations

  • Short-horizon tasks: evaluation focuses on short-horizon manipulation; extending KAI to long-horizon sequential tasks is a future direction.
  • Single robot platform: real-world experiments use a single robot platform; validation across different platforms is pending.
  • Joint type coverage: only prismatic and revolute joints are covered; extension to more complex joint types (e.g., spherical) remains to be explored.

6. Conclusion

KAI embeds kinematic priors into policy learning as a structured intermediate representation for articulated object manipulation. It predicts future locations and displacements through $K$ interaction keypoints on the moving part, regularized by isometric constraints and joint-type motion constraints (prismatic/revolute), explicitly encoding the kinematic structure. The phased-reasoning architecture ensures KAI serves as a structured bridge between perception and action. In simulation, 82.9% average success rate matches baselines with half the data; in the real world, policies trained in a single clean environment transfer robustly to unseen backgrounds and distractors. The action-agnostic design supports co-training with human interaction videos, achieving over 70% success rate under visual distractions. The core insight: the key to articulated object manipulation is not "learning actions" but "understanding structure"—KAI explicitly encodes physical common sense like "doors rotate about hinges, drawers slide along rails" as geometric constraints, upgrading the policy from data-driven guessing to structure-driven reasoning.

flowchart TD
    A["RGB-D images + language instruction + robot state"] --> B["Perception: DINOv2 semantic + 3D point cloud geometry"]
    B --> C["Spatial aligner fuses -> unified visual tokens"]
    C --> D["KAI prediction: K keypoints future locations + displacements"]
    D --> E["Geometric regularization: isometric + joint motion constraints"]
    E --> F{"Joint type?"}
    F -->|Prismatic| G["Translation invariance: minimize displacement variance"]
    F -->|Revolute| H["Rotation constraint: anchors fixed + projection invariance"]
    G --> I["KAI tokens"]
    H --> I
    I --> J["Action decoder: generate robot actions"]
    C --> J
    K["Human interaction videos HOI4D"] --> L["Co-training: action-agnostic design"]
    L --> D
The key to articulated object manipulation is not "learning actions" but "understanding structure"—KAI explicitly encodes physical common sense like "doors rotate about hinges, drawers slide along rails" as geometric constraints, upgrading the policy from data-driven guessing to structure-driven reasoning.

Related Papers

Pre-training Visual Dexterity in Simulation

Pre-training Visual Dexterity in Simulation

Large-scale pre-training has made robot policy fine-tuning increasingly data-efficient, but this progress has largely been driven by datasets and embodiments built around simple parallel-jaw grippers. Dexterous, multi-fingered hands remain comparatively data-starved because real teleoperation is costly to scale, while human hand video is off-embodiment and requires lossy pose estimation and retargeting. We introduce Simulation Pre-training for Dexterity (SPD), a pre-training framework for dexterous manipulation that uses data entirely collected in simulation. In SPD, humans manipulate virtual objects inside a VR headset, enabling on-embodiment trajectories and robot-free collection. With the help of five operators, we collect 75 hours of multi-task dexterous manipulation over one week, and use it to pre-train a causal transformer on a sequence modeling objective. We study the benefits of simulation pre-training on real-world tasks by fine-tuning on 1-2 hours of physical demonstrations on a 56-DoF bimanual dexterous setup. We find that our approach outperforms training behavior cloning policies from scratch, showing that simulation teleoperation is a viable pre-training source for real-world dexterous manipulation. We perform ablation studies, measuring the benefits of history conditioning and short action chunks for reactive control.

灵巧操作灵巧手预训练Aug 16, 2026
FA-RDP: A Frequency-Adaptive Reactive Diffusion Policy for Contact-Rich Manipulation

FA-RDP: A Frequency-Adaptive Reactive Diffusion Policy for Contact-Rich Manipulation

In contact-rich manipulation, action multimodality and reactivity dominate different stages of a single episode. Before contact, multiple trajectories might be equally valid, making it important to preserve diverse action modes. After contact, geometric constraints and force limits narrow the solution space, while successful execution demands rapid responses to force feedback. However, standard diffusion policies use a fixed inference frequency and sampling steps throughout the episode, forcing a fundamental compromise: low-frequency, multi-step sampling better preserves pre-contact multimodality but responds slowly to force feedback, whereas high-frequency sampling improves reactivity but tends to collapse distinct pre-contact modes. To resolve this tradeoff, we present FA-RDP, a frequency-adaptive reactive diffusion policy. A shared multi-frequency visual-force Transformer predicts action chunks at both low and high frequencies, while a learned multimodality indicator dynamically selects multi-step low-frequency sampling before contact and one-step high-frequency sampling as action ambiguity decreases. We further introduce Manifold Consistency Distillation (MCD), which reparameterizes the diffusion network to predict actions on the robot action manifold while retaining DDPM-based residual supervision. Experiments on three contact-rich manipulation tasks show that FA-RDP achieves the highest success rate while preserving diverse pre-contact trajectory modes. Code and videos are available at https://fa-rdp.github.io.

PaperJul 30, 2026
PhiZero: A World Model Built Around Physical Language

PhiZero: A World Model Built Around Physical Language

We introduce PhiZero, a physical world model built around physical language, a compact discrete representation of world-state transitions. Existing physical world models typically predict future videos directly in pixel space, leaving the underlying world dynamics implicit within high-dimensional visual predictors. Motivated by humans' ability to abstract predictive structure from visual experience and organize it in natural language for explicit reasoning, we learn physical language from in-the-wild videos through self-supervision and use it to explicitly reason about how the physical world evolves. Accordingly, PhiZero adopts a reason-then-render paradigm: it first infers future world evolution as a physical-language sequence and then renders the inferred transitions into videos. Extensive experiments across generation and understanding benchmarks validate the ability of PhiZero to model physically coherent world evolution. We further show its potential for realistic and interactive world modeling, fine-grained action-conditioned simulation, and zero-shot motion transfer.

PaperJul 30, 2026
Booster Lab: A Data-Centric Pipeline for Learning Deployable Humanoid Locomotion Policies

Booster Lab: A Data-Centric Pipeline for Learning Deployable Humanoid Locomotion Policies

Humanoid robot motion learning requires not only task-oriented control policies but also physically feasible and natural behaviors that can be transferred to real robots. However, robot-feasible motion data are often scarce: raw human demonstrations may be incompatible with the robot morphology, open-source clips vary in quality, and simulation-collected robot trajectories still require feasibility checking. To address these challenges, we propose a data-centric training and deployment pipeline that integrates motion data curation, real-to-sim model adaptation, AMP-based reinforcement learning, and sim-to-real deployment. We validate the framework on the Booster T1 robot and further provide preliminary cross-platform validation on Booster K1.

步态优化人形机器人AMPJun 26, 2026