Asynchronous Multimodal Diffusion Policy Composition via Latency-Aware Guidance Fusion

Background and Motivation

Diffusion policies have shown strong potential for robotic imitation learning, with recent extensions incorporating force/tactile, audio, and other modalities to improve manipulation performance. However, these modalities differ not only in information content but also in sensing rates and inference latencies—force and audio signals are lightweight and high-frequency, while vision is computationally heavy and low-frequency. Existing multimodal diffusion policies typically rely on synchronous fusion or manually designed multi-frequency architectures, the former slowing all modalities to the slowest's frequency (wasting high-frequency modalities' real-time capability), the latter limiting extensibility to new modality combinations. Force feedback is highly time-sensitive—if the policy cannot react promptly to contact changes, force feedback's value is greatly diminished. This suggests lightweight high-frequency modalities can only fully express their control capability when executed near their native frequency.

This paper proposes LAG-Fusion, a latency-aware guidance fusion framework for asynchronous multimodal diffusion policy composition. LAG-Fusion allows modality-specific policies to operate at their native inference rates and contribute denoising guidance whenever available. To make asynchronous composition consistent, a reference-frame rebasing rule for diffusion variables under relative action representations is derived, enabling delayed guidance to be aligned before fusion. LAG-Fusion is instantiated in contact-rich manipulation by composing a low-frequency vision policy with a high-frequency force policy, validated on Flip and Polish Curve tasks. LAG-Fusion achieves 60% success on Flip (vs synchronous 25%) and 55% Score on Polish Curve (vs RDP 45%), at 25Hz inference (vs synchronous 5Hz).

From a technical trajectory perspective, multimodal robot policy fusion has evolved from "input-level fusion" to "feature-level fusion" to "policy-level fusion." Early methods concatenated raw inputs from all modalities into a single network, requiring all modalities simultaneously available at consistent frequencies. Feature-level fusion merges after encoding each modality via attention or concatenation but still requires synchronization. LAG-Fusion belongs to policy-level fusion—each modality has an independent policy, composed through guidance signals during diffusion denoising. This design's advantage is modality decoupling: adding a new modality only requires training the corresponding policy and adding it to guidance fusion, without retraining the entire network. Diffusion policy's composition property provides a natural foundation—diffusion's linear superposition allows multiple denoising guidance signals to be weighted and fused without degrading generation quality. However, asynchronous composition introduces new challenges: different modality policies generate guidance at different times with potentially changed reference frames requiring precise alignment. This is the core problem LAG-Fusion's rebasing rule solves.

Method: Latency-Aware Guidance Fusion

Diffusion Policy and Guidance Fusion

Diffusion policies generate action sequences via iterative denoising. Given noised action:

$$a_k = \sqrt{\bar{\alpha}_k} a_0 + \sqrt{1 - \bar{\alpha}_k} \epsilon$$

($\epsilon \sim \mathcal{N}(0, I)$), modality-specific policies predict denoising directions: vision $\epsilon_k^V = \pi^V(a_k, k; c^V)$, force $\epsilon_k^F = \pi^F(a_k, k; c^F)$. Fused guidance is the weighted average:

$$\epsilon_k = \frac{w^V}{w^V + w^F} \epsilon_k^V + \frac{w^F}{w^V + w^F} \epsilon_k^F, \quad w^V, w^F \geq 0$$

Then update via DDIM step $a_{k-1} = \text{DDIMStep}(a_k, \epsilon_k, k)$. Weights use a curriculum schedule—vision weight decays linearly from $w_{\text{start}}^V = 0.8$ to $w_{\text{end}}^V = 0.2$, force weight $w_i^F = 1 - w_i^V$, $r_i = i / I_{\text{max}}$ ($I_{\text{max}} = 50$):

$$w_i^V = w_{\text{start}}^V + r_i (w_{\text{end}}^V - w_{\text{start}}^V)$$

This schedule makes vision guidance dominate early denoising (coarse planning) and force guidance dominate later (fine contact adjustment), matching the two-phase nature of contact-rich manipulation.

Reference-Frame Rebasing Rule

The core challenge of asynchronous composition is aligning delayed guidance. Suppose the vision policy generates guidance at time $t_1$ in frame $T_1$, and the force policy at later time $t_2 > t_1$ in frame $T_2$. Using relative action representations (actions relative to current end-effector pose), the two policies' reference frames differ. We need to rebase diffusion variables from $T_1$ to $T_2$. The relative transform is $T_{2 \leftarrow 1} = T_2^{-1} T_1 = (R_{2 \leftarrow 1}, p_{2 \leftarrow 1})$. Actions represented as $a = [x^\top, R_{\text{6d}}^\top, g]^\top \in \mathbb{R}^{10}$ (3D position + 6D rotation + 1D gripper), the rebasing transform is:

$$a' = A_{2 \leftarrow 1} a + b_{2 \leftarrow 1}$$

where $A_{2 \leftarrow 1} = \text{blockdiag}(R_{2 \leftarrow 1}, R_{2 \leftarrow 1}, R_{2 \leftarrow 1})$, $b_{2 \leftarrow 1} = [p_{2 \leftarrow 1}^\top, 0, 0]^\top$. The key derivation is rebasing the diffusion variables themselves—by estimating the clean action:

$$\hat{a}_0 = \frac{1}{\sqrt{\bar{\alpha}_k}} a_k - \frac{\sqrt{1 - \bar{\alpha}_k}}{\sqrt{\bar{\alpha}_k}} \epsilon_k$$

and transforming then re-noising, we get the rebased diffusion variables:

$$a'_k = A_{2 \leftarrow 1} a_k + \sqrt{\bar{\alpha}_k} b_{2 \leftarrow 1}, \quad \epsilon'_k = A_{2 \leftarrow 1} \epsilon_k$$

This ensures delayed guidance is aligned to the current frame before fusion, making asynchronous composition mathematically equivalent to synchronous but unconstrained by the slowest modality's frequency.

The mathematical derivation of reference-frame rebasing deserves deeper understanding. In relative action representation, action $a = [x^ op, R_{ ext{6d}}^ op, g]^ op$ has position $x$ as offset relative to current end-effector pose, rotation $R_{ ext{6d}}$ as 6D rotation representation (first two rows of rotation matrix, 6 components), and gripper $g$ as scalar. When the reference frame changes from $T_1$ to $T_2$, position offset needs rotation $R_{2 \leftarrow 1}$ and translation $p_{2 \leftarrow 1}$, rotation needs left-multiplication by $R_{2 \leftarrow 1}$, gripper unchanged. The key is rebasing diffusion variables $(a_k, \epsilon_k)$—applying affine transform $a'_k = A_{2 \leftarrow 1}a_k + \sqrt{ar{lpha}_k}b_{2 \leftarrow 1}$ to $a_k$ is derived by decomposing $a_k = \sqrt{ar{lpha}_k}a_0 + \sqrt{1-ar{lpha}_k}\epsilon$, transforming $a_0$ and $\epsilon$ separately then recombining. The noise term transform $\epsilon'_k = A_{2 \leftarrow 1}\epsilon_k$ has no translation since noise is zero-mean. This precise variable-level alignment ensures delayed guidance is not approximately but mathematically exactly aligned—the fused denoising direction is consistent with synchronous composition (in the reference frame transformation sense), only with more flexible generation timing.

flowchart TD
    V["Vision policy π^V
low-freq 2-5Hz"] -->|"ε^V_k at t₁, T₁"| RB["Reference-frame rebasing
T₂←₁ = T₂⁻¹T₁"]
    F["Force policy π^F
high-freq 25Hz"] -->|"ε^F_k at t₂, T₂"| FUSE["Latency-aware fusion
ε = w^V·ε^V + w^F·ε^F"]
    RB -->|"ε'^V_k aligned to T₂"| FUSE
    FUSE --> DDIM["DDIM step
a_{k-1} = DDIMStep(a_k, ε_k)"]
    DDIM --> ACT["Action output
25Hz control frequency"]

Experimental Results

Tasks and Setup

Evaluated on two contact-rich tasks: Flip (flipping a box, requiring visual localization + force feedback for flipping) and Polish Curve (wiping a curve, requiring visual tracking + force control for contact). 50 demonstrations per task collected via arm-to-arm teleoperation. Vision policy uses RISE, force policy is a lightweight MLP. Baselines include Training-Time Noise Fusion, Synchronous Composition, Policy Consensus, TA-VLA, and RDP.

Main Results

LAG-Fusion achieves 60% on Flip-Flip (vs synchronous 25%, Policy Consensus 40%) and 55% on Polish-Score (vs RDP 45%). Inference frequency of 25Hz far exceeds synchronous methods' 5Hz, proving asynchronous composition releases the real-time control capability of high-frequency force modality. Training Noise Fusion performs worst (Flip-Flip 17.5%) because forced temporal alignment during training damages force signal temporal information. Policy Consensus's learned router may overfit to the dominant modality causing imbalanced modality usage.

Figure 1: LAG-Fusion framework—asynchronous multimodal diffusion policy composition

Figure 2: Contact-rich manipulation tasks—Flip and Polish Curve

Figure 3: Effect of inference frequency on performance

From technical interpretation of experimental results, several key comparisons deserve deeper analysis. First, Training Noise Fusion (Flip-Flip 17.5%) performs even worse than vision-only (20.0%), showing that forced alignment of high-frequency force with low-frequency vision during training is not just unhelpful but harmful—force signal temporal information is destroyed, and the learned fusion produces erroneous guidance at test time. Second, Synchronous Composition (25% at 5Hz) shows limited improvement over vision-only (20%), indicating force feedback's real-time capability is severely weakened at 5Hz—reducing 25Hz force to 5Hz means 80% of force information is discarded. Third, RDP (24Hz, 35%) despite near-LAG-Fusion inference frequency still underperforms by 25%—RDP uses a single policy for all modalities, inter-modality coupling may disturb learning. Fourth, LAG-Fusion's (25Hz, 60%) significant advantage comes from two sources: force policy runs at native 25Hz fully expressing real-time control, and curriculum guidance fusion weights let vision and force dominate different denoising stages. Policy Consensus (40%) though better than synchronous still underperforms LAG-Fusion, showing learned router's fixed weights are less flexible than curriculum scheduling.

Training and Implementation Details

The curriculum weight schedule design reflects deep understanding of contact-rich manipulation's two-phase nature: early denoising (small $i$, small $r_i$) has vision weight $w^V = 0.8$ dominating, as visual planning of macro trajectories is needed (approaching object, locating flip point); late denoising (large $i$) has force weight $w^F = 0.8$ dominating, as force feedback is needed for fine contact force and direction adjustment. This "vision first, force later" curriculum aligns with human manipulation intuition—look first, then apply force. The force policy uses lightweight MLP [32, 64] ensuring high-frequency inference feasibility—25Hz means completing a denoising step every 40ms, heavy networks cannot meet this. The vision encoder reduces from 600 to 16-dimensional features, dramatically reducing computation.

Frequency Analysis

Frequency analysis shows force feedback policies achieve significantly higher success rates at higher inference frequencies—confirming force feedback's time sensitivity, where value depends on whether the policy can react promptly to contact changes. Lightweight high-frequency modalities can only fully express their control capability near their native frequency. This fundamentally motivates asynchronous composition: rather than slowing all modalities to the slowest's frequency, each modality should operate at its native rate and contribute guidance when available.

Limitations and Future Work

Limitation 1: Limited modality count. Only vision+force composition is validated; asynchronous composition with more modalities (tactile, audio) remains unverified. Rebasing and fusion complexity may grow with more modalities.

Limitation 2: Relative action representation constraint. The rebasing rule depends on relative action representations; absolute representations need different alignment strategies. Task diversity is limited (two contact-rich tasks); broader manipulation scenario validation is needed.

Summary and Insights

LAG-Fusion proposes a latency-aware guidance fusion framework for asynchronous multimodal diffusion policy composition. Key technical contributions include: the guidance fusion formula $\epsilon_k = \frac{w^V}{w^V+w^F}\epsilon_k^V + \frac{w^F}{w^V+w^F}\epsilon_k^F$ with curriculum weight scheduling (vision 0.8→0.2, force inverse); the reference-frame rebasing rule $a' = A_{2 \leftarrow 1}a + b_{2 \leftarrow 1}$ and diffusion variable rebasing $a'_k = A_{2 \leftarrow 1}a_k + \sqrt{\bar{\alpha}_k}b_{2 \leftarrow 1}$, $\epsilon'_k = A_{2 \leftarrow 1}\epsilon_k$ aligning delayed guidance before fusion; asynchronous composition allowing each modality to run at native frequency unconstrained by the slowest. On Flip and Polish Curve, LAG-Fusion achieves 60%/55% success/score at 25Hz, significantly outperforming synchronous composition (25%/25%) and RDP (35%/45%). Frequency analysis confirms force feedback's time sensitivity—high-frequency execution is needed to fully leverage its control capability. This work provides a mathematically rigorous and practically useful framework for asynchronous composition of multimodal robot policies, with direct application value for contact-rich manipulation requiring vision+force coordination. The rebasing rule ensures mathematical consistency of asynchronous composition—delayed guidance is neither simply discarded nor blindly used, but aligned to the current frame through precise coordinate transformation before fusion, ensuring composed policy quality is unaffected by inter-modality delay differences.

From a broader perspective, LAG-Fusion embodies the trend of robot policy design shifting from "unified synchronous" to "heterogeneous asynchronous." Traditional multimodal policies assume all modalities are simultaneously available at consistent frequencies—simple in theory but wasteful of high-frequency modalities' real-time capability in practice. In real robot systems, vision processing typically takes 50-200ms (CNN/ViT inference), force/tactile reading only 1-5ms, audio about 10ms—forcing these heterogeneous-frequency modalities to sync to the slowest (vision's 2-5Hz) severely wastes force/tactile's potential 100-1000Hz control bandwidth. LAG-Fusion's asynchronous composition lets each modality run at native frequency, composing through guidance fusion during diffusion denoising—this design not only improves control frequency (25Hz vs 5Hz) but more importantly releases high-frequency modalities' control potential. For scenarios requiring fast force response (precision assembly, contact polishing, fragile object manipulation), high-frequency force feedback real-time capability is the key success factor. The curriculum weight schedule also provides natural task phase modeling—early denoising corresponds to planning phase (vision-dominated), late to execution phase (force-dominated), making fusion weight design physically grounded rather than arbitrary tuning. From engineering perspective, lightweight force policy MLP [32, 64] at 25Hz has ~1ms inference latency, far below 40ms control period, ensuring real-time. Vision policy though heavy runs at 2-5Hz with ~200ms per inference, also acceptable. This "light-heavy mix, each at its own pace" architecture design provides direct reference value for practical multimodal robot policy deployment.