AeroAct: Action-Centered World-Action Models for Language-Conditioned Quadrotor Flight

AeroAct: Action-Centered World-Action Models for Language-Conditioned Quadrotor Flight

Authors: Xinhong Zhang, Qiyuan Zhu, Yubo Huang et al. (Beijing Institute of Technology) | arXiv:2607.14997v1 (2026-07-16) | Domain: cs.RO | Wan2.1-1.3B backbone | 8xA100 pretrain 36h

One-Sentence Summary

AeroAct is the first world-action model (WAM) demonstrated on a real quadrotor: it adapts a 1.3B video diffusion Transformer to predict dynamically executable fifth-order trajectory-action chunks from egocentric visual history, proprioception, and language, using future frames as dense consequence supervision during training but decoding actions directly without video generation at deployment, with a blockwise causal mask saving 37.8% inference time, and a DiffAero+Isaac+3DGS hybrid pipeline plus a handheld collection device solving the data bottleneck.

Background and Motivation

Aerial robots are increasingly expected to execute natural-language commands while flying safely through cluttered 3D scenes. This is substantially harder than standard VLN: the robot observes through a rapidly moving egocentric camera, its actions immediately alter the future visual stream, and unsafe commands quickly lead to oscillation, target loss, or collision. A practical language-conditioned flight policy must do more than identify the goal in an image—it must connect semantic grounding with a local, smooth, dynamics-compatible control representation.

Learning-based agile flight has shown RL and imitation learning can produce highly reactive policies, but these typically condition on geometric goals rather than open-vocabulary instructions. In parallel, VLA models improve semantic generalization by attaching action heads to large vision-language representations. But direct action decoding does not explicitly teach the policy how first-person observations evolve under its own actions. World-model advances suggest predicting future observations provides a compact substrate for control, planning, and representation learning. World-action models (WAMs) jointly train visual consequence prediction and action generation, letting dense video supervision regularize the learned action distribution.

Bringing WAMs to quadrotor navigation faces two challenges. First, training requires large-scale multimodal trajectories with tightly synchronized egocentric video, language, proprioception, and dynamically feasible actions—far harder to collect than tabletop demonstrations. Second, video-centric WAMs are expensive at inference: autoregressive future-video generation increases latency, and small visual prediction errors accumulate and corrupt downstream action decoding. AeroAct's core design: retain visual consequence supervision during training, decode only actions without video generation at deployment.

Figure 1: AeroAct overview—the policy receives egocentric visual history, proprioception, and language, predicting chunked local trajectories; training uses paired flight actions and future visual observations from Isaac/DiffAero/3DGS/handheld data; deployment decodes actions with self-guidance executed through a trajectory controller.

Method

Problem Formulation

Language-conditioned aerial navigation is formulated as sequential prediction from first-person sensing. At time $t$ the robot observes RGB image $o_t$, proprioceptive state $s_t$, and language instruction $l$. AeroAct conditions on a temporally strided observation history:

$$\mathcal{O}_t^h = (o_{t-(h-1)\Delta},\, o_{t-(h-2)\Delta},\, \ldots,\, o_t)$$

where $h$ is the number of reference frames and $\Delta$ the temporal stride. The policy predicts an action chunk $a_{t:t+p-1} = (a_t, \ldots, a_{t+p-1})$ of length $p$, each compactly parameterizing a dynamically executable local trajectory segment. Following the WAM formulation, a unified model $g_\Theta$ handles both action prediction and visual consequence modeling:

$$(a_{t:t+p-1},\, c_t) \sim g_\Theta(\cdot \mid \mathcal{O}_t^h,\, s_t,\, l)$$

where $c_t$ is an action latent conditioning visual forecasting. Given the same context and $c_t$, the model predicts future observations:

$$(o_{t+\Delta},\, o_{t+2\Delta},\, \ldots,\, o_{t+K\Delta}) \sim g_\Theta(\cdot \mid \mathcal{O}_t^h,\, s_t,\, l,\, c_t)$$

In implementation $\Delta=3$, $p=24$, so each chunk contains $K=\lfloor p/\Delta \rfloor = 8$ future visual frames and 24 low-level action steps.

Model Architecture

Figure 2: Action-centered world-action Transformer. Visual history, proprioception, and language condition the action decoder; future-frame tokens serve as consequence supervision during training but are masked from action tokens and disabled at inference.

AeroAct adapts a 1.3B Wan video diffusion Transformer. To avoid per-frame redundancy, one frame is encoded/forecast every $\Delta$ steps. At training, a clip is encoded by the frozen video VAE into spatiotemporal latent tokens, divided after 3D RoPE into reference tokens $T_o$ and future tokens $T_f$. Proprioception $s_t$ and action chunks are projected by MLPs into state tokens $T_s$ and action tokens $T_a$; the instruction is encoded into language tokens $T_l$. Non-language tokens are concatenated as $T_t = [T_s; T_o; T_a; T_f]$, with language tokens as cross-attention context.

Blockwise causal mask preserves action-consequence ordering: state and reference tokens cannot attend to predicted tokens; action tokens attend only to state, reference, and action tokens; future visual tokens attend to all. Thus action prediction cannot leak from future frames, while future-frame reconstruction remains conditioned on the generated action chunk. At inference $T_f$ is omitted and only the action stream decoded; with 4 denoising steps, inference drops from 0.296s to 0.184s, saving 37.8%.

The training objective couples action prediction with visual consequence modeling. Let $\epsilon_a$ and $\epsilon_f$ be the target noise/velocity fields for action and future-frame latents; the total loss is:

$$\mathcal{L}(\Theta) = \mathcal{L}_{\text{act}} + \lambda_{\text{vis}} \mathcal{L}_{\text{vis}}$$

$$\mathcal{L}_{\text{act}} = \mathbb{E}\left[\|g_\Theta^a(T_s, T_o, T_a, T_l) - \epsilon_a\|_2^2\right], \quad \mathcal{L}_{\text{vis}} = \mathbb{E}\left[\|g_\Theta^f(T_s, T_o, T_a, T_f, T_l) - \epsilon_f\|_2^2\right]$$

The visual term shapes the representation and action prior during training; deployment relies on $g_\Theta^a$ alone. This separation is critical for flight: the model benefits from consequence supervision, but the closed-loop controller is not forced to wait for or trust generated future frames.

Hybrid Data Pipeline

Figure 3: Simulation data pipeline. DiffAero provides a shared dynamics/control interface; Isaac Lab improves visual fidelity; the 3DGS branch scales scene and object diversity at lower asset-engineering cost.

Ordinary video and embodied datasets lack the needed synchronized supervision. The data stack centers on DiffAero (fast quadrotor dynamics and flight controllers), with two rendering branches sharing this control interface. The Isaac Lab branch provides realistic lighting, materials, shadows, and camera effects but limited scene diversity; the 3DGS branch renders RGB and depth directly from Gaussian-splat scenes, easily scalable but with lower close-range fidelity. The two are complementary.

Three classes of simulation demonstrations are collected: Isaac tracking (target moves through random corridors, expert controller keeps the quadrotor close, teaching motion anticipation and visual servoing); Isaac reaching (language-specified object among distractors, linking language grounding to goal-directed flight); 3DGS reaching (scene and object sampled and merged for object-approach demonstrations).

Real-world data collection: a handheld device with a fisheye camera, Intel RealSense T265, and lightweight computer estimates odometry via T265, converts motion to simulated quadrotor poses, and crops fisheye images into first-person views consistent with quadrotor camera motion. 858 real-world trajectories, 332,429 frames (~3 hours) are collected.

Self-Guidance for Smooth Trajectories

Figure 4: Inference-time self-guidance—consecutive action chunks overlap in time; the new chunk's prefix is guided by the previous chunk's suffix, reducing discontinuities before sampling the remaining horizon.

Diffusion/flow-matching models represent multimodal action distributions, but independently sampled chunks may differ across neighboring replanning calls, causing reference-trajectory jitter. Self-guidance makes consecutive chunks agree over their overlapping window: when inference $n+1$ starts before chunk $n$ expires, the two overlap; during denoising the new sample's prefix velocity is replaced with the previous chunk's corresponding suffix, and only the remaining suffix is sampled normally.

Experimental Results

Backbone is Wan2.1-1.3B; proprioception/action encoders are MLPs with hidden sizes 128/256. Simulation and real demonstrations are collected at 30 Hz and downsampled 3x (one video frame per three actions). Nine reference frames correspond to 2.4s of visual context. Pretraining uses 900K simulation clips (500K tracking + 200K Isaac reaching + 200K 3DGS reaching), 320M frames, on 8xA100 for 36h. AdamW with learning rate $4.3 \times 10^{-5}$. Inference on a single RTX 5090 24G uses ~4500MB VRAM.

Simulation Results

Task	Ref Frames	Success ↑	Collision ↓
tracking	1	20.0	90.0
searching	1	90.0	10.0
tracking	9	100.0	0.0
searching	9	100.0	0.0
tracking (FT)	9	95.0	0.0
searching (FT)	9	100.0	0.0

Table 1: Isaac Lab closed-loop simulation (20 episodes). 9 reference frames lift tracking success from 20% to 100% and collision from 90% to 0%.

With only 1 reference frame, closed-loop tracking achieves just 20% success and 90% collision—confirming a single egocentric image is insufficient for estimating relative motion and producing safe trajectory chunks. 9 reference frames improve both tracking and searching to 100% success with zero collisions. For unseen target objects, success rises from 75% to 100% and average final distance drops from 3.819m to 1.983m:

Model	Ref Frames	Avg Final Dist	Success ↑	Collision ↓
AeroAct	1	3.819	75.0	25.0
AeroAct	9	1.983	100.0	0.0
AeroAct-FT	9	1.988	100.0	0.0

Table 2: Searching task with unseen target objects. After real-world fine-tuning, simulation performance barely degrades, indicating real adaptation does not substantially compromise the learned closed-loop policy.

Real-World Flight

Figure 5: Experimental hardware. (a) UAV platform with onboard sensing and offboard inference via LAN, trajectories tracked by OM-MPC; (b) handheld portable collection platform recording cropped camera observations and trajectory annotations.

The quadrotor platform is built on a modified OddityRC 35Pro frame with an Intel RealSense D435i camera and Radxa ROCK 5C onboard computer, tested in a lab with a motion-capture system. Inference latency is approximately $t_{\text{infer}} = 0.8$s. This is the first WAM-based real-world quadrotor flight demonstration—a learned model directly connects visual-language task specification, predictive action generation, and closed-loop quadrotor execution. The instruction "fly to the yellow foam mat" was successfully executed.

Figure 6: Real-world language-conditioned WAM flight—AeroAct generates feasible flight commands guiding the quadrotor toward the specified target object.

System Architecture Diagram

flowchart TB
  subgraph Input["Input"]
    OH["Visual history O_t^h
9 frames stride Delta=3"]
    ST["Proprioception s_t"]
    LANG["Language instruction l"]
  end
  subgraph Model["Wan2.1-1.3B Video DiT"]
    VAE["Frozen video VAE
spatiotemporal latent tokens"]
    ENC["MLP encode state/action tokens"]
    TXT["Text encoder language tokens"]
    CAT["Concat T=[Ts;To;Ta;Tf]
language cross-attention"]
    MASK["Blockwise causal mask
action no leak from future"]
    DECODE["Action decode g_a
inference omits Tf saves 37.8%"]
  end
  subgraph Train["Training"]
    VIS["Visual consequence L_vis
future-frame dense supervision"]
    ACT["Action prediction L_act
trajectory action chunks"]
    LOSS["L = L_act + lambda*L_vis"]
  end
  subgraph Data["Hybrid Data Pipeline"]
    DA["DiffAero dynamics"]
    IS["Isaac Lab visual realism"]
    GS["3DGS scene diversity"]
    HW["Handheld device 858 trajectories"]
  end
  subgraph Deploy["Deployment"]
    SG["Self-guidance smoothing
overlap prefix guidance"]
    MPC["OM-MPC trajectory tracking"]
    QUAD["Physical quadrotor"]
  end
  Input --> VAE --> CAT
  Input --> ENC --> CAT
  Input --> TXT --> CAT
  CAT --> MASK --> DECODE
  DECODE --> Train
  VIS --> LOSS
  ACT --> LOSS
  Data --> Train
  DECODE --> SG --> MPC --> QUAD

From a methodological perspective, AeroAct's "world model during training, action head at deployment" design is a key pragmatic refinement of the WAM paradigm. Traditional WAMs generate future video at inference then decode actions, introducing two risks: autoregressive video-generation latency (unacceptable for flight) and accumulating prediction errors corrupting control. The blockwise causal mask structurally ensures action tokens cannot "peek" at future-frame tokens, so future frames supervise and shape the action representation during training while action decoding at inference is entirely independent of video generation. This shifts the world model's value from "explicit prediction at inference" to "implicit regularization during training"—a more efficient use of representation learning.

The temporal-context ablation is striking. The jump from 1 to 9 frames (tracking 20%→100%, collision 90%→0%) is not incremental but a phase transition. This shows that in egocentric quadrotor flight, a single frame almost cannot estimate relative motion—target displacement across frames is the primary motion cue. Nine frames (2.4s) suffice for the model to infer ego-motion velocity from pixel displacement and predict safe local trajectories. This finding informs action-space design for aerial VLN: any single-frame policy lacking temporal context may face similar fundamental difficulties in fast-flight scenarios.

The hybrid data pipeline's complementary design reflects engineering wisdom. Isaac Lab is visually realistic but scene-limited; 3DGS is scene-diverse but lower in close-range fidelity; together they cover the visual-realism vs scene-diversity trade-off. The handheld device ingeniously sidesteps the danger and cost of real quadrotor collection—using T265 odometry to "translate" handheld motion into quadrotor poses and fisheye cropping to simulate first-person views. This "human-collects-for-robot" strategy obtains real-world visual diversity at low cost (858 trajectories, ~3 hours), providing critical distributional supplementation for sim-to-real transfer. That simulation performance barely degrades after real fine-tuning (tracking 95% vs 100%) shows this strategy effectively avoids catastrophic forgetting.

Self-guidance is necessitated by diffusion models' inherent properties. Diffusion/flow-matching models represent multimodal distributions; independently sampled neighboring chunks may draw from different modes, causing jumps. For ground robots this may be tolerable, but quadrotor low-level controllers require smooth reference trajectories—reference jumps cause attitude oscillation or instability. Self-guidance enforces overlap-prefix consistency while preserving multimodal expressiveness, a key engineering adaptation for migrating diffusion policies from tabletop to flight platforms.

Limitations

Real experiments limited to short indoor trajectories (author-acknowledged). The temporal context suffices for single-stage target reaching but not yet for complex instructions requiring multiple semantic sub-goals, recovery behaviors, or long-horizon memory.

Relies on offboard inference (author-acknowledged). The video-diffusion backbone's computational cost forces offboard inference (workstation via ZeroMQ); $t_{\text{infer}}=0.8$s latency limits high-speed agile flight scenarios.

Single trajectory-segment action structure assumed. The fifth-order trajectory parameterization ensures smoothness and dynamic feasibility but may limit expressiveness for extreme maneuvers (sharp turns, rolls).

Visual prediction used only during training without fully validating its value boundary. While ablation shows temporal context is key, the specific contribution of the $\lambda_{\text{vis}}$ future-frame supervision, and the gap between representations learned without test-time video generation versus true video-generating WAMs, is not deeply quantified.

Summary and Outlook

AeroAct presents an action-centered WAM for language-conditioned quadrotor flight: outputting smooth local trajectory chunks rather than discrete actions/waypoints/velocities, using future egocentric visual prediction as training-time consequence supervision, and efficiently decoding actions without test-time video generation. The blockwise causal mask decouples the action stream from visual consequences, saving 37.8% inference time and avoiding error accumulation from imagined frames. The DiffAero+Isaac+3DGS+handheld hybrid pipeline solves the multimodal synchronized data bottleneck.

From a broader perspective, AeroAct represents the aerial extension of the "learn a world model during training, use the action head at deployment" pragmatic paradigm. The value of a world-action model lies not in generating video at inference but in regularizing the action distribution via visual consequence supervision during training—shifting "understanding action consequences" from explicit MPC to implicit representation learning. Combined with dynamics-feasible trajectory parameterization and self-guidance smoothing, this bridges from video generation to physical flight. Future work toward onboard efficient inference and long-horizon multi-stage instruction following could make WAMs a universal policy paradigm for aerial embodied intelligence.

Golden Lines

"The value of a world-action model lies not in generating video at inference but in regularizing the action distribution via visual consequence supervision during training."

"The model benefits from consequence supervision, but the closed-loop controller is not forced to wait for or trust generated future frames—learn the world during training, use actions at deployment."