PAPER DEEP DIVE
CosFly-VLA: A Spatially Aware Vision-Language-Action Model for UAV Tracking
Dynamic target tracking is essential for Unmanned Aerial Vehicles (UAVs) operating in complex urban environments, where both the target and the camera viewpoint change continuously. Existing Vision-Language-Action (VLA) policies can track visible targets effectively, but their performance often degrades when buildings, vegetation, or roadside objects block the line of sight. During sustained occlusion, a policy may lose the target state, execute actions toward an incorrect region, and amplify this error through subsequent observations until re-acquisition becomes impossible. To this end, we present CosFly-VLA, a spatially aware VLA model that jointly grounds the target, estimates its visibility, and generates continuous flight actions through a structured prediction interface. To train this policy, we use a large-scale recipe over diverse data sources. Spatially Grounded Continued Pretraining (CPT) on a 500k mixed pool injects UAV-view depth, distance, and 3-D spatial reasoning. A three-stage Curriculum-based Supervised Fine-Tuning (SFT) process then specializes the tracker through multi-head warm-up followed by two-stage curriculum learning over natural and hard / long-occlusion data. Chain-of-Thought (CoT) training subsequently teaches recovery-oriented reasoning traces before structured answers. Finally, a closed-loop Reinforcement Learning (RL) stage optimizes tracking behavior with a multi-component reward covering stand-off tracking, grounding quality, collision avoidance, and task success. Relative to OpenVLA, CosFly-VLA-0.8B reduces open-loop Average Displacement Error (ADE) by 34.1% on seen-test and 35.3% on unseen-test. Closed-loop optimization improves Success Rate (SR) by 29.8% and 2.5%, respectively. These results demonstrate progress from visible-frame imitation toward spatially grounded action-closed-loop control, evaluated under a shared oracle state history.
One-line summary
CosFly-VLA extends UAV target tracking from "follow what you can see" to a closed-loop recovery problem where the policy must maintain a spatial hypothesis about an unseen target, suppress drift from its own previous actions, and re-acquire the target when it reappears — trained through four progressive stages of spatially grounded pretraining, curriculum SFT, chain-of-thought training, and closed-loop RL.
Abstract
Dynamic target tracking is essential for UAVs operating in complex urban environments, where both the target and the camera viewpoint change continuously. Existing Vision-Language-Action (VLA) policies can track visible targets effectively, but their performance often degrades when buildings, vegetation, or roadside objects block the line of sight. During sustained occlusion, a policy may lose the target state, execute actions toward an incorrect region, and amplify this error through subsequent observations until re-acquisition becomes impossible. CosFly-VLA formulates occlusion-robust UAV tracking as a closed-loop recovery problem: the policy must maintain a spatial hypothesis about an unseen target, predict where it may reappear, and correct errors induced by its own previous actions. It jointly grounds the target, estimates its visibility, and generates continuous flight actions through a structured prediction interface. Training uses Spatially Grounded Continued Pretraining (CPT) on a 500k mixed pool, a three-stage Curriculum-based Supervised Fine-Tuning (SFT) process, Chain-of-Thought (CoT) training, and a closed-loop Reinforcement Learning (RL) stage. Relative to OpenVLA, CosFly-VLA-0.8B reduces open-loop Average Displacement Error (ADE) by 34.1% on seen-test and 35.3% on unseen-test; closed-loop optimization improves Success Rate by 29.8% (+17 percentage points) and 2.5% (+2 points), respectively.
1. Background and Motivation
For a UAV tracking a dynamic target in low-altitude urban flight, tracking failures often arise not from ordinary visible-frame localization errors, but from recurring occlusion. A target may be temporarily hidden by buildings, trees, vehicles, or crowds. Once the visual cue disappears from the field of view, conventional detection–re-identification–control pipelines lose the stable identity evidence needed for recovery. At that point, tracking is no longer simply "seeing and following" the target; it becomes a closed-loop decision problem under occlusion. The agent must infer where an unseen target may have moved, keep acting without direct visual observation, and re-acquire the target before tracking errors continue to accumulate.
Vision-Language-Action (VLA) models offer a natural modeling route, but existing training pipelines expose three limitations under sustained occlusion. First, the model needs stronger aerial spatial reasoning — after the target disappears, its likely reappearance depends on depth relations, relative scale, height difference, motion direction, and occlusion ordering under the UAV viewpoint, yet generic vision-language pretraining does not necessarily provide sufficient supervision for these UAV-specific geometric cues. Second, the training data must preserve enough long-occlusion samples, but many window-construction pipelines discard samples where the target is invisible in most historical frames — precisely the supervision needed for post-occlusion recovery. Third, target re-acquisition must be optimized in closed loop, since once the model makes an incorrect prediction at one step, its action changes the subsequent observation distribution.
CosFly-VLA makes sustained target disappearance, reasoning under occlusion, and target re-acquisition the central object of training and evaluation rather than an afterthought.
2. Task Formulation and Interface
CosFly-VLA formulates UAV dynamic target tracking as the closed-loop pursuit of a moving target by a single quadrotor agent equipped with an egocentric monocular camera. At decision step $t$, the policy $\pi_{\theta}$ receives a short visual history, a language task/target prompt, and textual state history, then emits the current target grounding, an action chunk, and a visibility estimate:
$$o_{t}=(I_{t-K+1{:}t},\,l,\,s_{t}^{\text{prev}}),\qquad\hat{y}_{t}=(\hat{b}_{t},\,\Delta_{t+1{:}t+H_{a}},\,\hat{v}_{t}).$$
Here $I_{t-K+1{:}t}$ is a $K{=}5$-frame RGB stream at $640{\times}360$ resolution, $l$ is the language task/target prompt, and $s_{t}^{\text{prev}}$ contains 4-DoF UAV poses, previous target boxes, and visibility flags. The output $\hat{y}_{t}$ consists of the current Qwen-style target box $\hat{b}_{t}$, an $H_{a}{=}8$ future waypoint-delta action chunk $\Delta_{t+1{:}t+H_{a}}\in\mathbb{R}^{H_{a}\times 4}$, and a target-visibility probability $\hat{v}_{t}$. Each action delta is $(\Delta x,\Delta y,\Delta z,\Delta\text{yaw})$ in the controller's world-frame convention. At deployment, a receding-horizon controller with $H_{c}{=}1$ executes only the first predicted delta before the next observation arrives.
Figure 1: Task overview. Blue trajectory = pedestrian motion, green = UAV flight path. At each step the agent observes five egocentric frames, previous target boxes, and the task prompt, predicting three synchronized outputs: target visibility, current-frame bounding box, and an 8-step 4-DoF waypoint action chunk.
Occlusion grading is based on a geometric visibility test rather than model predictions. Five line-of-sight (LOS) rays are cast from the UAV camera position to the target center, head, feet, left side, and right side. A frame is marked invisible if fewer than three rays are unblocked or the target falls outside the valid distance gate $(2,50)$ m:
$$\chi_{t}^{\mathrm{occ}}=\mathbf{1}\!\left[\mathrm{vis\_frac}_{t}<0.6\ \lor\ d_{t}\notin(2,50)\right],\qquad\mathrm{occ\_rate}=\frac{1}{T}\sum_{t=1}^{T}\chi_{t}^{\mathrm{occ}}.$$
Easy = $\mathrm{occ\_rate}<5\%$, Medium = $5\%\leq\mathrm{occ\_rate}<20\%$, Hard = $\geq 20\%$ (with $>70\%$ filtered out as nearly always-invisible targets that do not define a meaningful tracking problem).
3. Core Method: Backbone–Decoupled-Heads Architecture
The architectural core of CosFly-VLA is a backbone–heads separation: a shared vision-language backbone encodes multimodal understanding, while three lightweight task heads read different subsets of the representation through a meta-query interface to decode the structured continuous predictions required for closed-loop tracking.
Concretely, the framework keeps Qwen3.5 as a frozen backbone $\Phi$ with Low-Rank Adaptation (LoRA) and attaches three lightweight task heads through a meta-query interface. The backbone encodes multi-frame RGB observations, the language prompt, UAV pose history, and previous boxes into a shared visual-text-state representation; meta-query spans then route task-specific hidden states to a flow-matching Diffusion Transformer (DiT) action expert, an MLP bounding-box (bbox) head, and an MLP visibility head. This avoids brittle JSON-like text decoding by producing structured boxes, visibility scores, and waypoint chunks directly from task-specific hidden states.
The decomposition matches the structure of UAV tracking: grounding anchors the visible target, action prediction drives the controller, and visibility estimation tells the policy when recovery should rely more on spatial and temporal context than on appearance.
flowchart LR
A["5x RGB frames
640x360"] --> B["Qwen3.5 backbone
(frozen + LoRA)"]
L["Language prompt"] --> B
S["Pose history +
prev boxes +
vis flags"] --> B
B --> M["Meta-query spans"]
M --> H1["DiT Action Expert
(12-layer flow-matching)
8-step 4-DoF waypoints"]
M --> H2["MLP BBox Head"]
M --> H3["MLP Vis Head"]
style B fill:#e0e7ff,stroke:#2563eb
style M fill:#fef3c7,stroke:#d97706
style H1 fill:#dcfce7,stroke:#16a34a
style H2 fill:#fce7f3,stroke:#db2777
style H3 fill:#f3e8ff,stroke:#9333ea
Figure 2: CosFly-VLA architecture. Multi-frame observations, language instructions, state history, and boundary-tagged meta-query tokens are encoded by a frozen Qwen3.5 backbone with LoRA adapters. Meta-query hidden states route to three structured prediction heads.
4. Progressive Training Recipe (Four Stages)
Figure 3: Training recipe. (a) Spatially Grounded CPT; (b) three-stage Curriculum SFT; (c) CoT training; (d) closed-loop RL.
4.1 Spatially Grounded Continued Pretraining (CPT)
CPT injects five capabilities difficult to learn from visible-frame tracking windows alone: aerial 3-D spatial relations, metric distance and size estimation, object attribute grounding, aerial mathematical/counting reasoning, and caption-level scene description. The data mixture is roughly 500k samples: 70% internal UAV trajectory data and 30% six external aerial spatial-VL datasets (AirSpatial, Open3DVQA-v2, HRVQA, AVI-Math, AirCopBench, CapERA). CPT updates all backbone parameters including the vision encoder, producing a spatially aligned backbone for downstream SFT.
4.2 Three-Stage Curriculum SFT
A single end-to-end SFT run is unstable because the action, bbox, and visibility paths have different loss scales, convergence speeds, and sample requirements. The three-stage curriculum proceeds as follows. First, multi-head warm-up initializes the three task heads with the backbone frozen and LoRA disabled, so the heads learn to read useful hidden states before backbone adaptation begins. Second, the warmed heads are reloaded into a unified multitask model, LoRA is enabled, and the policy is trained on the natural tracking distribution. Third, the curriculum shifts the data distribution toward rare maneuvers and occlusion-heavy windows, including an explicit 15% long-occlusion quota. Window bucketing follows a fixed priority rule:
$$g(\cdot)=\mathrm{argmax}\,[\text{long\_occ}\succ\text{avoid\_tight}\succ\text{avoid\_loose}\succ\text{sharp\_turn}\succ\text{short\_occ}\succ\text{turn}\succ\text{small/alt}\succ\text{cruise}\,].$$
Stage-3 weights follow a $1{:}1.5{:}2{:}3$ ratio over cruise/turn/loose-avoidance/tight-avoidance buckets, so samples that most often lead to closed-loop failure exert larger training pressure.
The action head is trained with a rectified-flow target over normalized action chunks. Given a ground-truth waypoint tensor $x_{1}\in[-1,1]^{H_{a}\times 4}$ with $H_{a}{=}8$, Gaussian noise $x_{0}\sim\mathcal{N}(0,I)$, and $t\sim\mathcal{U}[0,1]$, we form $x_{t}=(1-t)x_{0}+tx_{1}$ and train the DiT to predict the constant flow direction:
$$\mathcal{L}_{\text{act}}=\mathbb{E}_{x_{1},x_{0},t}\left[\bigl\|\Psi^{\text{act}}(x_{t},t,z^{\text{act}}_{\text{pool}},z^{\text{act}})-(x_{1}-x_{0})\bigr\|_{2}^{2}\right].$$
The bbox head uses an $\ell_{1}$ term plus Generalized Intersection-over-Union (GIoU); the visibility head uses binary cross-entropy. The full SFT loss is:
$$\mathcal{L}=\sum_{i\in\mathcal{B}}w(g_{i})\bigl[\lambda_{a}\mathcal{L}^{(i)}_{\text{act}}+\lambda_{b}\mathcal{L}^{(i)}_{\text{box}}+\lambda_{v}\mathcal{L}^{(i)}_{\text{vis}}\bigr]+\lambda_{\text{lm}}\mathcal{L}_{\text{lm}},$$
where $w(g_{i})$ is the curriculum bucket weight, $(\lambda_{a},\lambda_{b},\lambda_{v},\lambda_{\text{lm}})=(1.0,1.0,0.05,0.0)$ during SFT, and $\lambda_{\text{lm}}=0.1$ in the CoT phase with the language-modeling loss masked away from meta-query positions.
4.3 Chain-of-Thought Training
CoT samples are generated by a Qwen3.5-397B teacher in 8-bit floating-point (FP8) precision, requiring the assistant to produce a reasoning trace in \think...\ before the structured \
4.4 Long-Occlusion Data Mining
The SFT pipeline starts from 6,785 raw Town10HD trajectories, removes 97 UAV-collision trajectories, applies target-clean filtering, and retains 5,861 clean trajectories yielding 1,740,278 multimodal samples. The default window builder uses max_invisible=2, which discards many windows needed for re-acquisition. A relaxed miner (max_invisible=4, min_visible_history=0, max_alt_jump=3.0 m) surfaces 233,437 candidate windows, including 1,307 windows with 4-of-5 invisible inputs and 5,254 with 5-of-5 invisible inputs, routed into the hard and long-occlusion buckets so disappearance is a first-class training case rather than a filtering error.
4.5 Closed-Loop RL (EG-FPO)
The final phase targets the gap between offline supervision and closed-loop deployment, where predicted boxes and actions are fed back into future observations and small errors can compound into drift, collision, or failure to re-acquire. CosFly-VLA fine-tunes the SFT policy with a closed-loop reinforcement fine-tuning pipeline (CosFly-RFT), keeping the Qwen3.5 backbone and meta-query connectors frozen while updating the flow-matching DiT action expert through Expert-Guided Flow-Policy Optimization (EG-FPO). EG-FPO combines three well-established ideas: flow/diffusion-style continuous action generation, PPO-style likelihood-ratio clipping, and group-relative advantage normalization from GRPO.
Rollout collection is performed in a heterogeneous closed-loop system: a CARLA simulator service exposes reset/step endpoints; a rollout collector maintains the 5-frame observation window and UAV pose history; an action-head service returns the deterministic action mean, bbox, and visibility. For each path, $K$ on-policy trajectories are sampled from a fixed-variance Gaussian proxy around the deterministic flow mean: $a\sim\mathcal{N}(\mu_{\theta}(s),\mathrm{diag}(\sigma^{2}))$. Each group is augmented with one expert-replay trajectory as a high-return anchor preventing group-advantage collapse when all online rollouts fail. After each update, the new action head is hot-reloaded into the serving process for the next collection round.
Figure 4: Closed-loop RL pipeline. From the SFT checkpoint, CosFly-VLA is served as an action-head policy in CARLA. The collector gathers on-policy trajectories plus one expert anchor per start condition, computes rewards from stand-off distance, target IoU, success, and collision terms, and hot-reloads the updated head for the next round.
5. Key Experiments
5.1 Open-Loop Evaluation
Open-loop evaluation scores structured predictions on held-out 5-frame windows: the model receives identical observations and prompts, and its predicted waypoints, target box, and visibility are compared against ground truth without executing any action. CosFly-VLA-0.8B (SFT+CoT) achieves overall ADE of 0.7175 m on seen-test versus 1.0887 m for OpenVLA — a 34.1% reduction; on unseen-test the reduction reaches 35.3%. Crucially, the advantage grows with occlusion severity: seen-test Hard ADE drops from 1.3542 m (OpenVLA) to 0.7218 m (−46.7%).
| Method | Overall ADE↓ | Easy ADE↓ | Medium ADE↓ | Hard ADE↓ | IoU↑ | Vis F1↑ |
|---|---|---|---|---|---|---|
| Opus-4.6 (closed VLM) | 1.1014 | 0.8230 | 1.3999 | 1.3499 | 0.033 | 0.939 |
| Gemini-3.1-Pro | 0.8913 | 0.7988 | 1.1616 | 0.7184 | 0.513 | 0.964 |
| OpenVLA | 1.0887 | 0.8442 | 1.3193 | 1.3542 | – | – |
| π₀ | 1.2050 | 0.9066 | 1.3986 | 1.6604 | – | – |
| π₀.₅ | 1.1407 | 0.8690 | 1.3301 | 1.5357 | – | – |
| CosFly-VLA-0.8B (SFT) | 0.8247 | 0.6719 | 0.9893 | 0.9599 | 0.649 | 0.938 |
| CosFly-VLA-0.8B (SFT+CoT) | 0.7175 | 0.6351 | 0.8522 | 0.7218 | 0.659 | 0.943 |
Table 1: seen-test open-loop evaluation (lower ADE/DE/center is better; higher IoU/Vis F1 is better).
5.2 Closed-Loop Evaluation
Closed-loop evaluation executes predicted action chunks and feeds action-induced observations back into the policy. The RL-tuned CosFly-VLA-0.8B achieves the best SR, rollout ADE, and dErr on both splits. Compared with the strongest general VLA baseline (OpenVLA), seen-test SR improves from 57% to 74% (+17 percentage points, +29.8% relative) and unseen-test from 80% to 82% (+2 points, +2.5% relative). Rollout ADE drops 10.0% on seen-test (9.0→8.1) and 8.7% on unseen-test (6.9→6.3); stand-off distance error decreases by 20.6% and 18.2% respectively.
| Method | SR Seen↑ | SR Unseen↑ | TCR Seen↑ | ATF Seen↑ | ADE Seen↓ | ADE Unseen↓ | dErr Seen↓ | dErr Unseen↓ |
|---|---|---|---|---|---|---|---|---|
| π₀ | 41% | 47% | 68% | 25.4 | 14.4 | 17.6 | 8.7 | 10.4 |
| π₀.₅ | 47% | 60% | 68% | 27.3 | 12.9 | 14.0 | 6.3 | 7.3 |
| OpenVLA | 57% | 80% | 73% | 28.4 | 9.0 | 6.9 | 3.4 | 3.3 |
| CosFly-VLA-0.8B (SFT) | 70% | 79% | 79% | 30.5 | 8.9 | 8.8 | 3.3 | 3.9 |
| CosFly-VLA-0.8B (RL) | 74% | 82% | 74% | 29.6 | 8.1 | 6.3 | 2.7 | 2.7 |
Table 2: Closed-loop comparison (SR/TCR/ATF higher is better; ADE/dErr lower is better).
5.3 Ablation Studies
The cumulative ablation shows gains are not confined to easy visible-frame tracking. Adding curriculum learning reduces seen-test overall ADE by 7.0% relative to SFT only (0.8924→0.8295). Adding spatially grounded CPT coincides with a 9.2% reduction in unseen-test overall ADE (0.7007→0.6364) and a 13.4% reduction in unseen-test Hard ADE (0.9284→0.8039). The CoT stage yields the largest marginal gain on seen-test Hard, reducing ADE by 24.8% (0.9599→0.7218), and further reduces unseen-test Hard ADE by 11.7% (0.8039→0.7096). From SFT-only to the final +CoT variant, overall ADE decreases by 19.6% on seen-test and 16.3% on unseen-test, with Hard ADE decreasing by 30.9% and 23.0% respectively.
| Variant | Curr. | CPT | CoT | seen Overall↓ | seen Hard↓ | unseen Overall↓ | unseen Hard↓ |
|---|---|---|---|---|---|---|---|
| SFT only | ✗ | ✗ | ✗ | 0.8924 | 1.0451 | 0.7087 | 0.9217 |
| +curriculum | ✓ | ✗ | ✗ | 0.8295 | 0.9670 | 0.7007 | 0.9284 |
| +spatial CPT | ✓ | ✓ | ✗ | 0.8247 | 0.9599 | 0.6364 | 0.8039 |
| +CoT supervision | ✓ | ✓ | ✓ | 0.7175 | 0.7218 | 0.5931 | 0.7096 |
Table 3: Cumulative training-recipe ablation (open-loop ADE, lower is better).
Figure 6: Qualitative closed-loop comparison on representative CARLA episodes. Columns compare π₀.₅, CosFly-VLA-0.8B (SFT), and CosFly-VLA-0.8B (RL). Bright-yellow curve = ground-truth UAV path; bright-magenta dashed = pedestrian motion.
6. Limitations and Future Work
Author-stated limitations:
- The state history $s_{t}^{\text{prev}}$ is populated from ground-truth state history in both open-loop and closed-loop evaluation; robustness to predicted-state feedback is left to future work (Section 7).
- Source code and training configurations are not yet publicly released (the repo README marks them as "being prepared, not yet public"), so reproducibility remains to be verified.
- The 2B/9B configurations are architectural scaling targets rather than evaluated rows in the present benchmark — empirical scaling results are not reported.
Analysis:
CosFly-VLA's core contribution is making occlusion recovery a first-class training and evaluation object rather than an afterthought. However, its reliance on ground-truth state history means the compounding effect of prediction error in real deployment is not yet fully tested in closed loop — the paper concedes this is future work. While CARLA provides controllable occlusion grading, the simulation-to-reality gap remains an open question. Additionally, the labeling cost of the 500k-sample CPT pool and the compute expense of the Qwen3.5-397B teacher for CoT data pose reproduction barriers for smaller teams. The fixed-variance Gaussian proxy in EG-FPO simplifies likelihood-ratio computation but may limit exploration efficiency; adaptive variance or more expressive policy representations are potential improvements.
7. Conclusion
The central idea of CosFly-VLA is that when a target disappears from view, tracking is no longer "seeing and following" but a closed-loop recovery problem requiring spatial-hypothesis maintenance, self-induced drift suppression, and active re-acquisition. To realize this it does three things: it makes occlusion recovery a first-class training citizen (long-occlusion mining + curriculum quota), injects aerial geometric priors into the backbone via spatially grounded CPT, and optimizes recovery on the action-induced observation distribution through closed-loop RL.
The experimental evidence supports this design: relative to OpenVLA, the 0.8B model reduces open-loop ADE by 34.1% on seen-test and improves closed-loop SR by 17 percentage points; ablations show CPT primarily improves cross-map generalization (unseen-test), CoT primarily improves severe occlusion (Hard), while curriculum learning and RL each contribute gains on distribution shift and closed-loop behavior respectively. These results mark a step from visible-frame imitation toward spatially grounded closed-loop control. The reliance on ground-truth state history, the unreleased code, and the simulation-to-reality transfer remain open questions to be validated.
SOURCE LINKS
