Zero-Shot Mission-Level Evaluation for Aerial MLLM Agents

MissionBench: Zero-Shot Mission-Level Evaluation for Aerial MLLM Agents

Authors: Suman Navaratnarajah et al. (8 authors)  |  Institution: University of Technology Nuremberg & Fraunhofer IVI & THWS & AIST  |  arXiv: 2607.22014v1  |  Website: gomtae.github.io/publications/missionbench

---

One-Sentence Summary

MissionBench is a benchmark for evaluating frozen MLLMs on mission-level aerial 3D tasks, comprising 120 missions across 5 simulated environments and 4 task families. The strongest of 22 MLLMs achieves only 34.8% success vs. 84.4% human, revealing the challenge of multi-step embodied tasks while showing consistent zero-shot gains from model scaling.

---

Background and Motivation

MLLMs are emerging as core reasoning modules for embodied agents, but whether general-purpose models can solve long-horizon embodied tasks from a single high-level instruction remains unclear. Existing benchmarks provide only partial evidence: VLN evaluates path-level instruction following, object navigation focuses on target localization, and task-driven benchmarks like ALFRED rely on step-level subgoals. None directly assess whether frozen MLLMs can coordinate perception, planning, control, and reporting.

Figure 1: MissionBench evaluates MLLMs on mission-level aerial reasoning across 4 task families and 5 environments.

Aerial environments are particularly demanding because UAV missions are viewpoint-sensitive and open-ended. Success requires reaching informative poses, maintaining visual contact, adapting actions over time, and reporting mission-specific outcomes. No prior benchmark jointly evaluates frozen MLLMs across multiple mission families with continuous-magnitude pose control, task-specific success criteria, and explicit reporting in a single closed-loop episode.

---

MissionBench Design

Task Formulation

Modeled as embodied instruction following. The agent receives mission instruction $M$, general task info, and initial egocentric RGB image $I_0$ at pose $p_0 = (x_0, y_0, z_0, \psi_0)$. The agent outputs a structured response: bounding box, reasoning, action primitive with magnitude, report info, and DONE flag.

Eight directional primitives (forward/backward/strafe/ascend/descend/turn) with permissible magnitude up to $d_{\max}$ meters or $3d_{\max}$ degrees. At step $t$, the agent receives $I_t$, up to 2 previous images, and action history, looping until DONE or $T_{\max}$ exhausted.

Four Task Families

• Reporting (37.5%): Reconnaissance — extract and report information from egocentric observations.
• Inspection (26.7%): Infrastructure inspection/SAR — reach observation-ready configuration.
• Manipulation (13.3%): Landing, payload delivery, sample collection — approach and execute physical action.
• Patrol (47.5%): Perimeter security, traffic monitoring — follow routes along environmental structures.

Figure 2: MissionBench closed-loop evaluation pipeline. The MLLM receives mission instruction and egocentric images, outputs structured response; actions are parsed into executable waypoints.

Simulation Suite

Built on Unreal Engine 5 and Cosys-AirSim with 5 high-fidelity environments: Neighborhood, City, Forest, Savannah, and AirSimNH. Ground-truth trajectory lengths from 13m to 4171m, median 93m, mean 260m.

---

Evaluation Metrics

Success Rate (SR): Mission completion, not proximity alone. Inspection requires final position satisfying $\|p_i - p^*\| < \tau_d$ ($\tau_d = 5$m) and orientation $|\psi_i - \psi^*| < \tau_\psi$ ($\tau_\psi = 15°$):

$$\text{SR}_{\text{insp}} = \mathbb{1}[\|p_i - p^*\| < \tau_d \wedge |\psi_i - \psi^*| < \tau_\psi]$$

Patrol computes IoU of 20m-buffered trajectories:

$$\text{SR}_{\text{patrol}} = \mathbb{1}[\text{IoU}_{\text{buffer}} > 0.5]$$

Oracle Success Rate (OSR): Whether agent passes within $\tau_d$ at any point. OSR vs. SR distinguishes navigation failure (low OSR) from last-mile perception/reporting failure (high OSR, low SR).

Mission Progress (MP): Continuous progress signal:

$$\text{MP} = \max\left(0, \frac{\sum_{t=1}^{T}(d_{t-1} - d_t)}{d_0}\right)$$

Step Efficiency (Eff): $\text{Eff} = 1 - T_{\text{used}}/T_{\max}$. Collision Rate (CR): Fraction of steps with collisions.

Step budget $T_{\max}$ is set by task type: patrol $T_{\max}=50$, others $T_{\max}=20$. Per-step allowable magnitude $d_{\max}$ is automatically set to 1/4 of start-to-goal distance:

$$d_{\max} = \frac{1}{4} \| p_{\text{start}} - p_{\text{goal}} \|$$

The oracle gap is defined as:

$$\Delta_{\text{oracle}} = \text{OSR} - \text{SR}$$

$\Delta_{\text{oracle}} > 0$ indicates the agent reached the target but failed to complete the mission — navigation succeeded but last-mile perception/reporting failed. Perception-task correlation:

$$r_{\text{mIoU,MP}} = 0.463, \quad r_{\text{RMSE,MP}} = -0.315$$

Showing single-frame spatial perception is only a moderate predictor of mission-level success.

flowchart TD
    A["Mission instruction M + initial image I₀"] --> B["MLLM agent reasoning"]
    B --> C["Structured output:
BBox + reasoning + action + report + DONE"]
    C --> D{"DONE?"}
    D -- "No" --> E["Parse action to waypoint"]
    E --> F["Simulator executes → new observation I_t"]
    F --> B
    D -- "Yes" --> G["Evaluate: SR/MP/OSR/CR/Eff"]
    D -- "Budget exhausted" --> G

---

Experimental Results

22 open- and closed-source MLLMs evaluated, no aerial fine-tuning. 3 runs per mission, 30-mission test split.

Model	SR(%)↑	MP↑	OSR(%)↑	CR(%)↓	Eff(%)↑
Random baseline	0.0	5.7	3.3	0.1	64.8
Gemini 3.1 Pro	34.8	73.3	17.2	4.3	53.6
Gemini Robotics 1.6	32.2	70.3	17.8	1.6	13.6
Gemma-4-31B-IT	26.7	59.3	11.1	4.7	16.6
GPT-5.4	2.0	45.0	0.0	5.0	25.7
Claude Opus 4.6	3.3	18.9	10.0	4.0	17.9
Qwen 3.5 27B	10.0	53.2	8.9	0.0	36.0
Human (keyboard)	84.4	94.2	51.1	—	—
Human (VLM interface)	70.0	79.0	36.7	—	—

Figure 3: Human ground truth (green) vs. MLLM trajectories. Green circle = success, red cross = failure.

Key Findings

• Highly challenging: Best model only 34.8% SR, most below 10%.
• Consistent scaling gains: Gemini 3.1 Pro vastly outperforms Flash Lite (73.3 vs 24.0 MP); Qwen 3.5 from 2B to 27B improves MP from 18.1 to 53.2.
• Oracle gap: OSR > SR pattern (e.g., Gemini 3 Flash: 20.2% vs 7.9%) shows agents reach target but fail final perception/reporting.
• Perception insufficient: Single-frame spatial perception only moderately correlated with mission performance (Pearson $r_{\text{mIoU,MP}} = 0.463$).

Framework Ablation

Ablation	Variant	SR(%)	MP
Temperature	T=0.0 / 0.7 / 1.0	40.0 / 57.1 / 46.7	68.1 / 68.4 / 59.8
Resolution	SD / HD / FHD	42.9 / 50.0 / 57.1	75.3 / 72.8 / 68.4
History length	1 / 3 / 5 past images	28.6 / 57.1 / 30.8	33.1 / 68.4 / 69.6
Structured output	Default / w/o BBox / w/o reasoning	57.1 / 42.9 / 20.0	68.4 / 70.2 / 58.9

Reasoning field is critical — removing it drops SR from 57.1% to 20.0%. Three past images optimal; more dilutes relevant information. High resolution mainly improves task completion over coarse progress.

Scaling Behavior Details

Within the Qwen 3.5 dense family, scaling from 2B to 27B consistently improves MP from 18.1 to 53.2 and SR from 0.0% to 10.0%. In the Gemini family, 3.1 Pro vastly outperforms Flash Lite (73.3 vs 24.0 MP) and Gemini 2.5 Pro (54.2 MP). Gemini Robotics improves from 1.5 to 1.6 (35.9→70.3 MP, 0.0%→32.2% SR). However, scaling is not strictly monotonic: Claude Sonnet 4.6 outperforms Opus 4.6, and Qwen 3.6 variants don't uniformly improve over 3.5. This suggests architectural and training differences also play a role.

Spatial Perception Analysis

To test whether performance is explained by single-frame 3D spatial perception, MLLMs are evaluated in isolation: given initial image $I_0$, predict target bounding box and distance. Results show only moderate correlation between perception accuracy and end-to-end performance. Gemini Robotics 1.6 achieves highest mIoU with strong task performance, but Claude Opus 4.6 achieves lowest RMSE yet underperforms on MissionBench, while Gemini 2.5 Pro reaches non-trivial MP/SR despite near-zero mIoU. This confirms static perception alone is insufficient — success requires integrating perception with instruction following, spatial reasoning, and sequential decision-making.

---

Failure Mode Analysis

1. Premature termination: Agent declares mission complete within first few steps despite being far from target.
2. Drift and oscillation: Agent exhausts step budget circling near landmarks without converging.
3. Unmet mission objectives: Agent reaches target vicinity but misidentifies object or attributes (e.g., rhino as elephant).

---

Limitations

1. Only evaluates frozen MLLMs, not fine-tuned or aerial-specific models.
2. 5 environments, while diverse, may not cover all real-world scenarios.
3. Small human baseline sample size (N=2-3).
4. Closed-loop simulation is computationally expensive, limiting large-scale evaluation.

---

Conclusion and Outlook

MissionBench is the first benchmark jointly evaluating frozen MLLMs on mission-level aerial tasks across 4 task families and 5 environments. The strongest model completes only ~1/3 of missions, and single-frame spatial perception is insufficient to explain success — mission-level competence requires coordinating perception, planning, control, and reporting. Scaling brings consistent zero-shot improvements, highlighting both the promise and risk of scaling-driven capability emergence for embodied AI.

Key insight: "Mission-level competence requires coordinating multiple capabilities beyond spatial perception, including multi-step planning and adaptive reasoning." — MissionBench reveals the fundamental challenge of general MLLMs in long-horizon embodied tasks, providing a benchmark for closed-loop evaluation and scaling-driven improvements.

---

Deep analysis generated by RobotWorld paper-detail-generator based on full-text reading | arXiv:2607.22014v1