Skip to content
RobotWorld
Back to Papers

PAPER DEEP DIVE

MLLM第一人称egocentric

Reinforcing Egocentric Spatial Perception in Multimodal Large Language Models via Ego Scene Augmentation

Egocentric Visual Question Answering (VQA) has attracted widespread attention as an important task for enabling Multimodal Large Language Models (MLLMs) to interact with the real world. However, existing MLLMs struggle to perform effective spatial reasoning in complex egocentric scenes due to their limited spatial perception capabilities. To this end, we introduce Ego Scene Augmentation (ESA), an egocentric spatial perception framework, which actively enhances the spatial perception capabilities from the egocentric perspective, powered by the proposed Ego-element Graph. Our core insight is leveraging the Ego-element Graph as an intermediary representation to augment the egocentric spatial perception of MLLMs via visual foundational models. Specifically, we 1) construct the Ego-element Graph, which encapsulates and integrates egocentric spatial features enabled by visual foundational models; 2) enhance the spatial perception capabilities of MLLMs via the Ego-element Graph for ego-perspective scenes. Our proposed ESA framework presents significant performance improvement on the EgoTextVQA benchmark. We achieve an 8.14% gain on the indoor setting and an 8.72% gain on the outdoor setting. Furthermore, our ESA shows the most impressive performance improvement in the shopping subset of the indoor setting. The project code is publicly available.

Chi Kit Wong, Ye Pan, Yuanhuiyi Lyu, Xu Zheng, Zidong Cao, Lutao Jiang, Zixin Zhang, Huiyu Zhou, Xuming HuJuly 16, 20265 min read
中文
Chi Kit Wong, Ye Pan, Yuanhuiyi Lyu, et al. (HKUST(GZ) · HKUST)
arXiv:2607.14497 · GitHub

One-Sentence Summary

ESA (Ego Scene Augmentation) is a plug-and-play egocentric spatial perception framework that constructs an Ego-element Graph (fusing object semantics, 3D positions, visual regions, and scene text) as an intermediate representation to inject visual foundation model spatial capabilities into MLLMs without additional training. On EgoTextVQA, it achieves 8.14% accuracy gain indoors, 8.72% outdoors, and 15.91% on the shopping subset.

Figure 1. ESA framework overview

Figure 1 — (a) In spatially complex egocentric scenes, current MLLMs lack spatial perception, mislocalize objects, and output "Unanswerable" without ESA. (b) With ESA, Ego-element Graph supports reliable localization and more accurate answers.

1. Background & Motivation

Egocentric VQA is an important task for MLLMs to interact with the real world. However, existing MLLMs have limited spatial reasoning in complex egocentric scenes: (1) difficulty adapting perception to egocentric viewpoint; (2) difficulty conducting accurate spatial perception in dynamic, cluttered scenes.

Core question: MLLMs rely on pre-trained visual perception insufficient for egocentric viewpoints — can visual foundation models (e.g., Depth-Anything) enhance MLLM spatial perception without additional training?

2. Core Method

ESA's key insight: use Ego-element Graph as intermediate representation to augment MLLM egocentric spatial perception via visual foundation models, with plug-and-play design.

Figure 2. ESA overall framework

Figure 2 — ESA overall framework. Ego-element Graph encodes object semantics, 3D positions, and visual regions, serialized as enhanced spatial prompt for MLLM.

2.1 Ego-element Graph

Directed attributed graph $G = (V, E, A)$: $V$ = nodes (objects/salient regions), $E$ = directed edges (pairwise spatial/contextual relations), $A$ = attribute vectors.

Node attribute vector:

$$a_i = [\, s_i \;\|\; p_i \;\|\; r_i \;\|\; t_i \,]$$

where $s_i$ = semantic features, $p_i \in \mathbb{R}^3$ = 3D position, $r_i \in \mathbb{R}^4$ = 2D region, $t_i$ = associated text cues.

Edge relation vector:

$$r_{ij} = [\, \mathbf{p}_j - \mathbf{p}_i \;\|\; \phi(\mathbf{r}_i, \mathbf{r}_j) \,]$$

Edges directed to reflect user-centric orientation ("left of", "in front of"). Graph: $G = f_{esa}(I)$. Serialized: $Prompt_{spatial} = Encoder(G)$ as JSON.

2.2 Pipeline Components

  • Region: detect objects and salient regions.
  • Depth: Depth-Anything for 3D position $p_i$.
  • Text: recognize scene text $t_i$.
  • Graph Organization: integrate multi-source cues into unified spatial structure.
flowchart TB
    A[Egocentric Image I] --> B[Region Extraction
objects + salient regions] A --> C[Depth Estimation
Depth-Anything: p_i in R3] A --> D[Text Recognition
scene text t_i] B & C & D --> E[Ego-element Graph G] E --> F[JSON Encoder] F --> G[Enhanced Spatial Prompt] G --> H[MLLM Inference
Qwen2.5-VL, LLaVA, etc.] H --> I[Accurate Egocentric VQA Answer] style E fill:#e0e7ff,stroke:#2563eb style C fill:#fef3c7,stroke:#d97706 style I fill:#dcfce7,stroke:#16a34a

3. Experimental Results

Evaluated on EgoTextVQA: ~1.5K egocentric videos, 7K Q&A pairs, indoor (household) and outdoor (driving/navigation). Compared against InstructBLIP, Emu3, LLaVA-NeXT, InternVL2, Qwen2-VL, Qwen2.5-VL (SOTA baseline).

Figure 3. Qualitative comparison

Figure 3 — Qualitative comparison. Only ESA correctly locates the super glue and aluminum foil.

MethodIndoor ACCOutdoor ACCIndoor ScoreOutdoor Score
InstructBLIP28.0921.072.121.84
Emu39.165.711.371.24
LLaVA-NeXT28.3521.002.131.84
InternVL228.5223.972.121.95
Qwen2-VL28.9724.922.151.99
Qwen2.5-VL (SOTA)31.2025.232.242.00
Ours (ESA)33.74 (+8.14%)27.43 (+8.72%)2.342.09
  • 8.14% indoor, 8.72% outdoor accuracy gain; 4.27% and 4.5% Score gains.
  • 15.91% gain on shopping subset — dense objects, overlapping text, complex layouts benefit most from structured spatial reasoning.
  • Conventional MLLMs rely on global embeddings and can't distinguish fine-grained positional relations; ESA explicitly models spatial/semantic relations via graph structure.

4. Ablation Study

Figure 5. Pipeline component ablation

Figure 5 — Pipeline component ablation: ESA produces more accurate, context-aware localization than existing MLLMs.

VariantDescriptionAvg ACCAvg Score
v0Baseline MLLM31.202.24
v1 + RegionAdd region extraction33.282.32
v2 + DepthAdd depth estimation33.542.33
v3 + TextAdd text recognition33.542.33
v4 + GraphAdd graph organization33.742.34
  • Depth (v2) contributes most, especially Gameplay and Others.
  • Text (v3) enhances semantic grounding, especially Gameplay and Book-Related.
  • Graph (v4) unifies multi-source cues, consistent improvement in Hands-on, Kitchen, Others.
  • Max nodes = 4 optimal — balance between information sufficiency and graph sparsity.

5. Main Contributions

  • Ego-element Graph: unifies object semantics, 3D positions, visual regions, and scene text into directed attributed graph as MLLM spatial perception intermediary.
  • Plug-and-play: no additional training, flexibly integrates with Emu3, LLaVA-NeXT, InternVL2, Qwen2-VL/2.5-VL.
  • Visual foundation model injection: Depth-Anything spatial capabilities enhanced via graph-serialized JSON prompt.
  • Significant gains: 8.14% indoor, 8.72% outdoor, 15.91% shopping — largest in cluttered text-rich scenes.

6.

场景图节点属性拼接

$$ a_{i}=\left[\,s_{i}\;\|\;p_{i}\;\|\;r_{i}\;\|\;t_{i}\,\right] $$

节点间关系特征

$$ {r}_{ij}=\left[\,{p}_{j}-{p}_{i}\;\|\;\phi({r}_{i},{r}_{j})\,\right] $$

空间推理输出

$$ \hat{Ans}=MLLM(Q,Prompt_{spatial}) $$

Limitations & Future Work

Author-stated limitations:

  • ESA sensitive to viewpoint changes; external perception module errors may propagate.
  • Without camera calibration, provides ordinal not metric depth.

Analysis:

ESA's core value is proving explicit spatial evidence effectively complements MLLM's implicit visual representations — graph-structured modeling of inter-object relations is more precise than global embeddings. Plug-and-play design lowers deployment barrier. Sensitivity to upstream module errors means detection quality sets the ceiling. Ordinal-only depth limits precise quantitative reasoning. Future: uncertainty-aware graphs (node/edge confidence), metric geometry (camera calibration), temporal aggregation (cross-frame integration). Optimal max-nodes=4 suggests current graph construction may lose some cues; adaptive node selection is a natural extension.

7. Conclusion

ESA is a plug-and-play egocentric spatial reasoning framework that strengthens MLLM spatial perception without additional training. By constructing an Ego-element Graph that explicitly organizes object semantics, visual regions, depth cues, and scene text, and serializing structured evidence into an enhanced spatial prompt for inference, ESA achieves 8.14% and 8.72% accuracy gains on EgoTextVQA indoor and outdoor settings. Ablation shows depth contributes most, while text cues and graph organization provide complementary benefits. Overall, explicit spatial evidence effectively complements the implicit visual representations of existing MLLMs in cluttered, text-rich egocentric scenes.

Inject depth perception into MLLMs via Ego-element Graph — make spatial reasoning move from implicit embeddings to explicit graph structure.

Related Papers

Zero-Shot Mission-Level Evaluation for Aerial MLLM Agents

Zero-Shot Mission-Level Evaluation for Aerial MLLM Agents

Multimodal Large Language Models (MLLMs) are emerging as core reasoning modules for embodied agents, yet it remains unclear how well general-purpose models can solve long-horizon embodied tasks from a single high-level instruction. We introduce MissionBench, a benchmark for mission-level evaluation of MLLMs in aerial 3D environments. It comprises 120 missions across five simulated 3D environments and four task families. Agents must autonomously plan, navigate, and report outcomes using only egocentric observations and its action history, without aerial-specific fine-tuning. Across 22 open- and closed-source MLLMs, the strongest model succeeds on fewer than 35% of missions compared to 84.4% human performance, highlighting the difficulty of multi-step embodied tasks. Despite large variations between model families, we observe gains from scaling, indicating that larger general-purpose models possess stronger zero-shot embodied capabilities. Our analysis shows that mission-level competence requires coordinating multiple capabilities beyond spatial perception, including multi-step planning and adaptive reasoning. This motivates closed-loop evaluation and highlights both the promise and risk of scaling-driven improvements for embodied AI.

航空Aerial多模态大模型Jul 24, 2026
Beyond Single Expert: Harmonizing Diverse Visual Priors in MLLMs for Spatial Understanding

Beyond Single Expert: Harmonizing Diverse Visual Priors in MLLMs for Spatial Understanding

Multimodal Large Language Models (MLLMs) have demonstrated substantial promise in spatial understanding. Existing works typically incorporate prior knowledge extracted from a pre-trained foundation model to further enhance the spatial awareness of MLLMs. In this paper, we first reveal that when integrating diverse foundation models into MLLMs, different models provide complementary spatial priors that benefit different tasks. Motivated by this, we propose $\textbf{ViPS}$, a novel multi-model prior framework designed to fully unleash the potential of incorporating multiple $\textbf{Vi}$sual $\textbf{P}$riors from diverse models into MLLMs for $\textbf{S}$patial understanding. Specifically, ViPS introduces an Efficient Prior Proxy to generate multiple foundational priors with minimal inference overhead, and a Dynamic Prior Fusion mechanism to achieve harmonious and context-aware prior fusion and injection from the prior proxies. Extensive experiments demonstrate that ViPS successfully harmonizes diverse visual priors, establishing new state-of-the-art performance across multiple complex spatial reasoning and 3D spatial understanding benchmarks. Project page: https://visual-ai.github.io/vips

MLLM空间理解spatial understandingJul 16, 2026