Beyond Single Expert: Harmonizing Diverse Visual Priors in MLLMs for Spatial Understanding

1. Background and Motivation

Spatial understanding is a foundational pillar for real-world interaction and reasoning. The mainstream approach to enhancing MLLM spatial awareness is to inject priors from a single pre-trained foundation model (e.g., VGGT). But this "single-expert" paradigm overlooks a key fact: different foundation models, due to their diverse training paradigms, encapsulate distinct representations and spatial semantics, each excelling at different things.

The authors validate this via prior analysis (Figure 2): the prior features of different foundation models (VGGT, DepthAnything3, TraceAnything, Wan2.1, RADIO) cluster into disparate regions in t-SNE space, and their activation heatmaps focus on different structural and semantic cues within the same scene. The cross-task evaluation (Figure 3) shows more: no single foundation model dominates all metrics — column-wise, the best model varies by sub-task; row-wise, each model peaks on its own best sub-task. This high complementarity directly motivates the core objective: efficiently extract and adaptively harmonize prior knowledge from multiple foundation models.

Figure 2: Prior analysis of diverse foundation models. Left: t-SNE of prior features. Right: spatial heatmaps showing each model focuses on different structural/semantic cues.

Figure 3: Relative performance of diverse foundation models as MLLM priors across spatial tasks — no single model dominates.

2. Core Method: The ViPS Framework

ViPS (Visual Priors for Spatial understanding) integrates prior knowledge from multiple foundation models into the MLLM via two pivotal mechanisms: the Efficient Prior Proxy and Dynamic Prior Fusion. The overall framework is shown in Figure 4.

Figure 4: ViPS framework overview. Integrates diverse prior knowledge via Efficient Prior Proxy and coordinates them via Dynamic Prior Fusion.

flowchart TB
    V["Input video V
32 frames"] --> ENC["Vision encoder Phi_vis"]
    ENC --> FVIS["Base embeddings F_vis"]
    V --> EBASE["Base model E_base
(e.g. VGGT)"]
    EBASE --> FBASE["F_base in R^(SxD)"]
    FBASE --> P1["Proxy phi_1 -> F_prior^1
(depth cues)"]
    FBASE --> P2["Proxy phi_2 -> F_prior^2
(motion tracking)"]
    FBASE --> PK["Proxy phi_K -> F_prior^K
(generic repr)"]
    P1 --> ZC1["ZeroConv_1"]
    P2 --> ZC2["ZeroConv_2"]
    PK --> ZCK["ZeroConv_K"]
    ZC1 --> FUSE["Dynamic Prior Fusion
w = Softmax(MLP(x_last^text))"]
    ZC2 --> FUSE
    ZCK --> FUSE
    FUSE --> FPR["Fused prior F_hat_prior"]
    FPR --> MLLM["MLLM (Qwen2-VL-7B / Qwen3-VL-8B)"]
    FVIS --> MLLM
    style EBASE fill:#e0e7ff,stroke:#2563eb
    style FUSE fill:#fef3c7,stroke:#d97706
    style FPR fill:#dcfce7,stroke:#16a34a

2.1 Efficient Prior Proxy

Directly extracting features from $K$ distinct foundation models $\{\mathcal{E}_{1},\dots,\mathcal{E}_{K}\}$ requires $K$ independent forward passes $\mathcal{E}_{k}(\mathcal{V})$, with latency and memory scaling linearly. To decouple computational overhead from the number of integrated models, ViPS uses a single robust vision encoder as the base model $\mathcal{E}_{base}$ to extract a unified foundational representation:

$$F_{base}=\mathcal{E}_{base}(\mathcal{V})\in\mathbb{R}^{S\times D_{base}}.$$

It then instantiates $K$ lightweight proxy networks (each a simple MLP $\phi_{k}(\cdot)$) to distill each model's prior from the shared base feature:

$$F^{k}_{prior}=\phi_{k}(F_{base})\in\mathbb{R}^{S\times D_{prior}}.$$

To guarantee fidelity, the outputs are supervised during training with the ground-truth features $F^{k}_{gt}=\mathcal{E}_{k}(\mathcal{V})$ from the $K$ original foundation models via an $L_{2}$ alignment loss:

$$\mathcal{L}_{alignment}=\sum_{k=1}^{K}\big\|F^{k}_{prior}-F^{k}_{gt}\big\|_{2}^{2}.$$

The key insight: different foundation models share substantial low- and mid-level visual semantics, so a robust base representation contains enough foundational knowledge from which distinct high-level priors can be efficiently derived via shallow proxies. Since a shallow MLP's cost is negligible compared to a full foundation model, the framework scales to an arbitrary number of priors without linearly increasing inference overhead.

2.2 Dynamic Prior Fusion

Given diverse priors $\{F^{1}_{prior},\dots,F^{K}_{prior}\}$, the challenge is integrating them harmoniously. Directly adding these heterogeneous priors fails to prioritize the most effective ones for a given task and can overwhelm the language model. ViPS designs a dynamic weighting mechanism: it extracts the representation of the final token of the text query $x_{last}^{text}$ (which, due to the LLM's causal mechanism, naturally aggregates the semantics of the entire instruction), passes it through an MLP to compute $K$-dimensional logits:

$$z=\text{MLP}_{weight}(x_{last}^{text}),\qquad w=\text{Softmax}(z)\in[0,1]^{K}.$$

However, experiments show directly using these weights fails to yield progressive improvement — the disparate prior distributions from different source models confuse the MLLM early in training. To enable progressive prior injection, an independent zero-initialized convolutional layer is applied to each proxy branch before fusion:

$$\tilde{F}^{k}_{prior}=\text{ZeroConv}_{k}(F^{k}_{prior}).$$

Zero-initialization ensures convolution outputs are initially zero, preserving original MLLM representations and preventing prior perturbations from disrupting early training, then gradually scaling up as the MLLM learns from diverse priors. The final fused multi-expert prior is:

$$\hat{F}_{prior}=\sum_{k=1}^{K}w_{k}\cdot\tilde{F}^{k}_{prior}.$$

This fused representation is added element-wise to the projected visual embeddings $\mathcal{P}(F_{vis})$ (image tokens). In practice, this mechanism is uniformly applied across 5 layers of the MLLM for deep, progressive integration.

3. Key Experiments

ViPS is evaluated on six datasets: VSI-Bench (8 spatial reasoning sub-tasks) and five ScanNet-series benchmarks (ScanRefer, Multi3DRefer, Scan2Cap, ScanQA, SQA3D). Base MLLMs are Qwen2-VL-7B and Qwen3-VL-8B; $K=5$ foundation models are integrated (VGGT as base + DepthAnything3, TraceAnything, Wan2.1, RADIO); 32 frames are uniformly sampled per video.

3.1 VSI-Bench Spatial Reasoning

ViPS achieves a leading average of 63.8%, surpassing the previous best spatial-enhanced model VLM-3R (57.2%). While trailing VG-LLM-8B in Appearance Order, it dominates most categories, proving the advantage of dynamically injecting multiple priors over relying on a single foundation model.

Table 1: VSI-Bench performance comparison (accuracy %). ViPS leads on average.

3.2 ScanNet-Series 3D Spatial Understanding

ViPS achieves top scores on visual grounding (ScanRefer 64.6% Acc@0.25) and QA (ScanQA 107.9 CIDEr), outperforming Vega-3D. While slightly trailing 3DRS on Scan2Cap, this confirms the unified proxy injection effectively harnesses multiple foundation priors for complex 3D spatial understanding.

Table 2: ScanNet-series key metrics.

3.3 Efficiency and Ablation

A key ablation compares "GT Priors" (independent forward passes of original foundation models) with the ViPS proxy version. ViPS achieves nearly matching performance at only 1× parameters and latency versus 5× for GT Priors, with an alignment loss estimation error of just 0.252. Removing the alignment loss degrades performance (ScanRefer 65.4→64.0, ScanQA 107.9→106.3), proving proxy supervision is necessary.

Figure 5: Dynamic prior weight distribution. Left: across tasks (ScanQA, ScanRefer, etc.). Right: across VSI-Bench question types. Weights adapt dynamically by task, not statically relying on a single expert.

4. Limitations and Future Work

Author-stated / analysis limitations:

• ViPS slightly trails VG-LLM-8B on Appearance Order (34.3 vs 49.4), suggesting temporal-memory tasks may need different prior combinations or longer frame sampling.
• Foundation model selection still requires manual prior knowledge (5 hand-picked complementary models); automatic search for the optimal model combination is unsolved.
• Code is not released (only a project page), so proxy training details and alignment-target implementation await reproduction.
• Zero-init convolution ensures progressive learning but may limit prior injection speed; more flexible gating (e.g., learnable gates) is a potential improvement.

Analysis: ViPS's core contribution is empirically confirming the "multi-expert complementarity" hypothesis and providing an efficient engineering solution. In principle, proxy distillation works because foundation models share low/mid-level features — if the selected models have highly divergent representation spaces, proxy fidelity may drop. The interpretability of dynamic weights (Figure 5) is a highlight: weights shift by task rather than being static, offering an analytical tool for understanding "which prior helps which spatial reasoning." Future work could extend the proxy mechanism to more modalities (tactile, audio) or dynamically select foundation-model subsets to further cut cost.

5. Conclusion

The core insight of ViPS is that there is no single best prior for spatial understanding — different foundation models each excel at different things, and the key is to extract efficiently and harmonize adaptively by task. To realize this it does two things: distills multiple model priors from a single backbone via lightweight proxies, cutting inference cost from $K\times$ to $1\times$; and achieves progressive, context-aware prior fusion via zero-initialized convolutions and task-conditioned dynamic weights. Empirically, VSI-Bench average reaches 63.8% (vs. VLM-3R's 57.2%), with leading results on multiple ScanNet-series tasks; ablations show the alignment loss and dynamic weights are both indispensable. The work provides a scalable, interpretable engineering path for "multi-expert prior integration," though automatic model selection, proxy fidelity boundaries, and code release remain open issues.