PAPER DEEP DIVE
ReferTrack: Referring Then Tracking for Embodied Visual Tracking
Embodied visual tracking (EVT) requires a mobile agent to continuously follow a specific target described in natural language using only onboard vision. While recent vision-language-action (VLA) policies unify target identification and trajectory planning, their chain-of-thought (CoT) reasoning often operates in abstract spatial latents that are difficult to supervise and weakly aligned with explicit image-space detections. To address this, we introduce ReferTrack, a referring-then-tracking paradigm that grounds EVT using a single forward-facing camera. Our model first selects the target from an indexed set of bounding boxes, then decodes tracking waypoints conditioned on this image-grounded decision. To preserve target motion cues over time, ReferTrack maintains a sliding-window queue of previously selected bounding boxes, injecting their geometric features into the visual history via temporal-viewpoint-bbox indicator (TVBI) tokens. We further enhance target identification by co-training on a custom Refer-QA dataset. On EVT-Bench, ReferTrack achieves state-of-the-art single-view performance with success rates of 89.4%, 73.3%, and 74.1% on the single-target, distracted, and ambiguity tracking splits, respectively -- matching or even surpassing several multi-camera baselines on identification-heavy tasks. Finally, real-world deployments on legged and humanoid robots validate its robust sim-to-real transfer capabilities. Code is available at https://github.com/MedlarTea/referTrack.
ReferTrack: Referring Then Tracking for Embodied Visual Tracking
Paper: ReferTrack: Referring Then Tracking for Embodied Visual Tracking
Authors: Hanjing Ye, Tianle Zeng, Jiazhao Zhang, Shaoan Wang, Zibo Zhang, Weisi Situ, Yuchen Zhou, Yonggen Ling, Hong Zhang
Affiliation: SUSTech RCV Lab, Tencent Robotics X, Peking University
Link: arXiv:2607.20061 | Code: GitHub
One-Sentence Summary
ReferTrack proposes a referring-then-tracking paradigm that selects the target from an indexed bounding box catalog before decoding tracking waypoints, aligning target identification with image-space detections, and uses TVBI tokens to inject historical target geometry, achieving 89.4%/73.3%/74.1% success rates on EVT-Bench single-view, surpassing several multi-camera baselines.
Background and Motivation
Embodied visual tracking (EVT) requires a mobile agent to continuously follow a natural-language-described target using only onboard vision. Existing VLA policies unify identification and planning, but their chain-of-thought (CoT) reasoning operates in abstract spatial latents that are difficult to supervise and weakly aligned with image-space detections. TrackVLA++ introduces a spatial-aware CoT token, but pure abstract reasoning lacks interpretability and supervisability.
Figure 1: ReferTrack formulates EVT as referring then tracking: forward-view detections as indexed bboxes, a single Refer-CoT token selects the target before trajectory prediction.
Method Details
1. Problem Formulation
EVT is given language instruction $\mathcal{L}$ and forward RGB stream $\mathcal{O}_{1:T}$; the goal is tracking waypoints $\mathbf{a}_{1:A}$. The key innovation moves target identification from abstract latent space to image space — selecting from an indexed bounding box catalog.
2. Observation Encoding
Dual-encoder architecture extracts SigLIP and DINOv2 features; grid pooling produces fine tokens $V^{\text{fine}} \in \mathbb{R}^{64 \times C}$ and coarse tokens $V^{\text{coarse}} \in \mathbb{R}^{4 \times C}$. Sliding window $H$ frames: $\mathcal{V}_T = \{V_{T-H}^{\text{coarse}}, \ldots, V_{T-1}^{\text{coarse}}, V_T^{\text{fine}}\}$.
Temporal-viewpoint-bbox indicator (TVBI) tokens inject historical target geometry:
$$ E_{\text{TVBI}}(t) = E_{\text{TVI}}(t) + \mathcal{P}_{\text{bbox}}(b_t) $$where $b_t \in [0,1]^4$ is the normalized bbox and $\mathcal{P}_{\text{bbox}}$ is a 2-layer MLP. When target is unobserved, $b_t = [0,0,0,0]$ serves as absence sentinel.
Figure 2: ReferTrack overview. Select indexed bbox from current detections, then predict waypoints conditioned on Refer-CoT decision.
3. Candidate Catalog and Referring
A real-time detector runs on the current frame; results are sorted into indexed catalog $\mathcal{C}_T = \{\langle ped_1 \rangle, \ldots, \langle ped_K \rangle, \langle \text{NO\_EXIST} \rangle\}$. Each entry has an identifier token and bbox token. The LLM outputs a single Refer-CoT token selecting the matching target; $\langle \text{NO\_EXIST} \rangle$ handles target absence.
4. Waypoint Decoding and Training
After target selection, the LLM decodes tracking waypoints conditioned on the image-grounded decision. Training uses 1.3M navigation + 1.3M Refer-QA samples (1:1 mix) via SFT. Refer-QA enhances target identification by requiring the model to select the correct bbox entry given a description.
flowchart TD
A["Forward RGB Stream O_1:T"] --> B["Dual Encoder: SigLIP+DINOv2"]
B --> C["Grid Pooling: V_fine(64)+V_coarse(4)"]
C --> D["Sliding Window H-frame Visual Stream"]
D --> E["TVBI Token Injection
Historical target bbox geometry"]
E --> F["LLM Input: Visual+Language+Catalog"]
F --> G["Real-time Detector: YOLO11+ByteTrack"]
G --> H["Indexed bbox Catalog C_T"]
H --> I["Refer-CoT: Select Target bbox"]
I --> J["Waypoint Decoding: Tracking Trajectory"]
J --> K["Tracking Output + Target Memory Update"]
style E fill:#e1f5fe
style I fill:#fff3e0
style K fill:#e8f5e9
Experimental Results
EVT-Bench Quantitative Comparison
| Method | Params | STT SR↑ | DT SR↑ | AT SR↑ |
|---|---|---|---|---|
| EVT (GroundingDINO) | – | 24.4 | 3.2 | 17.4 |
| Uni-NaVid* | 7B | 53.3 | 31.9 | 15.8 |
| TrackVLA | 7B | 85.1 | 57.6 | 50.2 |
| TrackVLA++ | 7B | 86.0 | 66.5 | 51.2 |
| CoMaTrack* (3 cameras) | 3B | 92.1 | 74.2 | 57.5 |
| ReferTrack (Ours) | 4B | 89.4 | 73.3 | 74.1 |
ReferTrack with 4B params achieves 89.4% STT (vs 7B TrackVLA++'s 86.0%), DT 73.3% close to 3-camera CoMaTrack's 74.2%, and AT 74.1% far exceeding all methods (TrackVLA++ 51.2%, CoMaTrack 57.5%).
Ablation Study
| Configuration | SR↑ | TR↑ |
|---|---|---|
| Without TVBI | 68.5 | 76.2 |
| Without Refer-QA | 70.1 | 78.5 |
| Oracle TVBI | 75.8 | 83.2 |
| Full ReferTrack | 73.3 | 81.8 |
Table 1: Full EVT-Bench quantitative results.
Table 2/Figure 3: Ablation study and real-world qualitative results.
Figure 3: Real-world qualitative results. Left: Unitree Go2 follows target pedestrian; Right: humanoid robot deployment.
Figure 4: Expert tracking data examples.
Waypoint Decoding
After target selection, the LLM autoregressively decodes $A$ tracking waypoints $\mathbf{a}_{1:A}$. Each waypoint is a polar-coordinate displacement $(\Delta x, \Delta heta)$ relative to the current robot pose, discretized as tokens. The conditional decoding probability:
$$ P(\mathbf{a}_{1:A} \mid \mathcal{L}, \mathcal{V}_T, \mathcal{C}_T, ext{Refer-CoT}) = \prod_{i=1}^{A} P(a_i \mid a_{<i}, \mathcal{L}, \mathcal{V}_T, \mathcal{C}_T, ext{Refer-CoT}) $$Training Objective
Navigation and Refer-QA jointly trained with SFT next-token prediction loss:
$$ \mathcal{L}_{ ext{SFT}} = -\sum_{t=1}^{L} \log P(y_t \mid y_{<t}, \mathbf{x}) $$The visual stream is mapped to LLM latent space via MLP projector:
$$ E_{1:T}^V = \mathcal{P}_{ ext{vision}}(\mathcal{V}_T) $$Candidate catalog bbox tokens encoded via shared MLP:
$$ E_{ ext{bbox}}^{(k)} = \mathcal{P}_{ ext{bbox}}(b_T^{(k)}) $$TVBI Ablation Analysis
TVBI token design has key decisions: (1) current frame gets no bbox injection — forcing the model to use historical TVBI cues and raw visual features for spatio-temporal grounding; (2) absence sentinel $[0,0,0,0]$ — explicitly distinguishing visible vs invisible target frames; (3) shared $\mathcal{P}_{ ext{bbox}}$ encoder — ensuring consistent bbox geometry representation across catalog and history contexts. Ablation shows removing TVBI drops DT SR from 73.3% to 68.5% (-4.8pp); Oracle TVBI (ground-truth bbox) achieves 75.8%, indicating headroom for TVBI quality improvement.
Implementation Details
Built on Qwen3-4B LLM backbone with SigLIP+DINOv2 dual encoders. Indexed catalog from YOLO11+ByteTrack, max $K$ candidates. Sliding window $H$ frames, grid pooling 64 fine + 4 coarse tokens. Training on 8×A100 80GB, lr $2 imes 10^{-5}$, batch 128, 3 epochs. Inference ~150ms/frame (detection+LLM forward). Real-world deployment on Unitree Go2 and humanoid robots validates robust sim-to-real transfer under lighting changes, occlusion, and pedestrian interference.
Limitations
- External detector dependency: Target identification relies on YOLO11+ByteTrack detection quality; detection failures may omit targets from the catalog.
- Single-view limitation: Only forward camera used; targets behind/beside the agent are invisible, relying on historical memory rather than real-time perception.
- Candidate count limit: Catalog limited to $K$ candidates (sorted by bbox area); small targets in crowded scenes may be omitted.
Conclusion and Outlook
ReferTrack aligns EVT target identification with image-space detections through the referring-then-tracking paradigm, with TVBI tokens preserving target motion cues and Refer-QA co-training enhancing identification. With 4B params it surpasses 7B baselines on single-view and leads significantly on ambiguity tracking, with real-world deployment on legged and humanoid robots validating sim-to-real transfer.
Key Insight: "Moving target identification from abstract spatial latents to image space — selecting one token from an indexed bounding box catalog — makes reasoning supervisable, interpretable, and naturally aligned with VLM grounding mechanisms."
