Describe Anything, Anywhere, at Any Moment (DAAAM)

Paper Information

Title: Describe Anything, Anywhere, at Any Moment (DAAAM)
Authors: Nicolas Gorlo, Lukas Schmid, Luca Carlone (MIT)
Paper: arXiv:2512.00565
Project: nicolasgorlo.com/DAAAM_25
Code: github.com/MIT-SPARK/DAAAM (open-source)

One-Line Summary

DAAAM introduces a real-time spatio-temporal memory framework that decouples geometric tracking from semantic annotation via optimization-based frame selection and batch inference, building hierarchical 4D scene graphs with detailed natural-language descriptions grounded in 3D at 10Hz.

Background and Motivation

Robotics and AR applications require perception systems that can answer spatio-temporal queries such as "where and when did you last see the red screwdriver?" or "can you go grab the component we assembled last week?" This demands internal memory representations that simultaneously support spatial reasoning and task planning, cover long time horizons and large environments, and can be built in real-time with limited computation.

Existing methods face a fundamental tradeoff. The first paradigm—metric-semantic maps, particularly 3D scene graphs—grounds semantic entities and relationships in 3D reconstructions. However, the pursuit of expressive scene descriptions is fundamentally at odds with real-time mobile computation. Current methods either use fast but closed-vocabulary segmentation or embeddings (lacking semantic detail) or query large multimodal models per-object for detailed but prohibitively expensive open-vocabulary annotations.

The second paradigm leverages multimodal LLMs to generate scene representations from natural language descriptions. These approaches annotate individual frames or video sequences and store them in databases for retrieval. While producing richer representations, annotations are organized by frame rather than by content, lacking 3D geometric grounding and spatio-temporal consistency. The same object observed from different viewpoints may be described redundantly without reconciliation.

DAAAM's starting point: can we simultaneously achieve semantic richness, geometric grounding, and real-time performance? The answer is yes—by decoupling geometric tracking from semantic annotation, using an optimization-based frame selection algorithm to pick optimal viewpoints, and invoking large description models in batch, all three goals can be unified.

Figure 1: DAAAM system overview. Given RGB-D input, DAAAM incrementally constructs a hierarchical 4D scene graph with detailed annotations as a scalable spatio-temporal memory for LLM agents.

Preliminaries

3D Scene Graphs represent semantic entities (objects, places, buildings) and their relationships as a topological graph grounded in 3D. Hierarchical scene graphs aggregate from low-level object nodes upward to regions, floors, and buildings. Khronos is the real-time 4D mapping frontend used by DAAAM, extracting temporally consistent object fragments and reconstructing their 3D shape and position.

Describe Anything Model (DAM) is a localized captioning model that generates detailed natural-language descriptions for segmented image regions. DAAAM's key insight: while DAM is slow per-object, frame selection plus batch inference can improve efficiency by an order of magnitude, making it viable for real-time systems.

Method

Overall Architecture

DAAAM comprises five modules: (A) Active Window and Real-time SG Construction, (B) Prompt Frame Selection, (C) Semantic Lifting, (D) Place Extraction, (E) Global Optimization and Region Clustering. Modules B and C run in a parallel thread, decoupled from real-time geometric processing—this is the key design enabling real-time performance.

Figure 2: DAAAM pipeline. RGB-D stream is segmented and tracked, then fed to Khronos for 4D mapping. The semantic lifting thread selects frames via optimization, batches them to DAM, and integrates descriptions back into the 4D scene graph.

(A) Active Window and Real-time SG Construction

The system receives RGB-D frame $I_t^{\text{rgb-d}}$ at each timestep $t$, segments it using Fast-SAM into fragments $s_j^t \in \mathbb{R}^{H \times W}$, and tracks them across frames using Bot-Sort. Each track creates an object fragment $o_j^{0 \ldots T_j}$ with $T_j$ observations. Khronos lifts each fragment to 3D and reconstructs its shape and position (including temporal changes for dynamic objects). Geometric segmentation, tracking, and reconstruction run at the sensor rate of 10Hz.

(B) Optimization-Based Frame Selection

This is one of DAAAM's core innovations. Since extracting detailed descriptions is expensive, the system does not call DAM for every object in every frame. Instead, it selects only the best-viewpoint frame for each fragment and batch-annotates. Frame selection is formulated as a two-step optimization problem.

Step 1: Set cover. Within time window $w_t = [t_{\text{start}}, t_{\text{start+m}}]$, let $\mathcal{O} = \{o_1^w, \ldots, o_m^w\}$ be tracked fragments and $\mathcal{F}^w$ all frames. For each pair $(f_i, o_j^w)$, define visibility indicator $v_{ij} \in \{0,1\}$ and view quality score $q_{ij} \in [0,1]$. First, find the minimum number of frames:

$$K^{\star} = \min_{\mathcal{S} \subseteq \mathcal{F}^w} \quad |\mathcal{S}| \quad \text{s.t.} \quad \forall o_j^w \in \mathcal{O}: \exists f_i \in \mathcal{S} \text{ with } v_{ij}=1$$

solved via a greedy algorithm. Step 2: Binary linear program. Given $K^{\star}$, solve:

$$\max_{x,y} \quad \sum_{i=1}^{n}\sum_{j=1}^{m} q_{ij} \cdot y_{ij}$$

$$\text{s.t.} \quad \sum_{i=1}^{n} x_i = K^{\star} + \epsilon, \quad \sum_{i=1}^{n} y_{ij} = 1$$

$$y_{ij} \leq x_i, \quad y_{ij} \leq v_{ij}, \quad x_i \in \{0,1\}, \quad y_{ij} \in \{0,1\}$$

where $x_i$ indicates whether frame $f_i$ is selected, $y_{ij}$ whether fragment $o_j^w$ is assigned to frame $f_i$, and $\epsilon$ is a slack parameter (set to 1). The objective maximizes total quality, constrained by: (i) total selected frames = $K^{\star}+\epsilon$, (ii) no assignment to unselected frames, (iii) each fragment assigned to exactly one visible frame.

The quality score $q_{ij}$ combines position and size:

$$q_{ij} = \alpha \cdot q_{ij}^{\text{pos}} + (1-\alpha) \cdot q_{ij}^{\text{size}}$$

The position score $q_{ij}^{\text{pos}}$ uses entropy of normalized coordinates, favoring centrally-located objects. The size score $q_{ij}^{\text{size}}$ uses a hyperbolic tangent that saturates for large objects while penalizing those below minimum area $A_{\text{min}}$. Parameter $\alpha = 0.5$.

(C) Semantic Lifting

Selected image-fragment pairs are batched to DAM, generating detailed descriptions for all fragments in a single forward pass. DAAAM extends DAM with a batch inference strategy: bundling multiple frames and masks into a single tensor, minimizing redundant computation and maximizing parallelization. This enables real-time operation while still leveraging large models like DAM.

Each fragment also receives a CLIP feature and a sentence embedding feature (also batched), aiding semantic search, clustering, summarization, and reconciliation of repeatedly observed objects. The frame selection naturally minimizes frames passed to DAM while processing many masks per frame, further improving batch inference efficiency.

(D) Place Extraction

Beyond objects, DAAAM extracts "place" nodes $p_j$ from the background. Rather than Voronoi diagrams or sampling, places are extracted based on ground traversability: convolving a robot bounding box with the local volumetric occupancy map, squashing along the Z-axis, then tessellating into largest traversable rectangles (max 2m each). For semantic lifting, each place is projected to ground, mapped to frames covering it, and descriptions assigned by majority voting.

Appendix: Place extraction pipeline — from 3D mesh to traversability analysis.

Appendix: Traversability map after convolving robot bounding box.

Appendix: Largest inscribed rectangles partitioning traversable area.

Appendix: Final place graph with topological connections.

(E) Global Optimization and Region Clustering

To achieve a spatio-temporally consistent 4D SG, DAAAM reconciles repeatedly observed objects in a global optimization step. Merged object descriptions are concatenated into a history with timestamps. Regions $R_i$ are extracted by assigning edge weights as cosine distance of semantic features and applying Hydra's most-stable-clique algorithm. Object nodes are assigned to nearest clusters. Region descriptions use farthest-point sampling from the mean, summarized by an LLM.

Retrieval-Augmented Reasoning

DAAAM uses a tool-calling agent for natural language queries, with tools to: (a) retrieve objects via semantic search over description embeddings, (b) retrieve region information, (c) retrieve agent information. Retrieved data includes spatial and temporal information for each 4D SG node.

flowchart TB
    A[RGB-D Input 10Hz] --> B[Fast-SAM Segmentation]
    B --> C[Bot-Sort Tracking]
    C --> D[Khronos 4D Map]
    D --> E[Active Window SG]
    E --> F[Parallel Thread]
    F --> G[Frame Selection Optimization]
    G --> H[Batch DAM Inference]
    H --> I[CLIP + Sentence Embedding]
    I --> J[4D Scene Graph]
    J --> K[Region Clustering]
    K --> L[Hierarchical 4D SG]
    L --> M[Tool-calling Agent]
    M --> N[Spatio-temporal QA]

Experimental Results

Spatio-Temporal Question Answering

Evaluated on NaVQA (based on CODa dataset), with 210 QA samples across binary, spatial, and temporal questions, and short (1.2min), medium (4.4min), and long (12.3min) sequences. DAAAM performs strongly even against ReMEmbR+VILA1.5-13b, especially for long sequences and temporal reasoning, indicating that geometric structuring of spatio-temporal memory aids scene understanding.

The authors identified several limitations in NaVQA: in-context examples appearing in test sets, spatial annotations reflecting observation positions rather than actual 3D positions (favoring view-based methods), and noisy labels. They re-annotated with actual object positions and evaluated full sequences (up to 35.8min), creating the OC-NaVQA benchmark:

Method	Question Acc. ↑	Pos. Error [m] ↓	Temp. Error [min] ↓
ReMEmbR - NVILA-Lite-2B	0.432	53.466	2.287
ReMEmbR - NVILA-Lite-8B	0.463	55.894	4.106
Concept-Graphs	0.299	111.29	—
DAAAM (Ours)	0.711	41.75	1.792

On OC-NaVQA, DAAAM's 4D SG scales to 35.8min and 1.64km, improving question accuracy by 53.6%, position error by 21.9%, and temporal error by 21.6%. ConceptGraphs struggles with memory limitations from maintaining full point clouds.

Sequential Task Grounding (SG3D)

Evaluated on SG3D for grounding natural language instructions in 3D:

Method	s-acc [%]	t-acc [%]
Hydra + GPT	8.18	2.44
Hydra (GT Seg) + GPT	14.2	6.34
HOV-SG	8.98	1.95
ASHiTA	21.7	8.78
DAAAM (Ours) + GPT	22.16	11.22

DAAAM surpasses Hydra (even with ground-truth segmentation labels), highlighting the importance of detailed descriptions. It also outperforms ASHiTA, a specialized hierarchical task analysis method, by 27.8% on task grounding accuracy. Note that SG3D is semi-synthetic (based on HM3D), introducing a real-to-sim gap for DAM trained only on real data.

Ablation Studies

Configuration	Q Acc. ↑	Pos. Error [m] ↓	Temp. Error [min] ↓
DAAAM + GPT-5-mini	0.711	41.75	1.792
w/o DAM descriptions	0.776	50.05	2.396
w/o region clustering	0.707	48.93	3.58
w/o frame selection quality	0.627	49.92	1.678

Without DAM descriptions, question accuracy is higher (0.776 vs 0.711) but position and temporal errors are worse, suggesting explicit descriptions aid compositional reasoning for spatial/temporal queries while image crops better serve binary verification. Region clustering improves all metrics, especially temporal queries. Frame selection quality heuristic significantly improves spatial and QA accuracy.

Runtime Analysis

DAAAM runs at 10Hz on a single NVIDIA RTX 5090, matching the CODa sensor rate. The main bottleneck is input segmentation and tracking; the parallel semantic annotation thread only becomes a bottleneck in highly cluttered or fast-moving scenes. Mean annotation time per fragment is $0.18 \pm 0.03$s, enabling ~5.2 new fragments/second per worker—sufficient for mobile ground robots.

Figure 3: DAM inference speedup via batching. Dashed red: baseline (batch=1), solid blue: batch processing. Speedup of ~10x at batch size 128.

Method	Frame Rate [Hz]
Concept-Graphs	0.075
ReMEmbR NVILA-Lite-2B	4.9
ReMEmbR NVILA-Lite-8B	4.6
DAAAM (Ours)	11.6

Frame selection latency: $1.2 \pm 0.74$s; semantic lifting latency: $9.2 \pm 1.4$s per full batch. The ~10s delay is inherent to running large models, but for large-scale long-horizon decision-making, throughput matters more than latency—geometric information is always maintained in real-time.

Limitations

Author-stated limitation 1: DAM's training data is relatively modest (1.5M samples), so generated descriptions sometimes fail on out-of-distribution objects or uncommon visual features and may hallucinate towards the mean (e.g., predicting elevator doors with handles). As multimodal LLMs evolve rapidly, future more accurate localized description models should integrate well into DAAAM.

Author-stated limitation 2: At 5.2 fragments/second per worker on a desktop GPU, the annotation speed suffices for mobile ground robots but may be too slow for dynamic aerial robots or VR headsets. Smaller models can run on smaller hardware (still "comparatively large") or at higher throughput.

Independent assessment: DAAAM maintains full description histories in dynamic nodes, which may grow indefinitely over long multi-day operations. While the reconciliation step reduces redundancy, the lack of a bounded-memory summarization strategy is a scalability concern—acknowledged but unresolved by the authors.

Additionally, the frame selection quality heuristic $q_{ij}$ relies on hand-designed position and size components with fixed $\alpha=0.5$, without learned adaptive weighting. In specialized scenarios (highly symmetric environments, extreme lighting), this heuristic may be suboptimal.

Conclusion and Outlook

DAAAM overcomes computational constraints of large vision annotation models by decoupling geometric tracking from semantic annotation through optimization-based frame selection and batch inference, enabling real-time construction of hierarchical 4D scene graphs with detailed natural-language descriptions. It achieves state-of-the-art results in spatio-temporal question answering and sequential task grounding, with 10Hz real-time performance scalable to 35+ minutes and 1.5+ km. Code and data are open-source. DAAAM provides a practical foundation for embodied agents to understand and interact with complex, large-scale, dynamic environments over extended time horizons.

Future directions include integrating stronger localized description models, developing bounded-memory summarization strategies, learning frame selection quality scores rather than hand-designing them, and extending to higher-dynamics scenarios like aerial robots and VR headsets.

"Decouple geometric tracking from semantic annotation: don't make large models chase sensor speed—let sensor data wait for the model to be ready. That is the correct path to real-time 4D understanding."

---

Source: arXiv:2512.00565 · MIT-SPARK/DAAAM