PAPER DEEP DIVE
SLAMFormer-$\infty$: Infinite SLAM Transformer for Unbounded Frontend and Backend Processing
We introduce the Infinite SLAM Transformer (SLAMFormer-$\infty$), the first geometric transformer capable of supporting both long-range frontend and backend processing without an explicit distance bound. Instead of relying on a first-frame-anchored formulation, SLAMFormer-$\infty$ employs memory conditions to define flexible coordinate systems and scales for input frames, enabling more expressive structural conditioning. Built upon this formulation, the frontend preserves efficient local computation, while the backend jointly optimizes long-range trajectories and scene geometry in a globally consistent manner. Experimental results demonstrate that SLAMFormer-$\infty$ achieves superior or highly competitive performance in both trajectory estimation and scene reconstruction across large-scale datasets. Notably, SLAMFormer-$\infty$ generalizes to extremely long trajectories, successfully operating on sequences exceeding $17\mathrm{km}$.
One-Line Summary
SLAMFormer-$\infty$ is the first geometric transformer capable of supporting both long-range frontend and backend processing without an explicit distance bound, leveraging memory conditions to free itself from first-frame coordinate anchoring and achieving globally consistent pose estimation and dense reconstruction on urban driving sequences exceeding 17 km.
1. Background and Motivation
Simultaneous Localization and Mapping (SLAM) is a cornerstone of autonomous systems—a robot must localize itself and construct maps of previously unknown environments in real time. Among all SLAM paradigms, monocular SLAM is particularly important for its simplicity and low hardware cost: requiring only a single RGB camera for autonomous perception and navigation. However, monocular SLAM faces severe challenges: scale ambiguity, tracking loss under rapid motion, cumulative drift over large-scale scenes, and quality bottlenecks in dense reconstruction.
The evolution of monocular SLAM follows a clear generational trajectory. Early methods were sparse feature-based, such as ORB-SLAM2, focusing primarily on camera trajectory estimation. While effective for localization, these approaches captured only limited scene geometry. To recover dense 3D structure, subsequent works introduced dense bundle adjustment or learning-based depth prediction. In recent years, neural scene representations and geometric foundation models have significantly advanced monocular SLAM: Gaussian Splatting-based methods demonstrated the potential of neural rendering representations for monocular dense reconstruction; geometric transformers like DUSt3R, MASt3R, and VGGT transformed multi-view geometry regression from hand-crafted matching and triangulation into end-to-end transformer prediction.
SLAM-Former further introduced transformers into the full SLAM pipeline—frontend tracking, mapping, and backend optimization integrated into a single model, achieving trajectory-geometry joint refinement through global attention without loop closure detection. This was a significant advance, but it has a fundamental limitation: fully data-driven approaches depend on the trajectory distribution of training data, and long-range performance is bounded by the scale and trajectory range of the training data. When sequence length exceeds the training distribution, SLAM-Former's performance degrades sharply.
On the other hand, MASt3R-SLAM and VGGT-SLAM/VGGT-Long address long-range consistency through pose-centric optimization. Their backends primarily refine camera poses while leaving scene geometry largely fixed. As a result, trajectory correction and geometric reconstruction are decoupled rather than jointly optimized within a unified framework. VGGT-Long extends to long sequences through local submap construction and global pose alignment, but its geometric reconstruction quality degrades noticeably in long-range scenarios and even collapses entirely at city scale.
The starting point of SLAMFormer-$\infty$ is precisely the intersection of these two fundamental limitations: can we preserve a single transformer model while both freeing ourselves from the training data's constraint on trajectory length and achieving joint pose-geometry optimization? The answer is yes—through the memory condition mechanism that redefines the coordinate system, enabling the transformer to operate within a bounded attention context while retaining long-range pose and geometric information.
2. Preliminaries
Understanding SLAMFormer-$\infty$ requires several core concepts. First is the Geometric Transformer—a class of models that transforms multi-view 3D perception from hand-crafted correspondence and optimization into transformer-driven geometry regression. DUSt3R predicts dense pointmaps from image pairs, MASt3R strengthens pairwise 3D matching, and VGGT extends this paradigm to feed-forward multi-view reconstruction, jointly predicting cameras, depth, and pointmaps.
Second is SLAM-Former's basic formulation. SLAM-Former models SLAM as inference within a single transformer:
$$p(\mathcal{X}, \mathcal{P} \mid \mathcal{I})$$
where $\mathcal{I}$ is the input image sequence, $\mathcal{X}$ is the camera trajectory ($\mathbf{g}_n \in SE(3)$), and $\mathcal{P}$ is the scene geometry representation. The frontend operates causally for incremental tracking, and the backend performs global refinement over the full history. This formulation unifies traditional SLAM's multiple modules into an end-to-end learning framework, but its long-range performance is limited by the distribution range of training trajectories.
Finally, the concept of memory conditions. Unlike the first-frame-anchored global coordinate system, memory conditions use condition chunks $(\mathcal{I}_C, \mathcal{X}_C)$ to define a local reference coordinate system, allowing the transformer to operate within a bounded context while indirectly preserving long-range geometric relationships through condition information. This design is key to SLAMFormer-$\infty$'s unbounded processing capability.
3. Methodology
3.1 Problem Formulation
Given a streaming monocular image sequence $\mathcal{I}_{1:N} = \{\mathbf{I}_1, \dots, \mathbf{I}_N\}$, the goal of SLAM is to estimate in real time both the camera trajectory $\mathcal{X}_{1:N} = \{\mathbf{g}_1, \dots, \mathbf{g}_N\}$ ($\mathbf{g}_n \in SE(3)$) and the scene geometry representation $\mathcal{P}_{1:N} = \{\mathbf{P}_1, \dots, \mathbf{P}_N\}$, while maintaining both accurate incremental tracking and global geometric consistency.
SLAM-Former formulates this problem as inference within a transformer, with the frontend operating causally for incremental tracking and the backend performing global refinement over the full history:
$$\text{Frontend: } M_n = f_\theta^f(I_n, M_{1:n-1})$$
$$\text{Backend: } \hat{M}_{1:n} = f_\theta^b(M_{1:n})$$
where $\mathcal{M}$ denotes map token representations, and poses and geometry are decoded via a head function: $(\mathcal{X}, \mathcal{P}) = f_\psi(\mathcal{M})$. However, this formulation is fundamentally constrained by the distribution of training trajectories, preventing effective generalization to long-range sequences.
3.2 SLAMFormer-$\infty$'s Conditional Design
To overcome this limitation, SLAMFormer-$\infty$ introduces memory conditions, redefining the model as:
$$p(\mathcal{X}, \mathcal{P} \mid \mathcal{I}, \mathcal{I}_C, \mathcal{X}_C)$$
where the memory condition $(\mathcal{I}_C, \mathcal{X}_C)$ defines a reference coordinate system. The key distinction from SLAM-Former is that SLAMFormer-$\infty$ performs both frontend and backend inference in a local coordinate system defined by the condition, rather than in a global coordinate system. This enables the transformer to process arbitrarily long sequences within a bounded attention context.
Specifically, given a keyframe graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$, SLAMFormer-$\infty$ operates on local neighborhoods $\mathcal{N}$:
$$\text{Conditional Frontend: } M_n = f_\theta^f(I_n, I_{n-k:n-1}, C_j \in \mathcal{N}(n-k))$$
$$\text{Conditional Backend: } \hat{M}_{n-w:n} = f_\theta^b(I_{n-w:n}, C_j \in \mathcal{N}(n-w))$$
where $\mathcal{C}_\cdot$ denotes neighborhood conditioning context $(\mathcal{I}_\cdot, \mathcal{X}_\cdot)$, and $w \in \mathcal{W}(n)$ is the nearest anchor retriever from the window set $\mathcal{W}(n)$. In the frontend, $\mathcal{I}_{n-k:n-1}$ and $\mathcal{C}$ assist $\mathcal{I}_n$ with KV caches obtained from previous backend and frontend processing. However, this relative formulation prevents direct global full-attention inference as in SLAM-Former, motivating an iterative backend for global consistency.
Figure 2: SLAM Transformer Comparison. (a) SLAM-Former single-model global consistent pose and map. (b) VGGT-Long with optimized long-range poses and stitched maps. (c) SLAMFormer-$\infty$ retains the single model while obtaining long-range global consistent pose and map with a conditional design.
3.3 Pose-Geometry Graph Optimization (PGGO)
SLAMFormer-$\infty$'s backend jointly processes poses and geometries, formulating the task as Pose-Geometry Graph Optimization (PGGO). PGGO operates over a pose-geometry interaction graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ with node set $\mathcal{V} = \mathcal{X} \cup \mathcal{P}$, where $\mathcal{X}$ and $\mathcal{P}$ denote pose nodes and geometry nodes. The edge set $\mathcal{E} = \mathcal{E}_\mathcal{X} \cup \mathcal{E}_\mathcal{P}$, where $\mathcal{E}_\mathcal{X}$ connects poses explicitly with relative pose constraints, while $\mathcal{E}_\mathcal{P}$ implicitly captures geometric correlation through transformer attention.
For each frame $n$, the joint state variable is defined as:
$$\mathbf{x}_n = (\mathbf{g}_n, \mathbf{P}_n)$$
Given an initialization $\tilde{\mathbf{x}}_n^0 = (\tilde{\mathbf{g}}_n^0, \tilde{\mathbf{P}}_n)$ obtained from pose graph optimization or frontend prediction, the goal is to jointly refine poses and geometries for global consistency:
$$\mathbf{x}^* = \arg\min_{\{\mathbf{x}_n\}} \sum_{n} \left\|\mathbf{x}_n - f_\psi \circ f_\theta(\mathcal{I}_n, \mathcal{C}_{n \in \mathcal{N}_{n-w}})\right\|^2 \tag{1}$$
The core idea of this optimization objective: for each node, the transformer's predicted pose-geometry joint output should be consistent with the current estimate. By iteratively minimizing this residual, globally consistent trajectory and scene geometry are achieved. To solve this optimization, iterative neural refinement is used:
$$\hat{\mathbf{x}}_n^{k+1} = f_\psi \circ f_\theta(\mathcal{I}_n, \mathcal{C}_{n \in \mathcal{N}_{n-w}})$$
followed by direct pointmap updates with damped pose updates:
$$\tilde{P}_n \leftarrow \hat{P}_n^{k+1}, \quad \tilde{g}_n^{k+1} = \exp\left[(1-\alpha)\log(\tilde{g}_n^k) + \alpha\log(\hat{g}_n^{k+1})\right]$$
where $\alpha$ is the damping factor. The damped update operates on the Lie group structure of $SE(3)$: through the logarithmic map to the Lie algebra, linear combination in the tangent space, then back to the Lie group via the exponential map. This design avoids numerical instability that direct pose replacement might cause, while allowing new predictions to correct historical estimates in a controlled manner. Written compactly as $\tilde{x}_n^{k+1} = h_\theta(\tilde{x}_n^k)$.
Figure 3: The SLAM pipeline. Frontend detects keyframes and provides online tracking. After a fixed number of keyframes, local backend refines the most recent window. When loop detection triggers or at sequence end, PGGO is applied with the same transformer function.
3.4 SLAM at Test Time
At test time, SLAMFormer-$\infty$ performs streaming reconstruction and pose estimation from RGB sequences $\{\mathbf{I}_t\}_{t=1}^T$ with keyframes $\{\mathbf{I}_n\}_{n=1}^N$. The system maintains a global graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ where each node $\mathbf{x}_n$ stores $(\mathbf{g}_n, \mathbf{P}_n)$.
Frontend: Each incoming frame $\mathbf{I}_i$ is either associated with an existing keyframe or inserted as a new node $\mathbf{I}_n$. Edges are updated based on the local window $\mathcal{W}(n)$. The new keyframe $\mathbf{I}_n$ is tracked using the method in Section 3.2 to obtain $\mathcal{M}_n$, assisted by previous KV caches.
Local Backend: Every $c_w$ keyframes, a local backend is triggered, using the conditional backend formula to update $\hat{\mathcal{M}}_{n-c_w:n}$, improving short-range pose and geometry consistency.
Global Backend: When loop detection is triggered or after the last frame, given the graph initialization from pose graph or frontend prediction, SLAMFormer-$\infty$ performs PGGO for global refinement: iterating from $k=1$ to $K$, for each node $\mathbf{x}_n \in \mathcal{V}$, $\tilde{\mathbf{x}}_n^{k+1} = h_\theta(\tilde{\mathbf{x}}_n^k)$. These yield a globally consistent reconstruction induced by the learned SLAMFormer-$\infty$ prior.
3.5 Training Strategy
SLAMFormer-$\infty$ is trained with four shared-weight modes that differ only in attention masks and conditioning patterns, matching test-time frontend, backend, and fine-stage inference. Training variants are initialized from a pretrained SLAM-Former model and adapted to different scene scales and motion statistics. The indoor variant uses 12-frame clips at long side 518; the outdoor variant uses 36-frame clips at long side 224. Both variants are trained for 10 epochs on 48 A100 GPUs.
Figure 4: Four training modes of SLAMFormer-$\infty$. In each mode, image tokens are fed into the transformer backbone $f_\theta$ with L layers of frame attention and various inter-frame attentions. Pose and pointmap are regressed by heads $f_\psi$.
Mode 1 trains online tracking and mapping: the first two frames initialize a local coordinate system with full attention, and later frames use causal inter-frame attention. Mode 2 trains backend refinement without memory condition: the first half of the clip uses full attention, and the second half is decoded causally from the refined prefix. Mode 3 trains backend refinement with memory condition: using detached Mode-2 predictions, a prefix span $[s, N/2)$ is aligned to an anchor and encoded as pose-injected frames for the remaining target segment. Mode 4 trains the fine stage with memory condition: conditioning the middle target chunk on front and back neighboring condition chunks.
flowchart TB
subgraph FE["Frontend: Causal Tracking"]
I1["Image I_n"] --> FK["Frame Attention L layers"]
KV1["KV Cache from prev"] --> FK
C1["Memory Condition
(I_C, X_C)"] --> FK
FK --> M1["Map Tokens M_n"]
M1 --> HEAD1["f_psi: Pose + Pointmap"]
end
subgraph LB["Local Backend: every c_w frames"]
M2["Map Tokens M_{n-w:n}"] --> FB["Backend Attention"]
C2["Memory Condition"] --> FB
FB --> M2H["Refined M_hat"]
end
subgraph GB["Global Backend: PGGO"]
PG["Pose Graph Init"] --> ITER["Iterative h_theta
k=1..K"]
ITER -->|damped update| XN["x_n = (g_n, P_n)"]
XN -->|converged?| ITER
XN -->|yes| OUT["Global Consistent
Trajectory + Geometry"]
end
FE --> LB
LB -->|loop detected| GB
Above: SLAMFormer-$\infty$'s three-stage processing flow. Frontend causal tracking, local backend short-range refinement, global backend PGGO iterative joint optimization of pose and geometry.
4. Experimental Analysis
4.1 Large Outdoor Scenes
On the KITTI Odometry dataset (Table 1), SLAMFormer-$\infty$ improves the average ATE RMSE from VGGT-Long's 26.358m to 23.011m. The improvement is similarly significant when excluding the high-speed sequence 01. Notably, many competing methods (CUT3R, Fast3R, VGGT) fail on multiple sequences due to out-of-memory (OOM) errors, while SLAMFormer-$\infty$ successfully runs on all sequences.
Table 1: KITTI Odometry tracking results (ATE RMSE [m], lower is better). LC denotes loop closure; OOM/TL denote CUDA out-of-memory/tracking lost. SLAMFormer-$\infty$ achieves the best average without calibration.
On the Waymo Open Dataset (Table 2), SLAMFormer-$\infty$ further reduces the average ATE RMSE from VGGT-Long's 1.996m to 1.813m across diverse urban driving scenarios (varying speeds, lengths, and traffic densities). On several specific segments the improvement is even larger, e.g., segment 371159869 from 3.452m to 1.464m.
| Method | Calibration | Reconstruction | KITTI Avg. (m) | Waymo Avg. (m) |
|---|---|---|---|---|
| ORB-SLAM2 (w/ LC) | Required | Sparse | 54.816 | - |
| DROID-SLAM | Required | Dense | 100.278 | 4.396 |
| MASt3R-SLAM | Free | Dense | - | 5.560 |
| VGGT-Long | Free | Dense | 26.358 | 1.996 |
| SLAMFormer-$\infty$ | Free | Dense | 23.011 | 1.813 |
For dense pointmap reconstruction (Table 3, Waymo), SLAMFormer-$\infty$ improves over VGGT-Long across all metrics: accuracy from 1.182 to 0.949, completeness from 2.860 to 2.777, and Chamfer distance from 2.021 to 1.863. This demonstrates that PGGO's joint optimization not only improves poses but substantially enhances scene geometry reconstruction quality—precisely the core advantage of SLAMFormer-$\infty$ over pose-only optimization methods like VGGT-Long.
| Method | Accuracy ↓ | Completeness ↓ | Chamfer ↓ |
|---|---|---|---|
| DROID-SLAM | 1.201 | 8.540 | 4.870 |
| MASt3R-SLAM | 3.772 | 3.177 | 3.474 |
| CUT3R | 3.884 | 6.801 | 5.343 |
| VGGT-Long | 1.182 | 2.860 | 2.021 |
| SLAMFormer-$\infty$ | 0.949 | 2.777 | 1.863 |
Figure 1 shows the qualitative comparison at city scale: on a self-collected 17 km urban driving sequence, VGGT-Long's global pose alignment is effective but leaves local geometry largely unrefined, and the large-scale map collapses; SLAMFormer-$\infty$ jointly optimizes pose and dense geometry, maintaining a consistent large-scale map.
Figure 1: City-scale reconstruction visualization. VGGT-Long performs global pose alignment but leaves local geometry largely unrefined; SLAMFormer-$\infty$ jointly optimizes pose and dense geometry. On a self-collected 17 km urban drive, SLAMFormer-$\infty$ maintains a consistent large-scale map where VGGT-Long collapses.
4.2 Small Indoor Scenes
On indoor benchmarks (TUM RGB-D, 7-Scenes, Replica), SLAMFormer-$\infty$ remains comparable to state-of-the-art indoor SLAM systems while preserving calibration-free dense reconstruction. On 7-Scenes, compared to VGGT-SLAM, ATE RMSE decreases from 0.068m to 0.046m, and accuracy/completeness/Chamfer improve from 0.054/0.060/0.057 to 0.029/0.049/0.039.
However, SLAM-Former achieves the overall best performance on indoor benchmarks, benefiting from its fully end-to-end learned architecture. SLAMFormer-$\infty$ is designed primarily for global optimization over much longer trajectories, and its global optimization is not learned end-to-end. Therefore, on short indoor benchmarks, it may still underperform fully data-driven models tailored to the same distributions, while prioritizing global optimization for unbounded sequences. This tradeoff is a deliberate design choice—sacrificing short-range peak performance for long-range generalization capability.
4.3 Ablation: Before and After Fine Stage (PGGO)
The ablation compares SLAMFormer-$\infty$ with and without the fine stage. On Replica, the fine stage reduces ATE RMSE from 0.061m to 0.052m, and reconstruction accuracy/completeness improve from 6.11/3.39 to 6.00/3.33. TUM RGB-D and 7-Scenes show the same trend, indicating that the fine stage improves local geometric consistency, though numerical margins remain limited. However, the qualitative effect is more evident: Figure 5 shows cleaner point-map surfaces and reduced local drift after the fine stage, making the visual improvement more noticeable than the score differences alone suggest.
Table 2: Waymo tracking results. ATE RMSE [m] on nine urban driving segments (lower is better).
5. Discussion
The core innovation of SLAMFormer-$\infty$ lies in introducing coordinate conditioning into geometric transformers—a shift in perspective with profound implications. Traditional geometric transformers (VGGT, Fast3R) assume all views fit into a finite attention context, with coordinate behavior tied to training-time sequence ranges. SLAMFormer-$\infty$ relaxes this assumption: fixed condition chunks define local reference geometry, and the transformer operates in a bounded context while indirectly preserving long-range pose and geometry through condition information. This design enables the transformer to process sequences far exceeding the training distribution.
PGGO's design also represents an important methodological shift. Traditional backend optimization (e.g., pose graph optimization, bundle adjustment) optimizes poses with fixed geometry, or vice versa. PGGO treats poses and geometry simultaneously as graph nodes, jointly refining them through transformer attention, breaking this decoupling. The damped update mechanism ensures numerical stability of the iterative process while allowing new predictions to correct historical estimates in a controlled manner. This "learned prior plus iterative refinement" paradigm shares structural similarities with denoising iterations in diffusion models.
The shared-weight training design is also noteworthy. The four modes differ only in attention masks and conditioning patterns but share the same transformer backbone. This means the model learns a unified geometric reasoning capability, adapting to the different requirements of frontend tracking, backend refinement, and fine-stage through different attention configurations. This design parallels the shared encoder concept in metric learning.
6. Limitations
Author-stated limitation: SLAMFormer-$\infty$'s PGGO requires a predefined graph input, which comes from the frontend and loop detection. The quality of graph connectivity affects performance, and this connection structure is not learned from data. This means if the graph structure constructed by the frontend or loop detection is suboptimal (e.g., poor keyframe selection, missed loop closures), PGGO's optimization effectiveness will be limited. In contrast, SLAM-Former implicitly constructs frame connections without relying on external graph structures.
Independent assessment: SLAMFormer-$\infty$ underperforms SLAM-Former and other fully data-driven methods on short-range indoor scenes. This is because the global optimization is not end-to-end learned and cannot fully exploit data-driven advantages on short sequences within the training distribution. Additionally, while the paper demonstrates 17 km sequence processing capability, training uses only 12-frame (indoor) or 36-frame (outdoor) clips—the generalization mechanism from short-clip training to long sequences requires deeper analysis. The outdoor variant uses 224-pixel long-side resolution, which may be insufficient for fine-grained reconstruction of distant scenes. Finally, the paper does not report system-level evaluation of inference speed and memory consumption—important missing information for practical deployment.
7. Conclusion and Outlook
SLAMFormer-$\infty$ addresses the fundamental limitation that SLAM transformers cannot learn to tackle unbounded-long distance sequences. Based on the memory-condition design, SLAMFormer-$\infty$ achieves efficient online dense reconstruction in the frontend and joint trajectory-geometry optimization in the backend for unbounded-long sequences. The method demonstrates better performance across large-scale datasets and exhibits generalization to extremely long trajectories exceeding 17 km.
From a broader perspective, SLAMFormer-$\infty$ represents an important step in SLAM research's evolution from "modular pipelines" toward "unified transformers." The conditioning design provides a general framework enabling geometric transformers to extend to arbitrarily long sequences while preserving single-model unification. Future work may explore: learning graph connectivity structures from data to eliminate reliance on external graph construction; unifying indoor and outdoor variants into a single model; and achieving real-time inference on mobile devices.
SOURCE LINKS
