PAPER DEEP DIVE
PRISM-VO: Scale-Aware Visual Odometry Using Photometric Plenoptic Bundle Adjustment
We introduce PRISM-VO, a novel pure optimization-based sparse photometric visual odometry framework for focused plenoptic cameras. The core of PRISM-VO is a novel photometric plenoptic bundle adjustment which jointly optimizes camera poses and inverse depth values of points in a sliding window. By combining geometric depth from a single plenoptic image with temporal multi-view constraints, PRISM-VO achieves accurate and drift-resilient motion estimation. Through explicit modeling of the plenoptic projection, PRISM-VO provides reliable metric-scale reconstructions, overcoming the scale ambiguity of monocular SLAM algorithms. Importantly, our approach relies solely on a single plenoptic sensor and avoids complex initialization, as depth priors are computed directly from plenoptic imaging. Experiments show that PRISM-VO outperforms the current state-of-the-art plenoptic visual odometry method on indoor and outdoor scenes. The proposed approach rivals other optimization- and learning-based methods while accurately and reliably recovering a metric scale of the scene. Project page: https://prism-vo.github.io/
1. Overview: Scale-Aware Visual Odometry Using Photometric Plenoptic Bundle Adjustment
PRISM-VO presents a novel pure optimization-based sparse photometric visual odometry framework for focused plenoptic cameras. The core innovation is photometric plenoptic bundle adjustment, which jointly optimizes camera poses and inverse depth values of points in a sliding window. By combining geometric depth from a single plenoptic image with temporal multi-view constraints, PRISM-VO achieves accurate and drift-resilient motion estimation. Through explicit modeling of the plenoptic projection, PRISM-VO provides reliable metric-scale reconstructions, overcoming the scale ambiguity of monocular SLAM. It relies solely on a single plenoptic sensor and avoids complex initialization — depth priors are computed directly from plenoptic imaging. It outperforms the state-of-the-art plenoptic VO method SPO on indoor and outdoor scenes and rivals optimization- and learning-based methods while accurately recovering metric scale.
2. Core Problem: Monocular Scale Ambiguity and the Plenoptic Advantage
3D reconstruction and motion estimation are essential for real-time localization in robotics, autonomous vehicles, drones, VR, and AR. Passive cameras offer high resolution, rich visual information, and a lightweight compact solution, but monocular cameras cannot recover absolute scale. Stereo, RGB-D, and ToF cameras address scale ambiguity but are constrained by stereo baseline, structured light range, or range-accuracy trade-offs. A plenoptic camera places a micro-lens array (MLA) between the main lens and sensor, simultaneously capturing spatial and angular information of the scene, producing multi-viewpoint micro-images with very wide depth of field. This makes it well suited for true-to-scale robust SLAM in compact systems. PRISM-VO combines micro-image disparity for absolute scale estimation with temporal disparity (large baseline) to improve tracking accuracy and robustness. The existing plenoptic VO method SPO only projects from the penultimate frame to the last without multi-frame bundle adjustment, making it less robust and more sensitive to drift.
3. Method: Plenoptic Direct Visual Odometry
3.1 Focused Plenoptic Camera Model
The method considers a focused plenoptic camera in Galilean mode, modeling the main lens as a thin lens and micro-lenses as pinholes. Key parameters: main lens focal length $f_L$, main lens to MLA distance $b_{L0}$, and MLA to sensor distance $B$. The virtual depth $v = B/b$ maps a 3D point $\mathbf{x}_C = [x_C, y_C, z_C]^T$ in camera coordinates to the 3D virtual image space $\mathbf{x}_V = [x_V, y_V, z_V]^T$, with projection denoted $\Pi_{\text{pl}}(\cdot)$. Virtual depth enables depth map generation without prior metric calibration. Each raw image is processed into a totally focused image, virtual depth map, and virtual depth uncertainty map.
3.2 Multi-Scale Image Representation and Point Selection
Pyramids are constructed for the totally focused image, virtual depth map, and uncertainty map. Inverse virtual depth is assumed normally distributed (proportional to micro-image disparity) and downsampled via variance-weighted averaging:
$$\bar{\rho}_{v,i+1} = \frac{\sum_{k \in N_x^{(i)}} \rho_{v,k} \cdot (\sigma_{\rho_{v,k}}^2)^{-1}}{\sum_{k \in N_x^{(i)}} (\sigma_{\rho_{v,k}}^2)^{-1}} \tag{1}$$Adaptive depth cutoff computes a percentile $p_a$ based on mean inverse virtual depth:
$$p_a = p_{\min} + w_{\text{cut}} \cdot (p_{\max} - p_{\min}), \quad w_{\text{cut}} = \frac{\rho_{v,\text{av}} - \rho_{v,f}}{\rho_{v,n} - \rho_{v,f}} \tag{2}$$Gradient-based scoring encourages orientation diversity; points with depth are weighted more heavily. Points without depth are also selected (lower weight) to ensure good distribution.
3.3 Photometric Plenoptic Bundle Adjustment
The state vector includes camera poses, inverse depths, and brightness parameters, estimated by minimizing a joint nonlinear least-squares problem. A point $\mathbf{x}_{V,i}$ is projected via the plenoptic camera model to target frame $j$:
$$\mathbf{x}_{V,j} = \Pi_{\text{pl}}\left(\mathbf{R} \cdot \Pi_{\text{pl}}^{-1}(\mathbf{x}_{V,i}) + \mathbf{t}\right) \tag{3}$$The photometric residual with affine brightness parameters $a_i, a_j, b_i, b_j$:
$$r^{\text{photo}} = \left(I_j[\Pi_{\text{pl}}(\mathbf{x}_{V,i})] - b_j\right) - \frac{e^{a_j}}{e^{a_i}}\left(I_i[\mathbf{x}_{V,i}] - b_i\right) \tag{4}$$The depth residual in inverse virtual depth space:
$$r^{\text{depth}} = \rho_v - \rho_v^{\text{meas}} \tag{5}$$The total energy is the Huber-norm weighted sum of photometric and depth terms:
$$E = \sum_k w_k^{\text{photo}} \|r_k^{\text{photo}}\|_\gamma + \eta \sum_l w_l^{\text{depth}} \|r_l^{\text{depth}}\|_\gamma \tag{6}$$where $w_l^{\text{depth}} = (\sigma_{\rho_{v,l}}^2)^{-1}$ so reliable depth measurements have greater influence. The scalar $\eta$ adaptively balances photometric and depth residuals based on their energy ratio, updated online.
4. Experiments
4.1 Comparison with Plenoptic VO Methods
Compared with SPO on 11 sequences from the [40] dataset. PRISM-VO achieves absolute scale error $d'_s \le 1.10$ on 8/11 sequences and $d'_s \le 1.30$ on 10/11. Rotation error $e_r < 4°$ on 8/10 sequences, with consistently smaller alignment error. PRISM-VO succeeds on one sequence where SPO fails, indicating greater robustness.
| Metric | Precision | PRISM-VO | SPO |
|---|---|---|---|
| Abs. scale $d'_s$ | ≤1.10 (high) | 8/11 | 7/11 |
| Abs. scale $d'_s$ | ≤1.30 (coarse) | 10/11 | 9/11 |
| Scale drift $e'_s$ | ≤1.05 (high) | 7/11 | 7/11 |
| Alignment $e_{align}$ | ≤2% (medium) | 6/11 | 6/11 |
| Rotation $e_r$ | ≤4° (coarse) | 8/10 | 8/10 |
4.2 Comparison with Monocular Methods and Ablation
Compared with DSO, ORB-SLAM3 (no loop closure), and DPVO. ORB-SLAM3 fails on all sequences due to limited field of view. PRISM-VO reliably recovers absolute scale with lower scale drift than DSO and on par with DPVO. On the LiFMCR dataset (7 sequences), PRISM-VO achieves the lowest translational RMSE on 6/7 sequences (millimeter-level) and best rotational RMSE on 4/7. Ablation study shows progressive improvement: pinhole baseline → plenoptic depth initialization (scale observable) → full PRISM-VO (variance-weighted depth).
| Sequence | PRISM-VO RMSEt(mm) | DSO RMSEt | DPVO RMSEt |
|---|---|---|---|
| 01 Plants | 41.66 | 59.32 | 98.79 |
| 04 Electronics | 10.53 | 150.89 | 268.47 |
| 05 | 9.32 | 239.16 | 219.66 |
5. Limitations
- Plenoptic-specific: The method is designed for focused plenoptic cameras and cannot be directly applied to conventional pinhole cameras or non-plenoptic sensors.
- Close-range advantage: Excels in close-range scenes where plenoptic depth is reliable; depth uncertainty increases with distance, potentially affecting accuracy.
- Computational efficiency: The pure optimization approach's real-time performance in large-scale scenes is limited by bundle adjustment window size, with higher deployment complexity than end-to-end learning methods.
6. Conclusion
PRISM-VO proposes sparse photometric plenoptic bundle adjustment that jointly optimizes plenoptic camera poses and environment point inverse depths in a sliding window. Three main contributions: (1) a sparse photometric plenoptic BA formulation jointly optimizing poses and points; (2) tight integration of the plenoptic camera model into both front-end tracking and back-end BA; (3) variance-weighted plenoptic depth residuals combined with temporal optimization for depth refinement. The core insight is that a plenoptic camera simultaneously captures spatial and angular information through its MLA — micro-image disparity provides absolute scale, temporal multi-view constraints provide tracking accuracy — and photometric plenoptic BA unifies both in a single optimization framework, with variance weighting giving reliable depth measurements greater influence and adaptive $\eta$ balancing the two residual types online. This enables a single plenoptic sensor to recover metric scale without complex initialization, outperforming existing plenoptic VO methods and rivaling mainstream monocular approaches.
flowchart TD
A["Raw plenoptic image"] --> B["Plenoptic camera processing"]
B --> C["Totally focused image"]
B --> D["Virtual depth map"]
B --> E["Virtual depth uncertainty map"]
C --> F["Multi-scale pyramids"]
D --> F
E --> F
F --> G["Front-end: photometric tracking"]
G --> H["Direct image alignment ref vs target"]
H --> I{"Keyframe?"}
I -->|Yes| J["Point selection: gradient + depth"]
I -->|No| K["Estimate current frame pose"]
J --> L["Back-end: plenoptic BA"]
K --> L
L --> M["Photometric residual r_photo"]
L --> N["Depth residual r_depth = ρv - ρv_meas"]
M --> O["Total energy E = Σw_photo‖r_photo‖ + ηΣw_depth‖r_depth‖"]
N --> O
O --> P["Variance weighting: w_depth = (σ²)^-1"]
P --> Q["Adaptive η online balance"]
Q --> R["Joint optimize poses + inverse depth"]
R --> S["Marginalize old keyframes"]
S --> T["Metric-scale 3D reconstruction"]