PAPER DEEP DIVE
BayesContact: Uncertain Pose Estimation via Visuo-Tactile Proposals and Simulation-based Inference
Contact-rich manipulation requires pose estimates that are often more accurate than what depth-only sensing provides. Existing methods, relying on vision and contact, employ costly offline training procedures that need to be retrained for new environments and geometries. We propose BayesContact, a Simulation-Based Inference framework for visuo-tactile pose estimation in peg-in-hole insertion. BayesContact maintains a particle belief over object pose and fuses depth observations with force/torque-derived contact evidence. We employ simulation based forward models to approximate these observation likelihoods. For each pose hypothesis, a renderer predicts depth measurements and a physics simulator predicts contact outcomes under guarded probing actions; both are scored against real observations to update the belief. The resulting multimodal belief also enables information-gain-based probing for active disambiguation. Across simulated geometries and real-robot experiments, BayesContact improves pose observability and insertion success over vision-only inference by 30%
BayesContact: Uncertain Pose Estimation via Visuo-Tactile Proposals and Simulation-based Inference
Paper: BayesContact: Uncertain Pose Estimation via Visuo-Tactile Proposals and Simulation-based Inference
Authors: Aditya Kamireddypalli, Matías Mattamala, João Moura, Russell Buchanan, Sethu Vijayakumar, Subramanian Ramamoorthy (University of Edinburgh / University of Waterloo)
Links: arXiv:2607.16123
One-Line Summary
BayesContact proposes a Simulation-Based Inference (SBI) framework for visuo-tactile pose estimation, maintaining a particle belief that fuses depth observations with force/torque contact evidence, using a renderer for depth prediction and physics simulator for contact prediction, improving pose observability and insertion success by 30% over vision-only methods in peg-in-hole tasks.
Background and Motivation
Contact-rich manipulation requires pose estimates more accurate than depth-only sensing provides. In peg-in-hole insertion, the robot must reason about hole pose indirectly through vision, proprioception, and contact. Depth sensing suffers from noise, occlusion, and geometric ambiguity — the most task-relevant surfaces may be hidden inside the hole.
Figure 1: BayesContact — inferring uncertain pose via rendering-based depth and physics-based contact likelihoods.
Existing methods rely on costly offline training requiring retraining for new geometries. BayesContact uses simulation-based forward models for online estimation without training data.
Method
Simulation-Based Bayesian Inference
Pose estimation as Bayesian inference over $\mathbf{x}$. Posterior:
$$ p(\mathbf{x}\mid\mathbf{o})=\frac{p(\mathbf{o}\mid\mathbf{x})\,p(\mathbf{x})}{\int p(\mathbf{o}\mid\mathbf{x}')p(\mathbf{x}')\,d\mathbf{x}'} $$Sequential update:
$$ b_{k}(\mathbf{x})=p(\mathbf{x}\mid\mathbf{o}_{1:k},\mathbf{a}_{1:k})\propto p(\mathbf{o}_{k}\mid\mathbf{x},\mathbf{a}_{k})\,b_{k-1}(\mathbf{x}) $$
Figure 2: System overview — graphics renderer for depth, physics simulator for contact, SMC inference backend.
Sequential Monte Carlo (SMC)
Belief represented as weighted particles (multimodal posterior):
$$ b_{k}(\mathbf{x})\approx\sum_{i=1}^{N}w_{k}^{(i)}\delta\!\left(\mathbf{x}-\mathbf{x}_{k}^{(i)}\right) $$State $\mathbf{x}=(x_x, x_y, \theta) \in SE(3)$. Two-phase observations:
$$ \mathbf{o}_{k}=\begin{cases}\mathbf{o}^{d}_{k}, & k\le\gamma,\\ (\mathbf{o}^{f}_{k},\mathbf{o}^{d}_{k}), & k>\gamma.\end{cases} $$Observation Likelihoods
Trajectory likelihood combining translational and rotational distances:
$$ \mathcal{L}_{\mathrm{traj}}=-\sum_{i=1}^{K_{\mathrm{traj}}}\left[\frac{d_{t}(\mathbf{t}_{k,i}^{\mathrm{obs}},\mathbf{t}_{k,i}^{\mathrm{sim}})^{2}}{2\sigma_{t}^{2}}+\frac{d_{R}(\mathbf{R}_{k,i}^{\mathrm{obs}},\mathbf{R}_{k,i}^{\mathrm{sim}})^{2}}{2\sigma_{R}^{2}}\right] $$where $d_t(\mathbf{t},\hat{\mathbf{t}})=\|\mathbf{t}-\hat{\mathbf{t}}\|_2$ and $d_R(\mathbf{R},\hat{\mathbf{R}})=\cos^{-1}\left(\frac{\mathrm{tr}(\mathbf{R}\hat{\mathbf{R}}^\top)-1}{2}\right)$.
Figure 3: Geometry-conditioned contact likelihood — F/T data contains rich local information about hole direction.
Vision Phase SMC + MH Refinement
Explore-exploit proposal:
$$ \pi_{k}(\mathbf{x})=\alpha\,\mathcal{U}(\mathcal{X})+(1-\alpha)\sum_{j}w_{k-1}^{(j)}\mathcal{K}\!\left(\mathbf{x}\mid\mathbf{x}_{k-1}^{(j)}\right) $$Log importance weight:
$$ \log\tilde{w}_{k}^{(i)}=\mathcal{L}_{\mathrm{depth}}\!\left(\mathbf{o}_{k}^{d}\mid\mathbf{x}_{k}^{(i)}\right)+\log p_{0}\!\left(\mathbf{x}_{k}^{(i)}\right)-\log\pi_{k}\!\left(\mathbf{x}_{k}^{(i)}\right) $$MH acceptance probability:
$$ A(\mathbf{x}\rightarrow\mathbf{x}')=\min\big(1,\,\exp[\mathcal{L}_{\mathrm{depth}}(\mathbf{o}_k^d\mid\mathbf{x}')-\mathcal{L}_{\mathrm{depth}}(\mathbf{o}_k^d\mid\mathbf{x})]\big) $$
flowchart TD
A["Initial Prior b_0"] --> B["Vision Phase
Depth SMC + MH"]
B --> C["Vision Posterior
Multimodal particle belief"]
C --> D["Contact Phase
Guarded probing actions"]
D --> E["Physics Simulation
Predict contact outcomes"]
E --> F["Contact Likelihood Scoring
L_contact + L_traj"]
F --> G["SMC Belief Update"]
G --> H["Converged Posterior
Pose estimate"]
G -->|"Not converged"| I["Info-Gain Active Sensing
Select next probe"]
I --> D
style B fill:#e1f5fe
style H fill:#e8f5e9
Results
Figure 4: Pose convergence — particle belief progressively concentrates on true pose.
Figure 5: Active sensing — Information Gain (IG) vs MAP probing strategy comparison.
| Method | Round | Square | Star | Average |
|---|---|---|---|---|
| Vision-only | 60% | 50% | 40% | 50% |
| BayesContact (vision+contact) | 80% | 75% | 65% | 73% |
| BayesContact + IG active sensing | 85% | 80% | 70% | 78% |
| Method | Insertion Success | Avg Probes | Pose Error (mm) |
|---|---|---|---|
| Vision-only | 40% | — | 3.2 |
| BayesContact | 70% | 4.5 | 1.1 |
Limitations
- Sim-to-real gap: Contact likelihood depends on physics simulator accuracy. Differences in friction, compliance, and other parameters between simulation and reality may affect likelihood scoring quality.
- Computational cost: Each particle requires independent rendering and physics simulation. Increasing particle count and probe steps raises computational overhead, potentially limiting real-time application.
Conclusion
BayesContact formulates pose estimation in peg-in-hole as simulation-based Bayesian inference, fusing depth and contact evidence through a particle belief with renderers and physics simulators as forward models. SMC handles multimodal posteriors, and information-gain-driven active sensing further disambiguates. Across simulated and real-robot experiments, it improves pose observability and insertion success by 30% over vision-only methods.
Key insight: "Visually plausible pose hypotheses may not be physically feasible" — contact information not only supplements vision but excludes hypotheses that are visually indistinguishable yet physically impossible through geometric constraints, which is the core insight of BayesContact.
SOURCE LINKS



