PAPER DEEP DIVE
See like a Robot: Robot-Centric Pointmaps for Vision-Language-Action Models
This paper addresses the frame mismatch in VLA models between camera-frame observation and robot-frame action by introducing robot-centric pointmaps—images whose pixels store 3D coordinates in the robot frame. Pointmaps provide robot-frame 3D geometry while preserving image structure, enabling cross-viewpoint generalization across diverse camera setups.
Paper Information
Title: See like a Robot: Robot-Centric Pointmaps for Vision-Language-Action Models
Authors: Byungkun Lee, Dongyoon Hwang, Dongjin Kim, Hojoon Lee, Minho Park, Jaegul Choo
Affiliation: KAIST AI, Holiday Robotics
Paper: arXiv:2607.11498
Project: davian-robotics.github.io/pointmap
One-Line Summary
This paper introduces robot-centric pointmaps—images whose pixels store 3D coordinates in the robot frame, re-centered on the end effector—fused into pretrained VLAs with minimal architectural change, significantly improving cross-viewpoint generalization.
Background and Motivation
Vision-language-action (VLA) models learn robotic manipulation policies from large-scale datasets, predicting actions from visual observations and language instructions. These actions are commonly defined in the robot's own 3D coordinate frame (e.g., task-space end-effector commands in the robot base frame), so reliable manipulation requires reasoning about where the target object lies in metric 3D coordinates relative to the robot.
However, most VLAs receive observations in the camera frame—whether as RGB images or depth maps. This creates a frame mismatch: the policy observes the scene in the camera frame but predicts actions in the robot frame. Under a fixed camera viewpoint, all observations share the same perspective relative to the robot, so a single observation-to-action mapping remains consistent. But as large-scale datasets aggregate demonstrations across diverse camera setups, the policy must generalize this mapping across viewpoints rather than memorize a single one.
Existing methods fall into three categories. The first supplies viewpoint information around camera-view images (Plücker rays or camera-frame action reparameterization), but the visual input stays a camera-view image—the scene is never expressed in the robot frame. The second synthesizes views to expand training coverage or canonicalize test views, but relies on a generative model whose synthesis degrades for viewpoints far from training data. The third builds shared 3D scene representations, but these no longer retain the image structure that pretrained VLAs expect.
The paper's starting point: can we express the observation directly as robot-frame 3D geometry while preserving the dense $H \times W$ image grid that pretrained VLAs expect? The answer is robot-centric pointmaps—images whose pixels store 3D coordinates in the robot frame.
Figure 1: Robot-centric pointmaps provide 3D geometry aligned with robot-frame actions. Training data is collected from diverse camera viewpoints; pointmaps preserve the dense H×W grid expected by pretrained 2D VLAs while providing robot-centric 3D geometry.
Preliminaries
VLA models receive RGB images (possibly with depth) and language instructions, outputting robot actions (e.g., end-effector displacements). This paper uses $\pi_{0.5}$ and SmolVLA as pretrained VLA backbones, both built on PaliGemma visual encoders. The key constraint: VLA visual encoders expect $H \times W$ grid input, so any 3D augmentation must preserve this structure to reuse pretrained weights.
Pointmaps are a standard representation in modern 3D vision—an image with the same resolution as RGB, but each pixel stores a 3D coordinate instead of color. Pointmaps retain image spatial structure, so they can be processed by standard 2D convolution/Transformer encoders. The contribution here is introducing pointmaps into pretrained VLAs and systematically studying which coordinate frame to express them in.
Method
Overall Architecture
The method has two steps: (1) lift each RGB-D observation into a robot-centric pointmap, (2) fuse pointmap tokens with RGB tokens before feeding into the VLA. No point cloud-specific encoder, voxelization module, or additional token sequence is needed.
Robot-Centric Pointmap Construction
Consider an $H \times W$ RGB image from camera $c$, with intrinsics $K_c$, rotation $R_c$ and translation $t_c$ from camera to robot base frame, and per-pixel depth $D_c$. First, lift each pixel into a 3D point in the camera frame:
$$P_c^{\text{cam}}(u,v) = D_c(u,v) \cdot K_c^{-1} [u, v, 1]^{\top}$$
Then apply the camera-to-robot transform to express the point in the robot base frame:
$$P_c^{\text{R}}(u,v) = R_c \cdot P_c^{\text{cam}}(u,v) + t_c$$
The robot-frame pointmap $P_c^{\text{R}}$ has the same $H \times W$ grid as the RGB image, with one 3D coordinate per pixel. Key property: the same physical scene point receives the same robot-frame coordinate regardless of the camera viewpoint, although the pixel $(u,v)$ where it appears may change.
Figure 3: RGB-D observations are lifted into camera-frame pointmaps and transformed into robot-frame pointmaps. The same scene point receives the same robot-frame coordinate across viewpoints.
End-Effector Centering
The pointmap is further re-centered on the current end effector:
$$P_c^{\text{EE}}(u,v) = P_c^{\text{R}}(u,v) - t_{\text{EE}}$$
where $t_{\text{EE}}$ is the end-effector position in the robot base frame. This centering expresses scene geometry relative to the current end-effector position. Since actions are defined as motions of that end effector, the observation and action space now share a common origin. Ablation shows end-effector centering is more robust under randomized evaluation viewpoints (0.3-point drop vs 2.0-point drop for robot-base centering).
Pointmap-RGB Fusion
Because the pointmap preserves the same $H \times W$ grid as the RGB image, a separate pointmap encoder $g_\phi$ (initialized from the RGB encoder $f_\theta$) maps $P_c^{\text{EE}}$ into tokens with the same shape as RGB tokens, added element-wise:
$$z_c = f_\theta(I_c) + g_\phi(P_c^{\text{EE}}) \in \mathbb{R}^{N_{\text{tok}} \times d}$$
The fused tokens $z_c$ replace the original RGB tokens as the VLA's visual input. Key advantages: no point cloud-specific encoder, no voxelization module, no additional token sequence—the pretrained VLA's visual pathway directly absorbs robot-frame 3D geometry.
flowchart TB
A[RGB-D Input] --> B[Camera-frame Pointmap]
B --> C[Robot-frame Transform]
C --> D[EE-centered Pointmap]
A --> E[RGB Encoder f_theta]
D --> F[Pointmap Encoder g_phi]
E --> G[RGB Tokens]
F --> H[Pointmap Tokens]
G --> I[Element-wise Add]
H --> I
I --> J[Fused Visual Tokens]
J --> K[VLA Backbone]
K --> L[Robot Actions]
Design Choice Analysis
Robot-Frame Pointmaps Outperform Camera Information Conditioning
Four inputs compared: RGB (no depth/calibration), RGB+Plücker rays, RGB+camera-frame pointmap, RGB+robot-frame pointmap. Pre-computing the camera-to-robot transform into the observation (robot-frame pointmap) is more effective than letting the policy infer it from raw camera information. Plücker rays provide 6D ray encoding but remain in the camera frame.
Image-Form Pointmaps Outperform Point Clouds
Comparing image-form pointmaps vs point clouds: pointmaps preserve the $H \times W$ grid for reusing pretrained visual weights, while point clouds require dedicated encoders and voxelization. Pointmaps outperform on RoboCasa, validating the importance of retaining image structure.
End-Effector Centering for Viewpoint Robustness
End-effector centering collapses interaction targets scattered across the robot-base frame to cluster near the end-effector origin. Two different tasks with different base-frame coordinates both lie near $(0,0,0)$ at grasp time in the EE-centered frame, giving the policy a consistent local geometry.
Figure 5: End-effector centering concentrates interaction targets near a common origin. (a) Target coordinates are broadly distributed in the robot-base frame but cluster near the origin in the EE-centered frame. (b) Two tasks show targets with different base coordinates lie near the EE origin at grasp.
Robustness to Increasing Viewpoint Variation
Under No variation, Low, and High settings, RGB-only drops 9.6 points (34.5%→24.9%), while RGB+pointmap drops only 1.8 points (37.6%→35.8%). This supports the core hypothesis: robot-frame geometry keeps the policy robust to training-time viewpoint variation.
Figure 6/10: Effect of training-time camera viewpoint variation. RGB performance drops as variation increases, while RGB+Pointmap stays stable.
Experimental Results
Simulation Experiments (RoboCasa)
Evaluated on RoboCasa across 24 tasks in 5 categories, with 50 demonstrations and 50 evaluation episodes per task. Compared against three groups: camera-aware VLAs (KYC, OC-VLA), 3D-augmented VLAs (GeoVLA, PointVLA), and point cloud policy (FP3).
| Method | Avg. | Doors | Drawers | Coffee | Pick-place | Turn |
|---|---|---|---|---|---|---|
| FP3 (point cloud) | 42.8 | 79.0 | 74.0 | 54.7 | 21.8 | 32.0 |
| OC-VLA | 56.3 | 80.0 | 80.0 | 42.0 | 50.5 | 48.9 |
| KYC | 59.1 | 86.5 | 86.0 | 51.3 | 48.2 | 51.4 |
| GeoVLA | 57.1 | 81.0 | 80.0 | 45.3 | 49.8 | 50.3 |
| PointVLA | 57.3 | 87.5 | 85.0 | 44.0 | 46.3 | 50.3 |
| $\pi_{0.5}$ (RGB) | 55.3 | 79.5 | 83.0 | 40.7 | 46.0 | 50.6 |
| $\pi_{0.5}$ + pointmap | 62.9 | 90.0 | 90.0 | 58.0 | 52.8 | 53.4 |
| SmolVLA (RGB) | 37.2 | 68.0 | 63.0 | 39.3 | 6.5 | 46.6 |
| SmolVLA + pointmap | 41.4 | 80.0 | 77.0 | 38.0 | 12.8 | 43.4 |
Adding pointmaps improves $\pi_{0.5}$ by +7.6 points (55.3→62.9) and SmolVLA by +4.2 points (37.2→41.4), with gains across all 5 categories. The strongest camera-aware baseline (KYC, 59.1) and 3D-augmented baseline (PointVLA, 57.3) both remain below pointmaps (62.9), confirming that expressing geometry in the robot frame and fusing on the RGB grid is more effective than conditioning on camera parameters or attaching a 3D module.
Real-World Experiments
Franka Research 3 with wrist D405 and external D435i cameras. External camera placed at 3 training positions; evaluation at seen and unseen placements.
| Eval Camera | Model | Avg. | Pick-place | Stack | Open drawer | Close drawer |
|---|---|---|---|---|---|---|
| Seen | DP3 | 63.3 | 60.0 | 40.0 | 60.0 | 93.3 |
| Seen | $\pi_{0.5}$ (RGB) | 73.3 | 80.0 | 53.3 | 73.3 | 86.7 |
| Seen | $\pi_{0.5}$+pointmap | 78.3 | 86.7 | 60.0 | 73.3 | 93.3 |
| Unseen | DP3 | 48.3 | 33.3 | 33.3 | 40.0 | 86.7 |
| Unseen | $\pi_{0.5}$ (RGB) | 55.0 | 40.0 | 26.7 | 66.7 | 86.7 |
| Unseen | $\pi_{0.5}$+pointmap | 66.7 | 53.3 | 46.7 | 73.3 | 93.3 |
At the seen placement, pointmaps improve $\pi_{0.5}$ by +5.0 points (73.3→78.3). At the unseen placement, the advantage widens: RGB-only drops from 73.3 to 55.0 (−18.3), while pointmap drops from 78.3 to 66.7 (−11.6). The margin over RGB grows from +5.0 to +11.7, directly validating the core value of pointmaps for cross-viewpoint generalization.
Figure 7: Real robot setup. (a) Franka Research 3 with wrist camera and external camera at 3 training positions (red) and 1 held-out position (blue). (b) Tasks: pick-and-place, stack blocks, open/close drawer.
Limitations
Author-stated limitation 1: The study does not ablate how the pointmap is injected relative to the action expert, nor how it interacts with the pretraining recipe, so the best combination remains unsettled. The comparison against point clouds uses a single sampling budget—a larger budget could narrow the gap.
Author-stated limitation 2: Pointmaps require calibrated camera intrinsics and extrinsics at both training and test time, restricting them to setups where calibration is available. The camera-variation results focus on placement and extrinsics changes, not yet covering changes in camera number or field of view.
Independent assessment: The method depends on RGB-D input (requiring depth sensors), which may be unavailable in some deployment scenarios—particularly RGB-only systems. The paper does not discuss how to obtain depth from RGB-only setups (e.g., monocular depth estimation). Additionally, the pointmap encoder is initialized from the RGB encoder but trained independently—the parameter count and inference cost increase from the dual encoder pathway are not discussed.
Conclusion and Outlook
This paper studies how VLAs should use 3D observations when demonstrations are collected under various camera viewpoints. The core finding: rather than leaving the camera-to-robot relationship for the policy to infer from RGB, depth, and calibration inputs, express each observed scene point directly in the robot frame as a robot-centric pointmap. The pointmap preserves the dense $H \times W$ image grid, allowing robot-frame 3D coordinates to be fused with RGB tokens through the existing visual pathway of a pretrained VLA.
The experiments point to a simple and effective input: a robot-frame, image-form pointmap centered on the end effector. These results suggest a general principle for 3D-aware VLAs: when calibration is available, express observations in the coordinate system where the robot acts before passing them to the policy.
"Don't make the policy guess the camera-to-robot transform—give it geometry in the frame where the robot acts, and let the pretrained visual pathway do the rest."
Source: arXiv:2607.11498 · Project Page



