PAPER DEEP DIVE
SERF: Spatiotemporal Environment and Robot Feature Map for Long-Horizon Mobile Manipulation
SERF represents the environment and the articulated robot body as neural points in one shared latent space and updates them online. Environment points follow object-level rigid registration, while robot surface points are posed by forward kinematics. A shared residual decoder maps local point features to DINOv3 patch embeddings, and a two-stage Point Transformer produces eight map tokens from three robot-centered radii, both end effectors, robot-only, environment-only, and global branches. These tokens condition a VLA policy together with images, proprioception, and language. Across three long-horizon BEHAVIOR-1K tasks, SERF raises average progress from 44.0% for fine-tuned PI0.5 to 58.7%, and the paper also reports gains when labels and base poses are estimated onboard, when object-drop recovery is required, and in unvisited regions.
Core result in one paragraph
SERF couples a persistent environment map and a pose-dependent robot-body map in one latent feature space. Object-level rigid registration moves environment points, forward kinematics moves robot surface points, and a shared decoder reconstructs DINOv3 patch features at arbitrary 3D queries. Eight map tokens describe local, end-effector, robot-only, environment-only, and global context. A map-conditioned vision-language-action policy then uses those tokens alongside images, proprioception, and language. Across three BEHAVIOR-1K mobile manipulation tasks, SERF raises average progress from 44.0 percent to 58.7 percent and improves recovery and cross-configuration behavior.
Why an image history is not a spatial state
A mobile manipulation episode is a chain of physical commitments. The robot moves a chair to reach a cabinet, carries a container into another room, and later has to remember which routes remain open. The scene at the end of the episode may look nothing like the first frame. A policy needs a compact state that survives these changes without replaying every previous image at every action step.
Image histories contain evidence of those changes, but they do not expose the changed state in a form that action generation can query directly. A stack of RGB frames still represents a cabinet as pixels from earlier viewpoints. The policy must infer depth, object identity, camera motion, and the robot's own displacement from the sequence. This inference becomes harder when an object leaves the field of view precisely because the robot moved it.
Geometric maps solve the persistence problem but usually expose the wrong interface. An occupancy grid answers whether a voxel is free, not what visual concept surrounds it. A colored point cloud preserves appearance but lacks learned semantic structure. A semantic map stores object labels, yet labels alone do not tell a transformer how a local patch of tabletop relates to the left gripper or how a distant cabinet relates to the current subgoal.
SERF treats the map as a policy input rather than a planning side product. A query at any 3D location returns a continuous feature that has been learned to match a visual foundation model. Environment points and robot points share the same feature dimension and decoder. The robot's body is therefore represented spatially, not as an unrelated vector appended to the policy.
The policy problem and the map contract
At time $\tau$, the policy receives a map $m_\tau$, an observation $o_\tau$, proprioceptive state $s_\tau$, and task embedding $e_\ell$. It produces an action chunk $a_\tau$. The behavioral cloning objective is:
$$\mathcal{L}_{bc}(\phi)=-\mathbb{E}_{\mathcal{T}}[\log\pi_{\phi}(a_{\tau}\mid m_{\tau},o_{\tau},s_{\tau},e_{\ell})].$$The expression is deliberately ordinary. The important question is what $m_\tau$ contains and how it changes. If the map is frozen at the first frame, the policy has a spatial memory but no way to account for its own interventions. If the map is rebuilt from the latest frame, it loses object persistence. SERF updates a persistent representation online and exposes it at multiple scales.
The map is a union of two point sets:
$$\mathcal{P}_{\tau}=\mathcal{P}^{e}_{\tau}\cup\mathcal{P}^{r}_{\tau},$$where the first set represents the environment and the second represents the robot surface. Each point $p_i$ carries a 64-dimensional latent feature $f_i$. During execution, labels and features remain fixed while positions change with rigid object motion or robot kinematics. This separation is central: the expensive semantic representation is not recomputed at every frame, but the geometry remains live.
A query does not retrieve a single nearest point. It aggregates features from a local neighborhood and decodes the result. This gives SERF a continuous field over space, unlike a discrete object list. It also allows the same query operation to read a table edge, a cabinet handle, and a robot forearm, provided their points have been placed in the shared representation.
Learning a feature field instead of a colored point cloud
The environment starts from RGB-D observations. The paper uses 480 by 480 RGB images; DINOv3 produces a 30 by 30 patch grid. Depth back-projection lifts these patches into 3D points, each with a learned 64-dimensional latent and an instance label. The paper evaluates both privileged simulation labels and estimated labels from SAM3 in a deployment-oriented experiment.
The map is trained so that a query at a spatial location predicts the visual embedding associated with that location. For a query $x$ and target visual feature $y$ drawn from a time window, the map loss is:
$$\mathcal{L}_{map}(\Theta)=\mathbb{E}_{\tau,(x,y)\sim\mathcal{D}_{\tau}}[\mathcal{L}(m_{\tau}(x;\Theta),y)].$$This objective explains why the map can be more than geometry. The 64-dimensional points are not required to encode only surface normals or RGB values. They are optimized as an intermediate representation from which a shared decoder can recover a 1280-dimensional DINOv3 patch feature.
At query time, SERF selects $K$ neighboring points and uses a distance-sensitive softmax:
$$F(x;\mathcal{P}_{\tau})=\sum_{i\in\mathcal{N}(x,\mathcal{P}_{\tau})}w_i f_i,\qquad w_i=\mathrm{softmax}(-\|x-p_{i,\tau}\|/\sigma).$$The implementation uses $K=6$ and temperature $\sigma=0.05$. The continuous weighting matters when a neighborhood is sparse near a surface boundary or when tracking introduces small coordinate errors. A hard nearest-neighbor assignment would make the feature field jump discontinuously.
The aggregated latent is decoded by a residual multilayer perceptron:
$$m_{\tau}(x;\Theta)=D_{\theta}(F(x;\mathcal{P}_{\tau})).$$Environment points and robot points use the same $D_\theta$. That design choice turns the robot body into a first-class occupant of the semantic map. A query near the left gripper can return robot-surface context or nearby environment context using the same decoder rather than switching between incompatible state encodings.
Figure 1 from the paper: RGB observations and instance labels are lifted into a neural-point representation. The map stores more than visible color because its points are trained to decode into visual features.
Environment updates are object-level rigid transformations
SERF does not resample every object from scratch after each frame. It tracks an instance between observations, estimates a rigid transform, and applies that transform to all neural points belonging to the instance. For point correspondences $q_k^{\tau-1}$ and $q_k^{\tau}$, the registration problem is:
$$(\hat R,\hat t)=\arg\min_{R\in SO(3),t}\sum_k\left\|q^{\tau}_k-(Rq^{\tau-1}_k+t)\right\|^2.$$The implementation uses FGR for coarse registration and an ICP refinement against a denser observation cloud when one is available. The resulting transform updates the points as:
$$p^{e}_{i,\tau}=\hat R\,p^{e}_{i,\tau-1}+\hat t.$$Because the instance label and latent feature stay attached to the point group, identity survives the move. This is a practical advantage over rebuilding semantic labels after every manipulation. A chair that the robot pushed aside is still the same instance in the map, even when its new position is far from the original training distribution.
The mapping code exposes the same logic. register_instance_for_frame collects correspondences from visible instance keypoints, solves the rigid transform, optionally runs ICP, checks whether motion is below a stationary threshold, and then commits the position update atomically. The stationary threshold is 0.015 meters. A larger reset threshold of 0.20 meters protects against implausible jumps. Accepted centroid error is 0.02 meters.
# SERF-mapping/tracking/utils/tracking_ops.py:768-793
T_reg[:3, :3] = R
T_reg[:3, 3] = t_vec
center_tgt_pred = R @ center_src + t_vec
if np.linalg.norm(center_tgt_pred - center_src) < cfg.stationary_threshold:
return "stationary"
new_pts = current_pts @ R_f.T + t_f
neural_points.remove_points_from_hash(Y_idx)
neural_points.map_points[Y_idx] = new_pts
new_grid_coords = torch.floor(new_pts / neural_points.voxel_size).to(torch.int64)
neural_points._hash_table_insert(new_grid_coords, Y_idx)The update assumes the tracked object is rigid. That covers tables, chairs, shoes, boxes, and many household containers, but it excludes articulated cabinet doors and deformable objects. The paper states this limitation explicitly. A hinge would need a kinematic model with a joint axis, while cloth or a bag would require a non-rigid deformation field or a different surface representation.
Forward kinematics makes the robot a moving part of the scene
The robot surface is represented by a fixed-topology set of points. Each point belongs to a link. For link $l_j$, the world-space position at state $s_\tau$ is:
$$p^{r}_{j,\tau}=R_{l_j}(s_{\tau})u_j+t_{l_j}(s_{\tau}).$$Here $u_j$ is the point in the link frame, and the rotation and translation come from forward kinematics. The point identity never changes. When the arm moves, only the pose-dependent coordinate changes. This is the opposite of the environment update: an object point follows an estimated transformation, while a robot point follows a known kinematic chain.
The robot base pose comes from proprioception. The arm and end-effector poses are computed by the forward-kinematics implementation. The policy tokenizer uses a robot center with an upward offset of 0.8 meters, together with the left and right end-effector positions. Those reference frames become query centers for local map tokens.
One might ask why the policy cannot simply receive joint angles. Joint state is useful for control, but it does not directly encode which part of the room is near the wrist or whether an obstacle lies between the elbow and a cabinet. Once robot points are transformed into world space, the tokenizer can ask spatial questions that combine body configuration and scene geometry.
The tokenizer compresses a large point set into eight task-relevant questions
A map with 25,000 points per policy query is too large for the language-model prefix. SERF first filters the points and sends them through a shared two-stage Point Transformer. The backbone uses 128 and 256 channels, two blocks per stage, stride four, and 16 nearest neighbors. Eight parallel branch heads have width 256, stride one, and attention pooling that produces a 2048-dimensional output.
The eight branches are not arbitrary slices of the point cloud. Three orbit the robot center at radii of 1, 2, and 4 meters. Two focus on the left and right end effectors with a 0.5-meter radius. One contains only robot points. One contains only environment points. The final branch is global. Their roles overlap, but each supplies a different spatial aperture.
The first radius is suitable for near-body collision and manipulation context. Two meters captures the immediate room layout around the robot. Four meters supports longer movement decisions. The end-effector branches isolate the regions where contact and grasp geometry matter. The robot-only branch can represent self-occlusion and the spatial footprint of the body. The environment-only branch provides a cleaner scene summary. The global branch preserves long-range context that would be lost if every query were local.
The implementation removes absolute coordinates and keeps relative positional encoding. This is not a minor optimization. A scene with a different room size or map normalization should not force the attention layers to relearn what “near” means in meters. Relative geometry gives the branch heads a more stable query language.
# SERF-VLA/src/serf_b1k/models/point_transformer_local.py:467-491
for i, radius_world in enumerate(self.robot_center_radii):
norm_radius = radius_world / pc_norm_scale
mask = robot_sq_dist <= (norm_radius ** 2)
tokens.append(self.robot_branches[i](stage2_xyz, stage2_feat, mask))
if self.use_robot_points:
tokens.append(self.global_branch(stage2_xyz, stage2_feat))
for ee_pos, branch in [
(left_ee_norm, self.left_ee_branch),
(right_ee_norm, self.right_ee_branch),
]:
sq_dist = jnp.sum((stage2_xyz - ee_pos[:, None, :]) ** 2, axis=-1)
mask = sq_dist <= (self.ee_radius / pc_norm_scale) ** 2
tokens.append(branch(stage2_xyz, stage2_feat, mask))The branch outputs are concatenated and inserted into the VLA prefix as non-autoregressive state tokens. They do not replace the current image. Images still provide immediate appearance and texture; map tokens provide persistent spatial relationships that the image alone cannot retain after an object leaves view.
Figure 2 from the paper: the map tokenizer and the map-conditioned VLA. Eight branches convert the neural-point field into compact state tokens that are appended to the visual-language prefix.
flowchart TB
OBS["RGB-D frames"] --> PATCH["DINOv3 patch features"]
PATCH --> BP["Depth back-projection"]
BP --> ENV["Environment neural points"]
TRACK["Instance correspondences"] --> FGR["FGR plus ICP"]
FGR --> RIGID["Rigid SE3 transform"]
RIGID --> ENV
PROP["Proprioceptive state"] --> FK["Forward kinematics"]
FK --> ROBOT["Robot surface points"]
ENV --> SHARED["Shared latent point field"]
ROBOT --> SHARED
SHARED --> DECODE["Shared residual decoder"]
DECODE --> VIS["1280-D visual features"]
VIS --> PT["Two-stage Point Transformer"]
PT --> LOCAL["Three base radii"]
PT --> EE["Left and right end effectors"]
PT --> SPLIT["Robot only and environment only"]
PT --> GLOBAL["Global branch"]
LOCAL --> TOKENS["Eight map tokens"]
EE --> TOKENS
SPLIT --> TOKENS
GLOBAL --> TOKENS
TOKENS --> PREFIX["Frozen VLM prefix"]
IMAGE["Current image"] --> PREFIX
LANGUAGE["Task language"] --> PREFIX
PREFIX --> LORA["LoRA action expert"]
LORA --> CHUNK["30-step action chunk"]
Flow matching connects the map to continuous robot control
The action model is built on PI0.5 and the BEHAVIOR-1K OpenPI stack. The visual-language model and vision encoder are frozen. LoRA adapters are trained in the action expert. This keeps the broad visual and linguistic knowledge of the base model while directing training capacity toward the new map interface and continuous control.
The action expert receives a condition that can be summarized as:
$$h_{\tau}=\mathrm{Concat}(E(o_{\tau}),e_m,e_s,e_{\ell}).$$Here $E(o_\tau)$ is the observation encoding, $e_m$ are map tokens, $e_s$ is proprioceptive state, and $e_\ell$ is the task embedding. The action head is trained with flow matching. A noisy action is formed by interpolating a demonstration action $A_\tau$ and Gaussian noise $\epsilon$:
$$A_{\tau}^{\alpha}=\alpha A_{\tau}+(1-\alpha)\epsilon,\qquad u_{\tau}=A_{\tau}-\epsilon.$$The network predicts the velocity field that moves the sample toward the demonstration action:
$$\mathcal{L}_{action}=\mathbb{E}\left\|v_{\psi}(A_{\tau}^{\alpha},h_{\tau},\alpha)-u_{\tau}\right\|_2^2.$$Each action chunk has horizon 30 and dimension 32. Training samples 15 flow examples per update and draws $\alpha$ from a Beta(1.5, 1) distribution clipped to the interval between 0.001 and 0.999. At inference, SERF performs 20 Euler integration steps, executes the first 26 actions, and interpolates them into 20 control commands.
The released configuration trains each task for 20,000 steps with batch size 16 and a peak learning rate of $2.5\times10^{-6}$. The vision backbone is frozen. Knowledge insulation uses a stop-gradient between the VLM prefix and the action expert, so the action training can adapt the interface without modifying the base visual representation.
What BEHAVIOR-1K actually stresses
The evaluation uses three household tasks. Each task has 200 demonstrations and is tested across 20 configurations. The workspaces are 4.4 by 3.0 meters, 5.5 by 9.0 meters, and 7.6 by 8.8 meters. Maximum episode lengths range from 15,384 to 52,120 steps. The larger tasks require movement between rooms, object transport, and repeated re-observation.
| Task | Workspace | Maximum steps | Demonstrations | Evaluation configurations |
|---|---|---|---|---|
| Task 21 | 4.4 by 3.0 m | 38,372 | 200 | 20 |
| Task 22 | 5.5 by 9.0 m | 15,384 | 200 | 20 |
| Task 26 | 7.6 by 8.8 m | 52,120 | 200 | 20 |
Task progress is a more informative measure than binary success for these episodes because a policy can complete several subgoals before failing. It also exposes whether a method reaches the right room quickly or wastes time revisiting locations. The reported standard deviations show that scene configuration still changes performance substantially.
The main comparison: consistent gains, with a large average shift
The most direct comparison holds the BEHAVIOR-1K training and evaluation setup constant and changes the map-conditioned state interface. The table reproduces the reported mean progress and standard deviation.
| Method | Task 21 | Task 22 | Task 26 | Three-task average |
|---|---|---|---|---|
| PI0.5 pretrained | 42.9 +/- 19.7 | 43.0 +/- 23.5 | 44.1 +/- 22.4 | 43.3 |
| PI0.5 fine-tuned | 40.7 +/- 18.8 | 43.0 +/- 19.3 | 48.4 +/- 21.1 | 44.0 |
| SBP | 57.9 +/- 17.8 | 52.5 +/- 24.1 | 51.6 +/- 11.7 | 54.0 |
| SERF environment-only | 57.9 +/- 15.3 | 59.0 +/- 20.5 | 49.4 +/- 13.4 | 55.4 |
| Full SERF | 63.5 +/- 16.7 | 60.1 +/- 19.1 | 52.5 +/- 13.8 | 58.7 |
Full SERF improves over fine-tuned PI0.5 by 22.8, 17.1, and 4.1 percentage points on the three tasks. The average moves from 44.0 to 58.7, a 14.7-point increase. The gain is largest where long detours and object relocation matter most. Task 26, the largest environment, gets the smallest benefit, which is a useful warning that a spatial map is not a universal answer to every long-horizon failure.
The environment-only variant already reaches 55.4 percent average progress. Adding robot points adds a substantial gain on Task 21 and a smaller one on Task 22. Task 26 is slightly worse with robot points than with environment points alone. The authors present this difference rather than hiding it. A plausible interpretation is that a robot-centric token can distract from route-level context in the largest scene, but the experiment does not isolate that mechanism.
SBP is a relevant baseline because it also uses spatial information. It reaches 54.0 percent on average. SERF's shared environment-robot field and its multi-aperture tokenization therefore add value beyond the generic statement that spatial input helps. The comparison does not eliminate all input-privilege differences, but it rules out a pure image-only story.
Figure 3 from the paper: qualitative trajectories. SERF reaches later subgoals earlier and follows a more direct route. The result ties progress to the cost of remembering and finding locations, not only to grasp quality.
Replacing simulation labels and poses with onboard estimates
The strongest deployment caveat is that the full model uses privileged simulation information. SERF receives instance labels and base poses that a real robot must estimate. The paper therefore replaces them with SAM3 instance segmentation and ORB-SLAM3 trajectory estimates while keeping the rest of the system fixed.
| Task 22 condition | Progress | Difference from PI0.5 fine-tuned |
|---|---|---|
| PI0.5 fine-tuned | 43.0 +/- 19.3 | Baseline |
| SERF with SAM3 labels and ORB-SLAM3 poses | 54.8 +/- 22.9 | +11.8 |
| SERF with ground-truth labels and poses | 60.1 +/- 19.1 | +17.1 |
ORB-SLAM3 achieves 2.42 centimeters translation error and 0.34 degrees rotational error on the reported trajectory. The estimated pipeline loses 5.3 progress points relative to the privileged version, but still beats the image-only policy by 11.8 points. This is more informative than a perfect-simulation number because it includes segmentation mistakes and pose drift.
The replacement also clarifies where the system can fail. An incorrect instance boundary can move the wrong points together. A pose error can insert an environment point at the wrong world coordinate. Once that error is committed, later local queries may repeatedly aggregate contaminated features. The atomic map update ensures that a crash does not leave a half-written instance, but atomicity is not the same as correctness.
Figure 4 from the paper: SAM3 instance labels and the ORB-SLAM3 trajectory comparison. The estimated-input experiment is important because it measures the cost of removing two privileged signals.
Ablation results show which spatial questions the policy relies on
The authors remove one token group at a time on Task 22. Removing end-effector tokens or robot-only tokens causes the largest drop, both falling from 60.1 to 54.0 percent. Removing the global branch gives 55.1. Removing the three robot-base radii gives 56.5. Removing the environment-only branch gives 58.5.
| Removed group | Remaining branches | Task 22 progress | Drop |
|---|---|---|---|
| Robot-base radii | 7 | 56.5 +/- 21.7 | 3.6 |
| End-effector queries | 7 | 54.0 +/- 21.5 | 6.1 |
| Robot-only | 7 | 54.0 +/- 22.2 | 6.1 |
| Environment-only | 7 | 58.5 +/- 20.9 | 1.6 |
| Global | 7 | 55.1 +/- 20.6 | 5.0 |
| None, full model | 8 | 60.1 +/- 19.1 | 0.0 |
The small drop for the environment-only branch should not be read as evidence that environment information is unimportant. Environment points still enter the base-radius, end-effector, and global queries. The branch removal measures the marginal value of one explicit summary, not the value of the whole map.
The image-history comparison addresses a different question. PI0.5 with image history reaches 50.0, 49.5, and 48.1 percent on tasks 21, 22, and 26. Full SERF reaches 63.5, 60.1, and 52.5. The image-history baseline is stronger than a current-frame policy, but it remains below the map-conditioned model on all three tasks. More temporal pixels help, yet they do not substitute for a queryable spatial state.
Recovery and unvisited space reveal the practical role of persistence
Object drops are a clean test of what a map remembers after the visible state changes. Across 20 object-drop episodes, SERF reports 95 percent recovery success and an average recovery time of 20.5 seconds. The policy can use the persisted spatial relation to search near where the object was last observed instead of treating the empty visual scene as the whole state.
In an unvisited region, progress rises from 28.0 percent to 51.0 percent compared with a no-map baseline. The neural representation permits interpolation from neighboring points, so the policy can form features in areas that were not directly covered by the current observation. This is not full scene completion and does not remove the need for exploration; it means the map can expose a useful prior where the point field is sparse.
These results fit the broader argument. SERF is not only a better short-range grasping model. It changes what the policy can know after moving away, after pushing a chair, or after dropping a container. That is where a persistent environment-robot state can matter more than another frame of RGB context.
Figure 5 from the paper: failure recovery after an object drop. The recovery chart reports success rate and average time across repeated trials.
Source-code correspondence
The mapping repository implements the shared neural representation and the rigid tracking pipeline. The entry point mapping/train_neural_points_env_and_robot.py constructs the environment map, the robot surface sampler, and a common decoder. The optimizer receives decoder, environment-point, and robot-point parameters together. The interpolation behavior is visible in neural_point_map.py and robot_neural_points.py: both use distance-weighted softmax aggregation.
The policy repository implements the read path. pi_serf_behavior.py computes forward-kinematic reference positions from the full proprioceptive state and appends map tokens to the VLM prefix. point_transformer_local.py defines the shared two-stage backbone and the eight branch queries. The training configuration contains the action horizon, point count, query radii, batch size, flow-sample count, and learning rate reported in the paper.
The code structure supports the main interpretation of the paper. SERF is not one large end-to-end model that rebuilds every representation from scratch. It is a split system: a mapping process maintains persistent features and geometry, while a policy process consumes a bounded set of queries. This modularity makes the method easier to adapt to another robot body, but it also creates an interface that must be maintained carefully.
Limitations that matter for deployment
The authors identify several limits. The method depends on prior learned map features and, in the main experiment, privileged simulation labels and base poses. It is evaluated in simulation rather than on a physical robot. Its environment updates assume rigid object motion, so articulated and deformable objects are outside the tested model. It does not use multiple temporal windows or predict future map states. Finally, the token subset is manually defined and is not conditioned on the task.
There are additional engineering risks visible from the system design. A 25,000-point transformer query, a DINOv3 feature pipeline, CoTracker-based tracking, SAM3 segmentation, and ORB-SLAM3 can be expensive to run online. The paper reports task progress rather than a full latency and power budget. A real-time implementation would need to decide which components run every frame and which update asynchronously.
Tracking error is the second major risk. A rigid registration failure can assign the wrong spatial displacement to an entire instance. The map may then look internally consistent while being wrong. The code includes stationary, reset, and centroid-error checks, but these are local safeguards. A robot needs a higher-level uncertainty model that can decide when to reobserve an object or distrust a region.
The evaluation scope is the third issue. Three BEHAVIOR-1K tasks demonstrate hard long-horizon behavior, but they do not cover every household layout, robot embodiment, or lighting condition. The image-only comparison also differs in input privileges, because SERF receives a learned spatial representation that the image baseline does not. The authors are careful to report the input replacement experiment, and that transparency makes the result more useful than a headline-only comparison.
Figure 6 from the paper: robot surface reconstruction samples under different configurations. The fixed-topology robot points are posed by forward kinematics, which gives the policy a spatial body representation but does not model soft-body deformation.
What this changes for embodied systems
The engineering lesson is that the interface between perception and action deserves as much attention as the perception backbone. A map that is accurate but awkward for a VLA to read will not improve control. SERF turns the map into a small set of learned queries with known spatial semantics, allowing the policy to ask about a local manipulation region and the global room at the same time.
A second lesson is that robot state should be represented in the same coordinate system as the environment. This is obvious in classical planning, but it is often lost when learned policies concatenate a joint vector with image embeddings. SERF makes the robot surface an occupant of the map, so the body can occlude, approach, and move through the scene in the same representation used for objects.
A third lesson is that memory needs an explicit update rule. The method does not simply concatenate frames and hope the transformer discovers change. It estimates rigid motion, applies a known transform, maintains identity, and keeps learned features stable. The update rule is narrow, but its narrowness is what makes the state auditable. The next step is to add uncertainty and richer object dynamics without losing that clarity.
Conclusion
SERF is a shared latent map for long-horizon mobile manipulation. Environment points follow object-level rigid registration, robot surface points follow forward kinematics, and both decode through a shared residual network into visual features. A filtered two-stage point transformer converts the field into eight local and global tokens for a map-conditioned VLA. On BEHAVIOR-1K, the full model improves average task progress from 44.0 to 58.7 percent, remains 11.8 points above PI0.5 when labels and poses are estimated onboard, and recovers from 95 percent of tested object drops.
The result is best understood as a new state interface, not a claim that long-horizon manipulation is solved. It shows that persistent spatial memory can be compressed into tokens that a VLA actually uses. It also leaves a clean research agenda: articulated and deformable objects, uncertainty-aware updates, task-conditioned query selection, and real-time operation on a physical robot.
