NEO: NeRF It Once, Edit It Many Times for Continuous Object Manipulation

1. Overview: NEO—NeRF It Once, Edit It Many Times

NEO, proposed by Mikolaj Zielinski, David Hall, Dominik Belter, and Peyman Moghadam in July 2026, is a unified framework providing language-guided NeRF editing for robotic manipulation. The core idea is "NeRF it once, edit many times"—train a NeRF+language field from a single scene scan, then repeatedly edit it without re-scanning to predict future scene states after robot manipulation. NEO introduces a three-stage editing pipeline: (i) language-guided object removal combining neural field resampling with multiview-consistent progressive inpainting; (ii) direct NeRF weight editing via knowledge distillation, composing original and edited NeRFs through a teacher-student model to coherently model future scene states before action execution; (iii) the first benchmark (NEO-Dataset) for quantitatively evaluating NeRF scene editing suitable for robot manipulation. NEO outperforms SOTA baselines in scene editing, producing visually coherent and geometrically consistent edits that reduce artifacts common in prior methods, and maintains persistent NeRF+language field representations for multi-stage assembly without additional re-scanning.

2. Core Problem: Robustness and Consistency of NeRF Editing

NeRFs as implicit neural scene representations are increasingly used in robotics, but existing NeRF editing methods, while producing compelling visual results, often fall short of the robustness and consistency required for robotic manipulation. The key challenge: robots need methods that reliably and permanently update scenes without re-scanning. DFF removes objects by suppressing density values for ray samples within the object region, but sparse sampling produces floating artifacts behind the object. NeRFiller reconstructs locally plausible regions but introduces more background artifacts. Seal-3D leaves many artifacts on the table surface. More critically, existing methods lack multiview-consistent progressive inpainting and direct NeRF weight editing capability, unable to coherently model future scene states before manipulation. These limitations highlight the need for scene representations supporting repeated object manipulation.

3. Method: NEO Framework

3.1 Neural Field Resampling

The network $F_\Theta$ is trained on images $\mathcal{I}=\{I_j\}_{j=1}^N$ from a single robot scan with camera poses $\Omega=\{\Omega_j\}_{j=1}^N$, learning a joint neural representation—scene field $\mathcal{S} \in \mathbb{R}^4$ and language field $\mathcal{L} \in \mathbb{R}^{d_l}$:

$$(\mathcal{S}, \mathcal{L}) = F_\Theta(\mathcal{I}, \Omega) \tag{1}$$

Given a user prompt $p$, the language field localizes the target object region $\mathcal{O} \subset \mathbb{R}^3$ by computing cosine similarity between text embedding $\phi(p) \in \mathbb{R}^{d_l}$ and language features $\mathcal{L}(\mathbf{x})$ at surface points, clustering high-relevance points, and fitting an oriented bounding box $b = (\mathbf{c}, \theta)$ where $\mathbf{c} \in \mathbb{R}^3$ is the center and $\theta \in \mathrm{SO}(2)$ is the yaw rotation. Unlike DFF's density suppression, NEO's resampling directly skips the object region:

$$\mathcal{O}' = \mathcal{O} \setminus \{b\}, \quad \text{sample}(\mathbf{r}) \cap b = \emptyset \tag{2}$$

This avoids floating artifacts from sparse sampling in DFF, providing a more reliable background representation.

3.2 Multiview-Consistent Progressive Inpainting

After object removal, the occluded background must be reconstructed. NEO uses multiview-consistent progressive inpainting: 2D inpainting models fill missing regions for views where the object was removed, then the results are distilled back into NeRF weights in a multiview-consistent manner. The inpainting loss is:

$$\mathcal{L}_{\text{inpaint}} = \sum_{j=1}^{N} \|I_j^{\text{inpaint}} - \tilde{F_\Theta}(\Omega_j)\|^2 \tag{4}$$

where $I_j^{\text{inpaint}}$ is the 2D inpainted result for view $j$ and $\tilde{F_\Theta}(\Omega_j)$ is the rendered image from pose $\Omega_j$. The process iterates: 2D inpainting first, then retraining NeRF $\tilde{F_\Theta}$ with inpainted images as supervision. Multiview consistency is enforced via the shared 3D representation:

$$\mathcal{L}_{\text{mvs}} = \sum_{j,k} \|\tilde{F_\Theta}(\Omega_j) - \mathcal{W}_{jk}(\tilde{F_\Theta}(\Omega_k))\|^2 \tag{5}$$

where $\mathcal{W}_{jk}$ is the warping from view $k$ to $j$.

3.3 Knowledge Distillation for Scene Reconfiguration

The scene reconfiguration module composes the original and edited NeRFs for object relocation modeling via a teacher-student model: the teacher is the original NeRF $F_\Theta$, the student is the edited NeRF $\tilde{F_\Theta}$. The knowledge distillation loss is:

$$\mathcal{L}_{\text{distill}} = \sum_{\mathbf{r}} \|F_\Theta(\mathbf{r}) - \tilde{F_\Theta}(\mathbf{r})\|^2 + \lambda \mathcal{L}_{\text{edit}} \tag{3}$$

where $\mathbf{r}$ are rays, $\mathcal{L}_{\text{edit}}$ is the inpainting supervision loss for edited regions, and $\lambda$ is a balancing weight. This avoids retraining from scratch while integrating edited changes into NeRF weights and preserving unedited regions. Conditioned on prompt $p$ and planned motion $\tau$, NEO produces an updated scene representation $(\mathcal{S}', \mathcal{L}') = F_\Theta(\mathcal{S}, \mathcal{L}, p, \tau)$, reusable as input for subsequent edits.

4. Experiments

4.1 Object Removal Results

Evaluated on five scenes against NeRFiller, Seal-3D, and DFF. NEO leads across all metrics—PSNR, SSIM, RMSE, and representation error $E_{\text{rep}}$—in both full-image and masked evaluation. Computationally, NEO completes inpainting in ~6 minutes vs. 13 minutes for DFF and ~1.5 hours for NeRFiller—a ~15x speedup.

Method	PSNR↑	SSIM↑	RMSE↓	$E_{\text{rep}}$↓	Time
NeRFiller	20.40	0.696	0.252	0.104	~90 min
Seal-3D	27.08	0.866	0.162	0.079	—
DFF	26.91	0.878	0.048	0.068	13 min
NEO	27.20	0.881	0.056	0.067	~6 min

4.2 Robotic Manipulation and Assembly Results

In pick-and-place against Seal-3D, NEO produces higher-quality reconstructions of moved objects—Seal-3D often leaves parts at the original location. Ablation shows mask-conditioned training is most critical. Consecutive assembly experiments demonstrate NEO's ability to predict future scene states—maintaining persistent NeRF+language field representations for iterative editing without re-scanning.

Metric	DFF	NeRFiller	Seal-3D	NEO
Floating artifacts	yes	yes	yes	no
Multiview consistent	no	partial	partial	yes
Continuous editing	no	no	no	yes
Speed	13min	90min	—	6min

5. Limitations

• Articulated objects: the framework focuses on rigid object relocation; kinematic editing of articulated objects is future work.
• Inpainting dependency: multiview inpainting relies on 2D inpainting model quality; complex occlusion scenarios may limit quality.
• Computational overhead: though 15x faster than baselines, each edit still takes ~6 minutes, limiting real-time operation.

6. Conclusion

NEO achieves a unified language-guided NeRF editing framework by combining neural field resampling, multiview-consistent progressive inpainting, and knowledge distillation for direct NeRF weight editing. Neural field resampling skips the object region to avoid floating artifacts; the teacher-student knowledge distillation integrates edited changes into NeRF weights while preserving unedited regions; conditioning on prompt and motion produces reusable updated representations for continuous editing. NEO-Dataset provides the first quantitative benchmark for NeRF editing suitable for robot manipulation. Object removal PSNR of 27.20 surpasses all baselines with a 15x speedup. The core insight: robotic manipulation doesn't require "re-scanning after manipulation"—NEO makes NeRF a persistently editable scene memory, letting robots "foresee" manipulation outcomes before execution, upgrading scene representation from passive snapshots to active prediction.

flowchart TD
    A["Single scan: images I + camera poses Omega"] --> B["Train NeRF F_Theta: scene field S + language field L"]
    B --> C["Language prompt p -> localize target region O"]
    C --> D["Oriented bounding box b = (c, theta)"]
    D --> E["Neural field resampling: skip object region"]
    E --> F["Multiview-consistent progressive inpainting"]
    F --> G["Knowledge distillation: teacher F_Theta -> student F_tilde"]
    G --> H["Updated representation S' L' = F(S, L, p, tau)"]
    H --> I["Robot manipulation execution"]
    I --> J["Subsequent edit: H as input"]
    J --> E