PhysCoRe: Physics-Corrected Residual World Models for Material-Aware Deformable Dynamics

PhysCoRe: Physics-Corrected Residual World Models for Material-Aware Deformable Dynamics

Paper: PhysCoRe: Physics-Corrected Residual World Models for Material-Aware Deformable Dynamics

Authors: Haocheng Yin, Shuohan Tao, Yongsheng Chen, Lu Gan

Affiliation: Georgia Institute of Technology

Links: arXiv:2607.20653

---

One-Sentence Summary

PhysCoRe couples a differentiable Material Point Method (MPM) simulator with two feed-forward neural networks — MfM infers per-particle material parameters and confidence from visual observations, while RfD injects bounded velocity residuals within the MPM cycle to correct sim-to-real gaps — achieving higher future prediction accuracy than PhysTwin and PGND on real deformable-object manipulation sequences while reducing material identification time from hundreds of seconds to 11.4s, and supporting confidence-guided active exploration.

---

Background and Motivation

To manipulate a deformable object with intent, a robot must anticipate how that object will move before committing to an action. Such anticipation relies on a dynamics model that maps a planned interaction to the resulting deformation, and its accuracy limits how good the plan can be. This requirement is particularly demanding for deformable objects: their behavior is governed by underlying material properties such as stiffness and plasticity, which differ from one object to the next, can vary even within a single body, and are not directly readable from RGB-D observations.

Existing methods pursue two largely separate lines. The first is physics-based modeling: representing the object with an analytical model (spring-mass, FEM, PBD, MPM) and recovering material parameters from observed motion through differentiable simulation or gradient-free optimization. These methods are interpretable and physically plausible, but identification is typically slow, object-specific, and constrained by the simulator's discretization and constitutive assumptions. The second is data-driven modeling: learning dynamics directly from interaction data via graph neural networks and similar architectures. These avoid calibration but require large datasets, generalize poorly out of distribution, and lack physical structure.

The core tension: no existing approach identifies material quickly enough to transfer across objects while keeping predictions grounded in physics. Physics methods are slow and non-generalizable; learning methods violate physical structure. PhysCoRe's design philosophy combines both strengths — retaining a physical simulator to advance dynamics while delegating material identification and residual correction to learned modules.

Figure 1: Overview of PhysCoRe. From segmented RGB-D observations, MfM infers the object's per-particle material. A differentiable MPM simulator then rolls out action-conditioned dynamics, with RfD injecting a velocity residual $\Delta\mathbf{v}_i$ to correct the grid velocity.

---

Preliminaries: Material Point Method (MPM)

PhysCoRe builds on a differentiable MPM simulator, inheriting its particle-grid state, B-spline kernel, constitutive stress model, and plasticity return map. One MPM cycle comprises four steps: particle-to-grid (P2G), grid update, grid-to-particle (G2P), and particle update.

In the particle update, positions and deformation gradients are updated as:

$$ \mathbf{x}_{p}^{h+1}=\mathbf{x}_{p}^{h}+\Delta t\,\mathbf{v}_{p}^{h+1},\qquad\mathbf{F}_{p}^{h+1}=\mathcal{P}\!\Bigl(\bigl(\mathbf{I}+\Delta t\sum_{i}\mathbf{v}_{i}\otimes\nabla w_{p,i}\bigr)\,\mathbf{F}_{p}^{h}\Bigr) $$

where $\mathbf{x}_p^h$ and $\mathbf{v}_p^h$ are particle $p$'s position and velocity at substep $h$, $\mathbf{F}_p^h$ is the deformation gradient, $w_{p,i}$ is the B-spline kernel, and $\mathcal{P}(\cdot)$ is the plasticity return map — either leaving $\mathbf{F}_p$ unchanged (Fixed Corotated Elasticity) or projecting it onto the von Mises Plasticity yield surface. MfM selects the branch per episode. The discrete action sequence $\mathbf{a}_{0:T-1}$ is temporally upsampled to the substep rate via Catmull-Rom spline.

---

Method

1. Problem Formulation

Given an initial multi-view RGB-D observation $\mathbf{O}_0$ of a deformable object and a proposed action sequence $\mathbf{a}_{0:T-1}$ from a robot gripper or human hand, the goal is to learn a physics-based dynamics model that predicts future particle configurations $\mathbf{X}_{1:T}$, where $\mathbf{X}_t = \{\mathbf{x}_p^{(t)}\}_{p=1}^{N}$ are the positions of $N$ continuum particles at camera frame $t$. Since material properties are not directly observable, the model infers them from observations and parameterizes the simulator accordingly, keeping action-conditioned predictions physically consistent.

2. MfM Module (Material from Motion)

MfM estimates an object's material from a short window of observed motion. Canonical positions and per-frame displacements of tracked and controller points are encoded via Fourier features, then fed to a graph U-Net that gives more weight to tracked points and particles near controllers during message passing, as these are most informative about material. A temporal module combines a depthwise 1D convolution with GRU cells to produce a single per-particle latent. A shared MLP decodes it into per-particle material $\boldsymbol{\phi}_p = (\log E_p, \nu_p)$:

$$ \log E_{p}=\mu_{\log E}+\sigma_{\log E}\,z_{p}^{(\log E)},\qquad\nu_{p}=\mu_{\nu}+\sigma_{\nu}\,z_{p}^{(\nu)} $$

where $E_p$ is Young's modulus, $\nu_p$ is Poisson's ratio, and $z_p$ is a spatially correlated field generated from multi-octave Perlin noise. A bounded sigmoid constrains parameters within sampled ranges. MfM also predicts a per-particle confidence $\mathbf{c}_p$ (one value per parameter), used to reweight the material loss during training and to flag least-confident regions at inference. Additionally, the per-particle latent is pooled across all particles and fed to a small MLP that outputs a per-episode probability $\pi$ of elastic versus plastic behavior.

3. RfD Module (Residual from Dynamics)

RfD closes the residual sim-to-real gap left by the analytical simulator. It acts inside the MLS-MPM cycle, between the grid update and the G2P gather. For each active cell it builds a compact feature $\mathbf{f}_i$ gathering the local grid state, MfM material estimates, and nearby contact geometry. A FiLM-conditioned sparse 3D U-Net maps this feature to a grid-velocity residual:

$$ \Delta\mathbf{v}_{i}=\delta_{\max}\cdot\tanh\!\Bigl(\mathcal{R}_{\psi}(\mathbf{f}_{i};\,\mathbf{u})\Bigr) $$

where $\mathbf{u}$ is a global context vector summarizing the current simulation regime. The scaled $\tanh$ limits every component of $\Delta\mathbf{v}_i$ to a maximum magnitude $\delta_{\max}$, ensuring the learned correction stays small enough not to destabilize the rollout. The network's output layer is zero-initialized, so the residual starts at exactly zero — RfD begins as an identity correction that leaves the analytical simulator unchanged. The corrected velocity $\mathbf{v}'_i = \mathbf{v}_i + \Delta\mathbf{v}_i$ replaces the analytical velocity in G2P and particle update. Because the correction is applied on the grid and carried to particles via the same APIC transfer, it preserves the momentum and angular-momentum structure of MLS-MPM to first order. RfD is applied periodically rather than at every substep, bounding the backpropagation graph while still exposing accumulated errors within each window.

4. Material Augmentation

Supervised material data is scarce, as measuring spatially varying stiffness and Poisson's ratio of everyday objects at scale is impractical. PhysCoRe generates 1,260 augmented episodes from 14 real PhysTwin episodes, each keeping the source capture's geometry and controller motion while varying only material. For each episode, a spatially correlated material field is drawn from multi-octave Perlin noise over canonical particles and affinely mapped to per-particle parameters. The MPM simulator is then rolled out under the same real controller action trajectory, recording per-particle yield events as an episode-level plasticity label $\pi^\star$, and re-rendering the result from saved camera views to produce per-frame point clouds for training supervision.

5. Two-Stage Training

MfM is first pre-trained in simulation against ground-truth materials on the augmented dataset, then RfD is trained on real episodes with MfM frozen. The MfM refinement loss is:

$$ \mathcal{L}_{\mathrm{refine}}=\mathbb{E}\!\left[\frac{1}{N}\sum_{p=1}^{N}\!\Bigl(\,\mathbf{c}_{p}\cdot\mathrm{SL1}\bigl(\bar{\boldsymbol{\phi}}_{p},\bar{\boldsymbol{\phi}}^{\star}_{p}\bigr)-\lambda\log\mathbf{c}_{p}\Bigr)\right]+\beta\,\mathrm{BCE}\bigl(\pi,\pi^{\star}\bigr) $$

where $\bar{\boldsymbol{\phi}}_p$ and $\bar{\boldsymbol{\phi}}^\star_p$ are predicted and ground-truth materials normalized to $[0,1]$, $\mathrm{SL1}$ is Smooth-L1 error, and $\mathbf{c}_p$ is the confidence weight. The log-confidence regularizer prevents $\mathbf{c}_p$ from inflating to trivially shrink the error. During RfD training, each episode is rolled out from the first frame with MPM, processed in windows of $K$ frames — at each window start MfM refreshes predictions, and RfD applies corrections every $H_r$ substeps. The per-frame loss combines a one-sided Chamfer distance from the point cloud with mean L2 error on visible tracked particles:

$$ \mathcal{L}_{\mathrm{corr}}^{(t)}=w_{\mathrm{C}}\,\mathrm{Chamfer}\bigl(\hat{\mathbf{Y}}_{t},\,\mathbf{X}_{t}\bigr)+w_{\mathrm{L2}}\,\frac{1}{|\mathcal{T}_{t}|}\!\sum_{p\in\mathcal{T}_{t}}\!\bigl\|\mathbf{x}_{p}^{(t)}-\hat{\mathbf{x}}_{p}^{(t)}\bigr\|_{2} $$

flowchart TD
    A["RGB-D multi-view observations"] --> B["Segmentation + Tracking
Grounded SAM2 + CoTracker3"]
    B --> C["MfM Module
Graph U-Net + GRU"]
    C --> D["Per-particle material phi_p
(log E, nu)"]
    C --> E["Per-particle confidence c_p"]
    C --> F["Elastic/plastic probability pi"]
    D --> G["Differentiable MPM simulator
P2G -> Grid update -> G2P -> Particle update"]
    E --> G
    F --> G
    G --> H["RfD Module
FiLM sparse 3D U-Net"]
    H --> I["Velocity residual Delta v_i
= delta_max * tanh(R(f_i))"]
    I --> J["Corrected grid velocity
v'_i = v_i + Delta v_i"]
    J --> G
    G --> K["Future particle configs X_1:T"]
    K --> L["3DGS rendering
visual comparison"]
    style C fill:#e1f5fe
    style H fill:#fff3e0
    style G fill:#e8f5e9

---

Experiments

Experimental Setup

The real-world dataset consists of 12 episodes of deformable object manipulation by human hand, covering elastic objects (a rope, a towel, and a plush toy bear) and elastoplastic objects (Play-Doh plasticines), with actions including lifting, pushing, stretching, and squeezing. Online confidence-guided exploration uses a KUKA arm with a Robotiq Hand-E gripper. Each episode is captured in RGB-D by three Intel RealSense D455 cameras; CoTracker3 provides tracked surface points and Grounded SAM2 provides object masks. Baselines are PhysTwin and PGND.

Table 1: Capability comparison — only PhysCoRe has all four capabilities

Method	Simulation	Online Refinement	Feed-Forward	Generalization
PGND	✗	✗	✓	✗
PhysTwin	✓	✗	✗	✗
EMPM	✓	✓	✗	✗
PhysCoRe	✓	✓	✓	✓

Future Prediction

Each method identifies material from the first 50% of an episode (identification window) and predicts the held-out remainder with it fixed. Geometric accuracy is measured by Chamfer distance (CD) and tracking loss; visual quality by IoU, PSNR, SSIM, and LPIPS.

Figure 2: Qualitative rollout comparison. Predicted object positions are rendered with 3DGS. Compared with PhysTwin, PhysCoRe predicts future dynamics that more closely match real observations.

Table 2: Quantitative results on future prediction — PhysCoRe achieves best overall performance

Method	Object Type	CD↓	Track↓	IoU%↑	PSNR↑	SSIM↑	LPIPS↓
PGND	Elastic	0.03420	0.13175	37.8	10.74	0.103	0.416
PhysTwin	Elastic	0.01617	0.03016	70.5	14.15	0.309	0.199
PhysCoRe	Elastic	0.00910	0.02557	68.2	14.47	0.302	0.194
PGND	Elastoplastic	0.04311	0.13263	23.9	9.87	0.081	0.440
PhysTwin	Elastoplastic	0.00778	0.01821	63.1	14.16	0.285	0.154
PhysCoRe	Elastoplastic	0.00541	0.01670	66.1	15.02	0.286	0.130

PhysCoRe reduces CD by 43.7% on elastic objects and 30.5% on elastoplastic objects relative to PhysTwin, with tracking loss reductions of 15.2% and 8.3% respectively. The largest gains are on elastoplastic objects — precisely where the analytical simulator faces the widest sim-to-real gap. Feed-forward inference is also far cheaper: material identification takes 11.4s versus 930.0s for PhysTwin and 8280.0s for PGND, making online adaptation to new objects practical.

RfD Residual Correction Ablation

Table 3: RfD ablation — RfD further reduces CD

Method	Elastic CD↓	Elastic Track↓	Elastoplastic CD↓	Elastoplastic Track↓
PhysCoRe (MfM only)	0.01047	0.02723	0.00658	0.01981
PhysCoRe (MfM + RfD)	0.00910	0.02557	0.00541	0.01670

MfM alone is already a strong physics-grounded predictor; adding RfD further reduces CD by 13.1% on elastic objects and 17.8% on elastoplastic objects. The larger elastoplastic gain confirms the intended division of labor: MfM identifies object-specific material, while RfD absorbs the residual errors that simplified analytical MPM leaves from contact, friction, and irreversible deformation.

Figure 3: Capability comparison table and experimental setup visualization.

Figure 4: Future prediction quantitative results table and confidence visualization.

---

Limitations

1. Limited material types (stated by authors): PhysCoRe currently focuses mainly on elastic and elastoplastic objects, relying on a predefined set of material behaviors, and does not directly extend to objects or scenes involving tearing, cutting, adhesion, fluid-like motion, or highly complex contacts.
2. Unverified generalization scope (stated by authors): While experiments demonstrate fast adaptation to new objects and manipulation interactions, the current evaluation is limited to in-category objects and does not test zero-shot category-level transfer.
3. Residual module scalability (stated by authors): RfD is currently trained on a limited set of deformable objects, and scalability to broader dynamics distributions remains unexplored.

---

Conclusion and Outlook

PhysCoRe couples a differentiable MPM simulator with two feed-forward modules — MfM infers per-particle material from visual observations, and RfD corrects the simulator's internal dynamics. This design predicts deformable-object behavior accurately while retaining physical structure, and transfers to unseen objects without per-object optimization. On real-world manipulation sequences, PhysCoRe outperforms SOTA baselines in prediction accuracy, and its predicted confidence reliably reflects where the material estimate can be trusted, showing potential for confidence-guided exploration and active learning.

Golden quote: "RfD's output layer is zero-initialized, so the residual starts at zero — it begins as an identity correction that leaves the analytical simulator unchanged, then gradually learns to absorb the sim-to-real gap, ensuring physical structure is preserved while gaining data-driven accuracy."