WARP-RM: Warp-Augmented Relative Progress Reward for Data Curation

Paper Meta

Title: WARP-RM: A Warp-Augmented Relative Progress Reward Model for Data Curation
Authors: Justin Yu, Andrew Goldberg, Kavish Kondap, Karim El-Refai, Ethan Ransing, Qianzhong Chen, Mac Schwager, Fred Shentu, Philipp Wu, Ken Goldberg
Affiliation: UC Berkeley AUTOLAB / BAIR; teleoperation data, training compute, and evaluation hardware from XDOF
Paper: arXiv:2606.28320
Project: uynitsuj.github.io/warp-rm
Code: released at github.com/uynitsuj/WARP-RM (training, LeRobot reward injection, WebUI inspector, simulation artifacts)

One-Sentence Summary

WARP learns a dense signed relative-progress velocity from successful demos via time-warp augmentations, then WARP-BC gates and reweights behavior-cloning action chunks so policies stay high-throughput as suboptimal teleoperation data grows.

Background and Motivation

Imitation learning is a strong recipe for long-horizon robot manipulation: policies fit visuomotor maps from human teleoperation. Better policy architectures and large-scale pretraining help expressivity and generalization, yet these systems remain sensitive to demonstration quality. Human teleops mix pauses, retries, and fumbles; if trained on indiscriminately, policies reproduce those inefficiencies, especially on long-horizon tasks.

Suboptimal segments are not pure noise. They often contain recovery behaviors reminiscent of DAgger-style corrective data. Coarse trajectory-level curation—discarding whole episodes below a quality threshold—fails in two ways at once: it throws away high-value progressive segments inside otherwise messy episodes, and it leaves local hesitations inside the episodes it keeps.

Recent work therefore learns frame-level progress for localized curation. Most progress reward models operate in an absolute-progress regime: ReWiND supervises with normalized episode duration; VIP and LIV use temporal contrastive alignment onto a global axis. Elapsed time is not task progress. Two demos at the same normalized frame index may sit at entirely different task stages because of pauses, failed grasps, or operator strategy differences—label noise follows. Denser models such as SARM reduce that noise with human subtask annotations, but annotation cost and inconsistency limit scale.

The paper’s claim is sharper: do not ask “what percent of the task is done?” Ask instead “relative to the start of this window, how fast and in which direction is the task moving?” If a model can mark progress, stall, and regression, behavior cloning can gate and reweight action chunks without subtask labels.

Figure 1: WARP-RM signed progress velocity $\hat{v}_t$ on an unseen mixed-quality demonstration. Large positives mark decisive progress; near zero marks stalls/fumbles; negatives mark regression.

Preliminaries

Behavior cloning and action chunks. Policy $\pi_\theta$ predicts a short action chunk $a$ (about one second here) from state $s$. The standard objective weights every sample equally, so hesitation-heavy demos inflate those behaviors in the fitted policy.

Flow-matching loss. Policy training uses a flow-matching objective $\mathcal{L}_{\mathrm{flow}}$. WARP-BC leaves the generator architecture unchanged and only multiplies each sample by a progress-derived weight $w(s,a)$.

Absolute vs relative progress. Absolute progress treats global time or a global embedding axis as completion. Relative progress only measures cumulative displacement from a window start. The latter is more robust to operator strategy differences and local pauses because labels come from replaying one successful demo at warped speeds, not from cross-episode frame indices.

Method

Overview and notation

WARP-RM is a vision model that estimates dense per-frame progress velocity from observations. Training is fully self-supervised on successful teleoperated demos. A demo is a length-$T$ RGB sequence $o_0,\ldots,o_{T-1}$ at fixed rate $f$; a frozen encoder $\phi$ yields per-frame features. A time-warp sampler draws window indices $i_0,\ldots,i_{N-1}$ (possibly non-monotonic). Pseudo-label $y_k$ is the normalized temporal displacement from window start $o_{i_0}$.

At inference, the model runs on linear windows with canonical stride $S$ seconds and aggregates predictions into $\hat{v}_t$. Calibration: $\hat{v}_t\approx 1$ matches average reference pace, $\hat{v}_t\approx 0$ stalls, and $\hat{v}_t<0$ regresses.

flowchart TD
  A[Successful demo video] --> B[Time-Warp Sampler
AR1 speeds + reversals]
  B --> C[Relative cumulative labels y_j]
  C --> D[Frozen DINOv3 + bidirectional Transformer]
  D --> E[Categorical bins to expectation y_hat]
  E --> F[Intra-window velocities v_j]
  F --> G[Overlap average to v_hat_t]
  G --> H[WARP-BC: terminal-velocity gate and reweight]
  H --> I[Weighted flow-matching BC]

Time-warp sampler

The sampler draws $N-1$ relative log-velocities from a stationary AR(1) process:

$$z_0\sim\mathcal{N}(0,\sigma_\infty^{2}),\quad z_k=\alpha z_{k-1}+\sqrt{1-\alpha^{2}}\,\sigma_\infty\,\epsilon_k,\quad\epsilon_k\sim\mathcal{N}(0,1)$$

Exponentiation yields positive playback speeds $\tilde{v}_k=e^{z_k}$. Log-space treats a $2\times$ speedup and a $0.5\times$ slowdown symmetrically; $\alpha$ controls smoothness of successive speed changes. A path-length budget $\ell\sim\mathrm{Unif}[fL/3,\,5fL/3]$ (capped at $T-1$) rescales speeds:

$$\tilde{u}_k=\ell\,\tilde{v}_k\Big/\sum_{j=0}^{N-2}\tilde{v}_j$$

Reversals supply negative displacement: draw $R=\min(R_0,N-2)$ reversal points with $R_0\sim\mathrm{Poisson}(\lambda_{\mathrm{rev}})$, and flip all signs with probability 0.5. After cumulative displacements $c_j=\sum_{k=0}^{j-1}u_k$, a valid start $i_0$ yields $i_j=\mathrm{round}(i_0+c_j)$.

Figure 2: Time-Warp Sampler. Variable playback spans slow-motion to fast-forward; random reversals expose negative progress. Accumulated speeds form a 32-frame source window whose offsets from the start are self-supervised progress labels.

In the open-source repo, the log-space AR(1) speed process and Poisson reversal points live in sample_speed_process and sample_reversal_signs (warp_rm/data/curve_parameterizations.py). The full sampler is ARSampler in warp_rm/data/samplers.py, with iid_speed=True as the IID ablation used in the paper.

# warp_rm/data/curve_parameterizations.py — AR(1) log-speeds
z[0] = rng.normal(0.0, sigma_inf)
noise_scale = math.sqrt(1 - alpha**2) * sigma_inf
for k in range(1, n_gaps):
    z[k] = alpha * z[k - 1] + noise_scale * rng.normal()
return np.exp(z)

Progress-model training

Relative cumulative targets normalize source-frame displacement by $C_{\mathrm{norm}}=f(N-1)S$:

$$y_j=(i_j-i_0)/C_{\mathrm{norm}},\qquad j=0,\ldots,N-1$$

Rather than regress $y_j$ directly, WARP-RM predicts a categorical distribution over evenly spaced bins and trains with cross-entropy against two-hot soft targets that interpolate between adjacent bin centers. Categorical training mitigates regression instability; the release implements this via soft_bins and C51-style CE in warp_rm/core/loss.py.

Architecture

Like SARM, the stack uses a frozen visual backbone plus a transformer temporal aggregator, but replaces stage classifiers with a single progress-velocity head. The backbone is DINOv3 ViT-B/16 (768-d). Each token concatenates the frame embedding with its temporal difference $[\phi(o_{i_j}),\,\phi(o_{i_j})-\phi(o_{i_{j-1}})]\in\mathbb{R}^{1536}$ (zero difference at $j=0$), then applies a linear projection, fixed sinusoidal positions, a bidirectional Transformer, and a linear categorical head. Inference takes the distributional expectation as $\hat{y}_j$.

Figure 3: WARP-RM architecture. A 32-frame window is encoded by frozen DINOv3 and a bidirectional Transformer that emits 30 cumulative-progress bins per frame. Differencing expectations yields intra-window velocities; averaging across overlapping windows produces $\hat{v}_t$.

Code mapping is direct: DINOv3 in warp_rm/models/backbones/dinov3.py; bidirectional aggregation and C51 heads in TransformerAggregator (warp_rm/models/aggregators/transformer.py), whose docstring lists temporal-diff concatenation, relative-progress bins, and an absolute-progress auxiliary head.

WARP-BC: curating BC with progress velocity

Overlapping windows contain $N$ frames separated by stride $S$ seconds, shifted by one source frame. Intra-window velocities are

$$v_j=(N-1)(\hat{y}_j-\hat{y}_{j-1})$$

Averaging all $v_j$ covering source frame $t$ yields dense $\hat{v}_t$. For one-second action chunks, the terminal-frame velocity $\hat{v}_{\mathrm{end}}$ defines continuous weights:

$$w(s,a)=\hat{v}_{\mathrm{end}}\cdot\mathbf{1}_{\hat{v}_{\mathrm{end}}>\tau}$$

The binary variant sets retained weights to one: $w(s,a)=\mathbf{1}_{\hat{v}_{\mathrm{end}}>\tau}$. Chunks with $w=0$ are filtered before training to preserve effective batch size. The policy loss is weighted flow matching:

$$\mathcal{L}_{\mathrm{BC}}=\mathbb{E}_{(s,a)\sim\mathcal{D}}\big[w(s,a)\cdot\mathcal{L}_{\mathrm{flow}}(\pi_\theta;s,a)\big]$$

Terminal-frame aggregation outperforms chunk-mean aggregation. $\hat{v}_{\mathrm{end}}$ is an empirical progress-velocity proxy for advantage, not an RL advantage with an explicit value baseline.

Experiments

Evaluation spans three settings: real bimanual T-shirt folding, real bottle-in-bin placement, and a reproducible MuJoCo bottle benchmark. Hardware uses dual I2RT YAM arms. Folding requires retrieving a crumpled shirt from a bin, flattening, folding sleeves, folding twice, and moving the shirt to the top-left; trials time out at 240 seconds. Throughput counts successful folds per hour and charges failures the full timeout.

Figure 4: Completion-time distributions for successful trials across tiers $\mathcal{D}_1$–$\mathcal{D}_3$ with increasing demonstration sub-optimality.

Dataset	Method	Success	Mean TTC (s)	Thrput (/hr)	Chunks kept
$\mathcal{D}_1$ (≤60s)	Vanilla BC	20/20	113.8	31.6	100%
	WARP-BC	20/20	63.9	56.3	35.7%
$\mathcal{D}_2$ (≤90s)	Vanilla BC	2/20	199.0	1.5	100%
	WARP-BC	19/20	118.8	27.4	34.4%
$\mathcal{D}_3$ (≤120s)	Vanilla BC	0/20	—	0.0	100%
	WARP-BC	14/20	117.4	16.3	22.5%

Table 1: Cross-tier T-shirt folding. As suboptimal demos enter training, vanilla BC collapses; WARP-BC yields ~18× throughput on $\mathcal{D}_2$ (27.4 vs 1.5).

Against SARM, DemInf, and SCIZOR on matched sets $\mathcal{D}_4,\mathcal{D}_5$, SARM and SCIZOR fall from 19/20 to 2/20 on the broader tier, DemInf keeps 18/20, and WARP-BC reaches 20/20 with the highest throughput. Retention is matched to DemInf; the simulation study further locks all curation methods to 31.5% retention.

Method	Data kept	Bottles/scene	Thrput (/hr)	All 6 cleared
Vanilla BC	100%	3.885	237	9.4%
Random	31.5%	3.770	230	10.9%
ReWiND	31.5%	3.781	231	9.4%
SARM (oracle)	31.5%	4.191	265	20.5%
DemInf	31.5%	4.332	271	18.8%
WARP-BC	31.5%	4.533	290	25.0%

Table 2: Simulated bottle-in-bin (512 paired scenes). Curation methods keep 31.5% of data; WARP-BC gains +53 bottles/hr over vanilla BC (95% CI [42, 64]).

Figure 5: Real bottle-in-bin placement times. WARP-BC places 74/80 bottles at 11.3s mean and 237.8/hr throughput versus 59/80, 15.9s, and 147.8/hr for vanilla BC.

Ablations show continuous weighting at $\tau=1$ beats binary gating and $\tau=0$; terminal-velocity aggregation beats chunk means; AR(1) sampling beats IID at matched retention in simulation (290 vs 269 bottles/hr). On 452 independently annotated expert folding episodes, the progress signal’s frame-level mistake AUROC is 0.83, above proprioceptive speed (0.61) and garment-area change (0.68).

Limitations

The authors note that WARP-BC only attenuates suboptimal data; the policy remains confined to behaviors present in the offline set. Future work should combine DAgger, offline/online RL, and reward-aligned iterative self-improvement to generalize beyond those demos.

Negative-progress supervision comes entirely from reversed playback—an approximation borrowed from ReWiND that can be physically implausible. Real regressions unfold forward in time under causal dynamics, so warped reversals create a distribution gap. The paper argues the overlap is often enough in practice, but the alignment is task-dependent and should be validated before deployment.

Experiments cover two real manipulation tasks and one simulated task on a single bimanual embodiment; cross-embodiment transfer remains open. With $n=20$ hardware trials, near ties in success counts can sit inside binomial noise, so throughput and completion time are more informative when success rates are close.

Conclusion

WARP reframes “where is this demo progressing?” as a learnable relative-velocity problem and uses that signal for fine-grained BC curation. Versus trajectory filters and absolute-progress models that need subtask labels, it stays robust as suboptimal demos grow, with consistent throughput gains on real folding and real/sim bottle placement. The public repo wires the sampler, C51 loss, DINOv3+Transformer stack, and LeRobot annotation injection into a reproducible pipeline—score your own demos before wiring the weights into policy training.

Golden Line

Elapsed time is not task progress; learn relative window velocity, and keep only the action chunks that actually move the task.