RealVDeblur: One-Step Diffusion for Generalizable Real-World Video Deblurring

RealVDeblur: One-Step Diffusion for Generalizable Real-World Video Deblurring

Paper: RealVDeblur: One-Step Diffusion for Generalizable Real-World Video Deblurring

Authors: Renbiao Jin, Mingxin Yang, Yutian Chen, Junhao Zhuang, Xin Cai, Mulin Yu, Linning Xu, Wenxian Yu, Danping Zou, Shi Guo, Tianfan Xue

Affiliations: Shanghai Jiao Tong University / Shanghai AI Laboratory / CUHK MMLab / Tsinghua University

Links: arXiv:2607.20628 · Project Page

---

One-Sentence Summary

RealVDeblur repurposes a pre-trained Wan2.1 video diffusion model as a generative prior for real-world video deblurring through three key designs — disabling VAE temporal compression for frame-wise encoding, distilling to a one-step generator via Distribution Matching Distillation, and a training-free Temporal Window Mask — paired with the OmniBlur dataset synthesized from 3DGS assets and high-frame-rate videos, achieving strong cross-dataset generalization and improving downstream 3DGS reconstruction robustness under severe motion blur.

---

Background and Motivation

Real-world video deblurring is an important problem in computer vision. Motion blur frequently occurs in practical video capture due to camera shake, dynamic scenes, and long exposure. The task aims to recover sharp frames from blur-degraded sequences. Video deblurring plays a critical role in downstream applications: modern mobile imaging and video enhancement pipelines rely on stable sharp frames, and 3D reconstruction is particularly sensitive to motion blur, which significantly degrades geometric accuracy.

Video deblurring methods have progressed from explicit motion alignment (CDVD-TSP, EDVR), to implicit correspondence learning (STFAN, PVDNet), and more recently to transformer-based spatio-temporal aggregation (VRT, RVRT, BSSTNet). Although these approaches achieve strong results on synthetic benchmarks, their performance often degrades on real-world videos, showing limited generalization and unsatisfactory deblurring quality. This limitation is mainly caused by two factors.

The first factor is training data: available training data is limited in scale and often mismatched to real motion blur. Existing models are typically trained on datasets captured from a small number of scenes (e.g., GoPro contains only 20 scenes), providing insufficient diversity for learning robust motion statistics. Moreover, common synthetic blur methods (frame averaging) differ physically from real blur, causing a noticeable synthetic-to-real domain gap. The second factor is the lack of a realistic video prior: regression models that directly map blur to sharp tend to produce over-smoothed results under severe degradation, while diffusion models, though rich in priors, assume smooth inter-frame transitions in their 3D VAE temporal compression — an assumption frequently violated in blurry sequences.

Figure 1: Visual comparison of RealVDeblur against state-of-the-art video deblurring methods in real-world scenarios. While traditional regression-based methods produce over-smoothed results, this generative framework faithfully recovers sharp structures and high-frequency details.

---

Preliminaries

RealVDeblur is built upon Wan2.1, a large-scale Diffusion Transformer (DiT) trained via Flow Matching on large-scale image and video datasets. The model operates on latent representations produced by a Causal 3D VAE (Wan-VAE). Given a video $V \in \mathbb{R}^{T \times 3 \times H \times W}$, it is first encoded into a compact latent representation $z$ and then patchified into a sequence of latent tokens. The DiT processes these tokens through a stack of transformer blocks, each comprising spatio-temporal self-attention, cross-attention for text conditioning, and a feed-forward network.

The Flow Matching training objective defines an interpolated latent over continuous time $\tau \in [0,1]$:

$$ x_{\tau} = (1-\tau)\, x_{0} + \tau\, \epsilon $$

where $x_0$ is the clean latent, $\epsilon$ is noise, and the target velocity is $u_{\tau} = \epsilon - x_0$. The model learns to predict the velocity field $v_{\theta}(x_{\tau}, \tau, z_{\mathrm{blur}})$. This framework provides a powerful spatio-temporal prior for video generation that RealVDeblur repurposes for deblurring.

---

Method

1. Conditioning Injection and Parameter-Efficient Fine-Tuning

To enable conditional video deblurring, the blurry latent $z_{\mathrm{blur}}$ is projected to the latent feature space using a 3D convolutional network. The resulting embedding is added to the noisy latent $x_{\tau}$ before being fed into the DiT. Parameter-efficient fine-tuning uses LoRA modules attached to the transformer blocks, while the pre-trained VAE remains frozen. Text guidance is disabled via a null text embedding. Training follows the Flow Matching objective:

$$ \mathcal{L}_{\mathrm{FM}} = \mathbb{E}_{\tau,\epsilon}\, \bigl\lVert v_{\theta}(x_{\tau}, \tau, z_{\mathrm{blur}}) - u_{\tau} \bigr\rVert_{2}^{2} $$

Here $z_{\mathrm{blur}}$ is the blurry video conditioning latent, and $u_{\tau} = \epsilon - x_0$ is the target velocity. LoRA (rank 64) fine-tunes only a small number of parameters to adapt the DiT to deblurring while preserving the pre-trained prior.

2. Frame-wise Latent Encoding (Key Innovation 1)

Figure 2: Overview of the RealVDeblur framework — three key components: frame-wise VAE encoding, DMD one-step distillation, and training-free Temporal Window Mask.

Modern video diffusion models commonly employ 3D VAEs to jointly compress videos along spatial and temporal dimensions. For example, Wan-VAE performs temporal downsampling ($\times 4$) to obtain compact spatio-temporal latents. However, such temporal compression is less suitable for video restoration, particularly motion deblurring. Motion blur integrates scene content over time, causing large differences between adjacent frames and making temporal compression in the VAE prone to information loss.

To address this, RealVDeblur removes temporal compression in the VAE and instead adopts a frame-wise 2D encoder. For an input video $V = \{y_t\}_{t=1}^{T}$, each frame is encoded independently and the resulting latents are concatenated along the temporal dimension:

$$ z_{\mathrm{blur}} = \mathrm{concat}\bigl(E(y_{1}), \ldots, E(y_{T})\bigr) \in \mathbb{R}^{T \times C \times H' \times W'} $$

where $E$ denotes the frame-wise 2D VAE encoder. The key insight is that Wan2.1 is trained on both images and videos, where images can be regarded as videos with temporal length one, so the pre-trained DiT naturally supports frame-wise latent representations without introducing a latent distribution mismatch. Ablation confirms this is the highest-contribution design of all components — replacing frame-wise encoding with the default Causal 3D VAE causes a 4.87 dB PSNR drop.

3. One-Step Distillation Inference (Key Innovation 2)

Deploying multi-step VDMs faces two efficiency challenges: the high latency of iterative sampling (e.g., 50 denoising steps) and the poor scalability of global attention with RoPE extrapolation on long sequences. RealVDeblur adopts Distribution Matching Distillation (DMD) to distill the multi-step diffusion process into a one-step generator. DMD trains a Student model to match the output distribution of a frozen Teacher via the score difference between the Teacher and a learned Critic.

In this implementation, Student, Teacher, and Critic share the same frozen DiT backbone and differ only in their independently trainable LoRA parameters. Starting from pure noise $z$, the Student directly predicts the clean latent $\hat{x}_0$ in a single step. The Student is trained using the DMD objective together with pixel-space losses computed on VAE-decoded frames:

$$ \mathcal{L}_{\text{student}} = \mathcal{L}_{\text{DMD}} + \lambda_{L1}\, \mathcal{L}_{L1} + \lambda_{LPIPS}\, \mathcal{L}_{LPIPS} $$

where $\lambda_{L1} = \lambda_{LPIPS} = 2$. The Critic is trained on the Student's detached outputs using a standard Flow Matching objective. Counter-intuitively, the distilled one-step model slightly outperforms the 50-step Teacher — attributed to the pixel-space supervision ($\ell_1$ and LPIPS) introduced during distillation, which provides direct image-domain gradients absent in the original latent-space Flow Matching training.

4. Training-Free Temporal Window Mask (Key Innovation 3)

For video deblurring, each frame's restoration primarily depends on temporal redundancy from nearby frames rather than global context across the entire sequence. Directly applying VDMs to long video sequences faces two challenges: the quadratic memory cost of global self-attention, and performance degradation from RoPE extrapolation when processing sequences longer than those seen during training. RealVDeblur introduces a Temporal Window Mask (TWM) that replaces global self-attention with local window attention at inference — for each frame $t$, attention is restricted to a temporal window of size $W$:

$$ \text{Attn}(Q_{t}, K, V) = \text{Softmax}\left(\frac{Q_{t}\, K_{[t-W/2,\, t+W/2]}^{T}}{\sqrt{d}}\right) V_{[t-W/2,\, t+W/2]} $$

By constraining attention to a fixed window $W=20$ matching the training sequence length, the relative distances between frames during inference remain strictly within the well-trained manifold of the rotary embeddings. This alignment between inference and training positional distributions effectively eliminates instability caused by RoPE extrapolation. The model thus maintains consistent restoration quality on arbitrarily long videos with constant memory. Ablation shows: global attention without TWM produces severe artifacts on 150-frame videos; a narrow window $W=5$ avoids extrapolation but limits temporal aggregation leaving residual blur; $W=20$ achieves the best balance.

5. Large-Scale Physically Grounded Blur Synthesis (OmniBlur Dataset)

Real-world video blur arises from three physical mechanisms: camera shake, object motion, and optical defocus. Existing training datasets cover only a subset of these degradations and are drawn from a limited number of scenes. RealVDeblur builds a large-scale training set OmniBlur from two complementary data sources: 3DGS reconstructions for camera-motion and defocus blur, and high-frame-rate videos for object-motion blur.

For camera-motion blur, a smooth camera trajectory is generated through each 3DGS scene with random 6-DoF perturbations via Bézier curves to mimic hand-held shake; $K$ sub-frames are rendered and averaged in linear space to produce the blurry frame. For defocus blur, a thin-lens camera model with per-pixel depth from 3DGS rendering is used: $N$ views are sampled on a circular aperture disk around a randomly chosen focal plane, producing realistic depth-dependent bokeh. Since both often co-occur, compound blur is also generated ($K \times N$ renderings per frame). Object-motion blur is synthesized by averaging high-frame-rate video frames. OmniBlur ultimately contains approximately 2,000 3DGS scenes and 3,000 high-frame-rate videos.

The blur synthesis model can be formalized as:

$$ \mathbf{B} = \frac{1}{N}\sum_{i=1}^{N} \mathbf{S}_i \circ \mathbf{H}_i + \mathbf{n} $$

where $\mathbf{S}_i$ is the $i$-th high-frame-rate sharp frame, $\mathbf{H}_i$ is a homography transform, $\mathbf{n}$ is noise, and $N$ is the number of frames within the exposure. Rendered images are noise-free and skip the camera ISP, creating a gap with real captures; therefore, following RSBlur, Poisson-Gaussian noise is injected in the RAW domain and the ISP (mosaicing, white balance, demosaicing) is simulated to produce realistic training pairs.

flowchart TD
    A["3DGS scene assets"] -->|"Bezier 6-DoF perturbation
sub-frame averaging"| B["Camera-motion blur"]
    A -->|"Thin-lens model
aperture disk sampling"| C["Defocus blur"]
    D["High-frame-rate videos"] -->|"frame averaging"| E["Object-motion blur"]
    B --> F["OmniBlur dataset
~2000 scenes + ~3000 videos"]
    C --> F
    E --> F
    F --> G["ISP-aware noise augmentation
Poisson-Gaussian + demosaicing"]
    G --> H["Realistic training pairs"]
    H --> I["Frame-wise VAE encoding
temporal compression disabled"]
    I --> J["DiT conditional denoising
LoRA fine-tuning"]
    J --> K["Multi-step diffusion Teacher"]
    K -->|"DMD + pixel-space losses"| L["One-step Student generator"]
    L --> M["Temporal Window Mask W=20
stable long-video inference"]
    M --> N["Sharp video output"]
    style I fill:#e1f5fe
    style L fill:#fff3e0
    style M fill:#e8f5e9

---

Experiments

Real-World Video Deblurring Benchmarks

Experiments are evaluated on five real-world benchmarks — BSD, RealBlur, RSBlur, FEVD, and RWBI — rather than synthetic ones like GoPro/DVD. Evaluation metrics cover PSNR, SSIM (distortion), LPIPS, FID (perceptual), MUSIQ, NIQE (no-reference quality), and tOF (temporal consistency). RealVDeblur achieves the best or second-best results on nearly all metrics across all benchmarks, demonstrating strong cross-dataset generalization.

Table 1: Quantitative comparison on real-world video deblurring benchmarks (BSD dataset)

Method	Train Data	PSNR↑	SSIM↑	LPIPS↓	FID↓	MUSIQ↑	NIQE↓	tOF↓
ESTRNN	BSD	31.39	0.936	0.114	10.8	43.88	5.793	2.146
VRT	REDS	26.72	0.850	0.199	33.5	39.34	5.605	2.583
RVRT	GoPro	26.06	0.849	0.196	28.6	39.72	5.563	3.725
BSSTNet	GoPro	20.90	0.707	0.320	60.5	38.16	7.370	6.532
RealVDeblur	OmniBlur	28.76	0.884	0.118	9.1	52.49	4.583	1.974

Three key findings corroborate the design motivations. First, methods trained on synthetic datasets (e.g., BSSTNet, RNN-MBP on GoPro) suffer notable degradation on real-world benchmarks, confirming the domain gap between synthetic frame-averaging blur and real motion blur, validating the physically grounded data construction pipeline. Second, ESTRNN trained on the real-world BSD dataset achieves the highest PSNR on BSD (31.39) but consistently underperforms in perceptual metrics (MUSIQ 43.88 vs. our 52.49), suggesting that deterministic regression without a video prior tends to produce over-smoothed textures, supporting the adoption of a diffusion-based generative framework. Third, RealVDeblur achieves the best temporal consistency (tOF) on all four benchmarks, demonstrating that the distilled one-step inference with the temporal window mask effectively maintains inter-frame coherence.

Figure 3: Qualitative comparison on multiple real-world evaluation datasets. Under severe conditions including low light and overexposure, baseline models produce heavily smoothed or distorted outputs, while RealVDeblur recovers sharp high-frequency details and maintains geometric consistency.

Table 2: Reference-free RWBI benchmark evaluation

Method	Train Data	MUSIQ↑	CLIP-IQA↑	NIQE↓
ESTRNN	BSD	47.85	0.356	5.759
VRT	REDS	45.87	0.250	4.913
RVRT	GoPro	45.59	0.257	5.356
BSSTNet	GoPro	43.58	0.215	5.774
RealVDeblur	OmniBlur	63.50	0.510	4.250

On the RWBI benchmark, RealVDeblur surpasses all baselines by a significant margin (MUSIQ: 63.50 vs. 47.85, CLIP-IQA: 0.510 vs. 0.356), further confirming the real-world generalization capability enabled by the large-scale training data and generative prior.

Downstream 3DGS Reconstruction Evaluation

Figure 4: Visual evaluation of 3DGS reconstruction under real-world camera-motion blur (top row) and defocus blur (bottom row).

Table 3: Quantitative comparison of 3DGS reconstruction quality on the DeblurNeRF dataset

Method	Blur Type	PSNR↑	SSIM↑	LPIPS↓
BAGS	Real Camera Motion	24.98	0.7458	0.2310
Deblurring-3DGS	Real Camera Motion	25.67	0.7695	0.2231
VRT + 3DGS	Real Camera Motion	22.56	0.7220	0.3225
RealVDeblur + 3DGS	Real Camera Motion	25.41	0.8095	0.1628
BAGS	Real Defocus	23.32	0.7242	0.1846
RealVDeblur + 3DGS	Real Defocus	22.87	0.8335	0.1798

RealVDeblur as a preprocessing module for 3DGS pipelines consistently outperforms end-to-end methods on camera-motion blur and achieves competitive performance on defocus blur, without requiring any task-specific joint optimization — demonstrating it is a strong general-purpose preprocessing module.

Ablation Studies

Table 4: Ablation of proposed components (BSD benchmark)

Variant	PSNR↑	SSIM↑	LPIPS↓	tOF↓	Time/Frame
w/o Frame-wise VAE (Causal 3D VAE)	23.89	0.748	0.262	9.106	-
w/o 3DGS-Rendered Data	26.46	0.810	0.188	2.782	-
w/o DMD (50-step Teacher)	28.02	0.873	0.135	2.097	3.53s
w/o TWM (Global Attention)	27.20	0.845	0.165	3.786	0.29s
RealVDeblur (Full Model)	28.76	0.884	0.118	1.974	0.15s

Figure 5: Causal 3D VAE vs. Frame-wise VAE. In sequences with rapid motion (Frames 27-32), the 3D VAE exhibits significant artifacts and temporal incoherence, while the Frame-wise VAE maintains sharp details and high reconstruction fidelity.

Frame-wise VAE encoding is the highest-contribution component: replacing it with the default Causal 3D VAE causes a 4.87 dB PSNR drop. The 3D VAE compresses adjacent frames into shared temporal latents assuming smooth inter-frame transitions — an assumption violated by varying blur magnitudes across frames. Removing 3DGS-rendered data causes a 2.30 dB drop, as high-frame-rate datasets primarily cover object-motion blur but lack camera ego-motion and defocus blur. One-step distillation not only accelerates inference (3.53s to 0.15s/frame) but also slightly exceeds the Teacher due to pixel-space supervision. TWM enables stable, constant-memory long-video inference.

Figure 6: Qualitative ablation of the Temporal Window Mask on a 150-frame video. Global attention causes severe artifacts due to RoPE extrapolation; a narrow window W=5 avoids extrapolation but limits temporal aggregation; the default W=20 achieves the best balance.

---

Limitations

1. Perception-fidelity trade-off (stated by authors): As a generative method, RealVDeblur has an inherent trade-off between perceptual quality and pixel-level fidelity — under extreme blur it may generate semantically plausible but not entirely accurate details, and PSNR may not exceed deterministic regression methods (e.g., ESTRNN's 31.39 on BSD is higher than this method's 28.76).
2. Computational resource requirements: Fine-tuning Wan2.1-1.3B requires 32×A100 GPUs, and one-step distillation and multi-step Teacher inference still demand substantial GPU memory, posing a barrier for resource-constrained deployment scenarios.
3. One-step distillation boundaries: Although one-step inference slightly exceeds the Teacher in most scenarios, it may be less precise than multi-step inference under extremely complex blur patterns, and the distillation process may lose some generative diversity.

---

Conclusion and Outlook

RealVDeblur achieves strong generalization in real-world video deblurring through large-scale physically grounded blur synthesis, frame-wise video diffusion prior adaptation, and one-step distillation inference. The three key designs each serve a distinct purpose: frame-wise VAE encoding faithfully models frame-level blur variations (highest contribution, 4.87 dB), DMD distillation enables efficient one-step inference without quality loss, and the temporal window mask enables stable, constant-memory long-video inference. Across multiple real-world benchmarks it demonstrates strong perceptual quality, semantic fidelity, and temporal consistency, and as a general-purpose preprocessing module it improves downstream 3DGS reconstruction robustness under severe motion blur.

Golden quote: "Removing temporal compression in the VAE in favor of frame-wise 2D encoding — because motion blur causes large inter-frame differences that violate the smooth-transition assumption of temporal compression — is the core obstacle to generalizing video diffusion models to real-world deblurring."