PAPER DEEP DIVE
Fashion-3DLR: A Controllable 3D Garment Generation Using Pairwise Fashion Elements for Intelligent Design
AI-generated content (AIGC) has made significant progress, with 2D generative models becoming ready-to-use tools for the digital fashion industry. However, 3D garment generation remains in its nascent stage, where in the realm of fashion, the semantic information of diverse design elements exhibits intricate coupling relationships in 3D representations, posing substantial challenges for generating diverse 3D garments. In this work, to handle the above problem, We introduce Fashion-3DLR, a novel 3D garment generation framework that utilizes diverse design elements to create high-quality, versatile 3D garment assets. Specifically, to bridge the semantic gaps between different fashion elements, we propose a Garment Feature Fusion Diffusion Transformer (GFF-DiT) module to integrate 2D fashion design elements, e.g., sketch and texture, into latent space. Within the latent space, we then employ a rectified flow transformer to generate geometry latents, which can be decoded into various 3D garment representations, including 3D Gaussians and meshes. Furthermore, we integrate Fashion-3DLR into downstream tasks, achieving the 3D Gaussian Splatting (3DGS)-driven cloth physical simulation and mesh-based virtual try-on. Experimental results indicate that Fashion-3DLR surpass the previous state-of-the-art methods, which verify that the proposed work can generate well-structured, non-watertight garments capable of physical simulation and virtual try-on, underscoring its potential as a versatile 3D garment design tool.
Fashion-3DLR: A Controllable 3D Garment Generation Using Pairwise Fashion Elements for Intelligent Design
Paper: Fashion-3DLR: A Controllable 3D Garment Generation Using Pairwise Fashion Elements for Intelligent Design
Authors: Shenghao Yang, Hongtao Zhang, Yuhan Yi, Zhihao Tang, Zihao Cui, Lian Wen, Han Yan, Yuan Gao, Mingbo Zhao
Affiliation: Donghua University, Shanghai AI Laboratory
Link: arXiv:2607.23189 | Code: GitHub
One-Sentence Summary
Fashion-3DLR proposes a controllable 3D garment generation framework that fuses heterogeneous design elements (sketch, texture) into a unified latent space via a GFF-DiT module, then generates geometry latents through a rectified flow transformer decodable into 3D Gaussians and meshes, supporting 3DGS-driven cloth simulation and mesh-based virtual try-on.
Background and Motivation
While 2D generative models are mature tools for the digital fashion industry, 3D garment generation remains nascent. The core challenge lies in the intricate coupling relationships between diverse design elements (sketches, textures, oil-brush strokes) in 3D representations. Existing 3D generation methods (Hunyuan3D, Tripo, Trellis) suffer from geometric instability, texture degradation, and topology errors when processing multi-modal fashion inputs — often producing collapsed "pancake" structures or watertight meshes contradicting garment intent.
Furthermore, 3D garment generation must satisfy three key requirements: cross-modal semantic disambiguation (semantic alignment between heterogeneous design elements), high-quality generation (structurally sound non-watertight garment topology), and intuitive editability (designer-controllable attribute modification). This paper builds an end-to-end framework around these three requirements.
Figure 1: Given paired design elements (e.g., sketch and texture), Fashion-3DLR generates high-quality 3D garment assets.
Method Details
1. Overall Pipeline
The Fashion-3DLR pipeline starts from paired design elements (sketch $b$ and texture $o$). The GFF-DiT module first extracts and fuses semantic information into a unified latent space. A rectified flow transformer (RFT) then generates garment latents $\mathcal{S}$ with integrated 3D structural and appearance details. Different upsampling decoders convert $\mathcal{S}$ into 3DGS representations or meshes for flexible downstream applications.
Figure 2: Overall pipeline for Fashion-3DLR, processing paired design elements to produce 3DGS and meshes.
2. Garment Feature Fusion Diffusion Transformer (GFF-DiT)
The GFF module is the core feature extraction and fusion component. Sketch $b$ and texture $o$ are fed into convolutional layers for downsampling to intermediate features $\hat{b}_i$ and $\hat{o}_i$. Texture feature $\hat{o}_i$ passes through a residual block and two different convolutional layers to produce learnable factors $\Lambda_i$ and $\Theta_i$:
$$ \mathcal{G}(o) \rightarrow \{\Lambda_i, \Theta_i\} $$Simultaneously, sketch feature $\hat{b}_i$ computes mean $\hat{b}_i^m = \text{mean}(\hat{b}_i)$ and variance $\hat{b}_i^s = \text{var}(\hat{b}_i)$. To enable interactive communication between sketch and texture, the texture-derived learnable factors modulate the sketch feature:
$$ \mathcal{F}_{\{\Lambda_i, \Theta_i\}}(b) = (\Theta_i \mid \mathcal{G}(o)) \otimes \left[\frac{\hat{b}_i - \hat{b}_i^m}{\hat{b}_i^s}\right] \oplus (\Lambda_i \mid \mathcal{G}(o)) $$where $\oplus$ denotes multiplication and $\otimes$ indicates element-wise addition. A reverse modulation follows, passing the fused sketch feature through $\mathcal{G}$ to generate new factors $\Lambda_{i+1}$ and $\Theta_{i+1}$ that modulate the texture feature:
$$ \mathcal{F}_{\{\Lambda_{i+1}, \Theta_{i+1}\}}(o) = \Theta_{i+1} \otimes \left[\frac{\hat{o}_i - \hat{o}_i^m}{\hat{o}_i^s}\right] \oplus \Lambda_{i+1} $$The final fused feature $x_i$ is obtained through a zero convolution layer $\mathcal{Z}$:
$$ x_i = \mathcal{Z} \cdot \mathcal{F}_{\{\Lambda_{i+1}, \Theta_{i+1}\}}(o) $$This symmetric treatment eliminates disparities within the data. The fused feature set $\mathcal{X}_i = \{x_i, x_{i+1}, x_{i+2}, x_{i+3}\}$ passes through a linear layer to produce latent noise, which is processed by the DiT module with timestep embedding to generate fused conditional features $\mathcal{M}$ serving as the controlling condition for 3D garment generation.
Figure 3: Structure details of the GFF module, showing the bidirectional modulation between sketch and texture.
3. Garment Latents Generation (Rectified Flow Transformer)
The RFT generates garment latents $\mathcal{S}$. The transformer block consists of a self-attention layer, a cross-attention layer, and a feed-forward network. A dense noisy grid is serialized into the noise latent, combined with positional encodings, and fed into transformer blocks for denoising. 3D garment assets are divided into voxel grids, each containing activated voxel markers and visual features.
Specifically, images are rendered by randomly sampling camera perspectives on a sphere, feature maps are extracted using pretrained encoders (DINOv2), and each voxel is projected onto multi-view feature maps to retrieve features from corresponding positions and averaged. Condition $\mathcal{M}$ is injected through cross-attention as keys and values, while the noise latent is injected through self-attention as queries. After $N$ transformer blocks, a garment latent $\mathcal{S}$ is obtained that reflects both the input texture $o$ and sketch $b$ while possessing correct 3D features.
4. Two-Stage Training Optimization
The GFF-DiT module training loss:
$$ \mathcal{L}^{GFF}(\theta) = \mathbb{E}_{z_0, t, o, b, \epsilon} \left[\left\|\epsilon - \epsilon_\theta(z_t, t, \mathrm{f}(o,b))\right\|_2^2\right] $$where $\mathrm{f}$ represents the GFF process and $\epsilon_\theta$ is the DiT network. After training, GFF-DiT weights are frozen. The RFT uses a linear interpolation forward process with data samples $n_0$ and noise $\varepsilon$:
$$ n(t) = (1-t)\,n_0 + t\,\varepsilon, \qquad v(n,t) = \nabla_t n $$The velocity field $v_\theta$ is trained by minimizing the conditional flow matching (CFM) objective:
$$ \mathcal{L}^{CFM}(\theta) = \mathbb{E}_{t, n_0, \varepsilon} \left[\left\|v_\theta(n,t) - (\varepsilon - n_0)\right\|_2^2\right] $$5. Physical Garment 3DGS
A static cloth instance is reconstructed as unstructured 3D Gaussian kernels $\{x_p, \sigma_p, A_p, C_p\}_{p \in \mathcal{P}}$, where $x_p$, $\sigma_p$, $A_p$, $C_p$ denote Gaussian centers, opacities, covariance matrices, and spherical harmonic coefficients. For cotton, silk, wool, and nylon, a corresponding material model is instantiated with reference dynamics generated in a commercial cloth simulator. Each Gaussian is endowed with continuum-mechanics-based kinematics using the MPM solver. Physical parameters $\vartheta = \{k_b, k_s, k_{sh}, \rho\}$ (bending stiffness, stretching stiffness, shear stiffness, area density) govern the Gaussian field evolution:
$$ A_p(t) = F_p(t)\, A_p\, F_p(t)^\top, \qquad x_p(t) = \phi(X_p, t) $$where $F_p(t)$ is the deformation gradient and $\phi(\cdot)$ is the deformation map. Parameters are optimized by minimizing the discrepancy between simulated and reference cloth video:
$$ \mathcal{L}^{\text{gauss}}(\vartheta) = \arg\min_{\vartheta} \left\|F^{\text{video}} - F^{\text{gauss}}(\vartheta)\right\|^2 $$
flowchart TD
A["Input: Sketch b + Texture o"] --> B["GFF-DiT Feature Fusion"]
B --> B1["Texture features → Λ_i, Θ_i"]
B1 --> B2["Bidirectional modulation: Sketch↔Texture"]
B2 --> B3["Zero conv → Fused features X"]
B3 --> C["DiT generates condition M"]
C --> D["Rectified Flow Transformer RFT"]
D --> D1["Voxel grid + Visual features"]
D1 --> D2["Self-attention: noise latent as Q"]
D2 --> D3["Cross-attention: condition M as K,V"]
D3 --> E["Garment latent S"]
E --> F["3DGS decoding"]
E --> G["Mesh decoding"]
F --> H["3DGS physical simulation"]
G --> I["Virtual try-on"]
style B fill:#e1f5fe
style D fill:#fff3e0
style E fill:#e8f5e9
Experimental Results
Setup
Training data: GFF-DiT trained on ~15.2K samples (~30 hours, ~40s inference). RFT fine-tuned on SewFactory (19.1K high-quality 3D garments) and DeepFashion3D. Metrics: FID, CLIP↓ (semantic consistency, lower better), LPIPS↓ (contour similarity, lower better), CLIP-I score↑ (semantic correspondence, higher better).
Quantitative Comparison
| Method | FID CLIP ↓ | LPIPS ↓ | CLIP-I score ↑ |
|---|---|---|---|
| Hunyuan3D-3.0 [2025] | 0.4648 | 0.7001 | 0.7946 |
| Tripo [2024] | 0.4524 | 0.6803 | 0.7961 |
| Trellis [CVPR2025] | 0.7142 | 0.7842 | 0.6615 |
| Sparc3D [NeurIPS2025] | 0.8308 | 0.7825 | 0.6638 |
| ReconViaGen [ICLR2026] | 0.8637 | 0.8113 | 0.6395 |
| Fashion-3DLR (ours) | 0.2743 | 0.5946 | 0.9271 |
Fashion-3DLR leads across all metrics: FID drops from best baseline 0.4524 to 0.2743 (39.4% reduction), CLIP-I score rises from 0.7961 to 0.9271 (16.5% improvement).
Virtual Try-on Comparison
| Method | FID CLIP ↓ | LPIPS ↓ | CLIP-I score ↑ |
|---|---|---|---|
| DressCode [SIGGRAPH2024] | 0.4524 | 0.6803 | 0.8311 |
| ChatGarment [CVPR2025] | 0.4648 | 0.7001 | 0.7946 |
| Fashion-3DLR (ours) | 0.2743 | 0.5946 | 0.9271 |
Ablation Study
| Configuration | FID CLIP ↓ | LPIPS ↓ | CLIP-I score ↑ |
|---|---|---|---|
| Without GFF-DiT | 0.6874 | 0.7429 | 0.6983 |
| Full model | 0.2743 | 0.5946 | 0.9271 |
Removing GFF-DiT increases FID from 0.2743 to 0.6874 (150.6% degradation) and drops CLIP-I score from 0.9271 to 0.6983, demonstrating the critical role of GFF-DiT in multi-modal feature fusion.
User Study
24 participants evaluated 200 sets of generated results. Fashion-3DLR achieved 83.82% user preference, significantly outperforming ChatGarment (73.25%) and DressCode (64.41%).
Figure 4: Qualitative generation results using sketch+texture and sketch+oil-brush inputs.
Figure 5: Qualitative comparison with baselines, showing Fashion-3DLR's structural fidelity and texture consistency.
Limitations
- Cross-representation interaction: 3DGS garments cannot directly interact with mesh-based human body models; cross-representation physical simulation compatibility remains unsolved.
- Missing sewing-pattern structure: Mesh garments lack sewing-pattern-level structural information, limiting integration with industrial pattern-making workflows.
- Training data scale: GFF-DiT trained on only ~15.2K samples; generalization to extreme styles or rare design elements (asymmetric cuts, complex draping) may be limited.
Conclusion and Outlook
Fashion-3DLR achieves unified latent-space fusion of heterogeneous fashion elements through GFF-DiT, combined with a rectified flow transformer and dual decoders (3DGS/mesh) for controllable 3D garment asset generation. For 3DGS physical simulation, the MPM solver enables fabric-dependent dynamic behaviors for cotton, silk, wool, and nylon. Experiments demonstrate comprehensive superiority over existing SOTA methods across all quantitative metrics and user preference.
Key Insight: "By symmetrically fusing sketch and texture semantics in latent space through bidirectional modulation — rather than simple concatenation — Fashion-3DLR achieves semantically consistent translation from 2D design elements to 3D garment assets, providing a unified generation framework with both mesh and Gaussian dual-path decoding for intelligent fashion design."
SOURCE LINKS
