Real2Sim2Real for Vision-Language-Action Manipulation: An AMD ROCm-Based Pipeline

1. Overview

Physical AI—the integration of large vision-language-action (VLA) models with embodied agents—has emerged as the next major AI frontier. This paper presents an end-to-end, fully AMD-accelerated technology stack for embodied manipulation, spanning data-center training silicon, Radeon PRO simulation/rendering GPUs, and Ryzen AI edge compute, unified by the open ROCm software stack. It demonstrates that training and deploying VLA-based manipulation policies does not require a CUDA-locked ecosystem.

Four progressive demonstrations: (1) a Sim-to-Real manipulation pipeline trained with SmolVLA and deployed on a physical Franka arm; (2) a semantic, language-grounded object-selection task ("one-of-three"); (3) a Real2Sim synthetic-data pipeline fusing 3D Gaussian Splatting (3DGS) reconstructions with the Genesis physics engine; (4) large-scale RL for quadruped and humanoid locomotion across multiple hardware platforms. All pipelines run natively on ROCm + PyTorch (RDNA4 R9700 and RDNA3.5 W7900), reproducible on the free Radeon Cloud Platform.

2. Four Demonstrations

2.1 Demo 1: Sim-to-Real Manipulation

Synthetic manipulation data generated in Genesis physics engine, training SmolVLA policy, deployed on physical Franka arm. Establishes the basic sim-to-real loop: synthetic data → policy training → real robot deployment.

2.2 Demo 2: Semantic Object Selection

Given a natural-language instruction, the policy selects the correct object in cluttered scenes ("one-of-three"). Shows the pipeline supports not just motor competence but instruction-following behavior.

2.3 Demo 3: Real2Sim Synthetic Data

Real environments and camera poses reconstructed via 3DGS, re-injected into physics simulation to generate photorealistic, physically-interactive training data. Turns real-world captures into an inexhaustible source of simulation-grade supervision, forming the Real2Sim2Real closed loop.

2.4 Demo 4: RL Locomotion

Large-scale RL training of quadruped (Unitree Go2) and humanoid locomotion across multiple hardware platforms. Demonstrates the open compute stack's scalability from data-center accelerators to edge-class silicon.

3. Overall Architecture

flowchart LR
    SIM[Genesis Physics Sim
Synthetic Data] --> TRAIN[VLA Policy Training
SmolVLA]
    TRAIN --> DEPLOY[Physical Franka Arm]
    DEPLOY --> CAP[Real Environment Capture]
    CAP --> 3DGS[3D Gaussian Splatting Reconstruction]
    3DGS --> REINJECT[Re-inject into Simulation]
    REINJECT --> SIM
    TRAIN --> RL[Large-Scale RL
Quadruped + Humanoid]
    RL --> EDGE[Ryzen AI Edge Deployment]
    style 3DGS fill:#4f9eff,color:#fff
    style TRAIN fill:#ff6b6b,color:#fff
    style DEPLOY fill:#51cf66,color:#fff

4. Technology Stack

5.

Task loss

$$ \mathcal{L}_{\mathrm{task}}=\mathbb{E}\left[\|y-\hat{y}\|^{2}\right] $$

Consistency loss

$$ \mathcal{L}_{\mathrm{consist}}=\|f(x)-f(g(f(x)))\|^{2} $$

Optimization objective

$$ \theta^{*}=\arg\min_{\theta}\sum_{i}\mathcal{L}(f_{\theta}(x_{i}),y_{i}) $$

Soft action selection

$$ p(a|s)=\frac{\exp(Q(s,a)/\tau)}{\sum_{a^{\prime}}\exp(Q(s,a^{\prime})/\tau)} $$

Q-learning target

$$ \hat{Q}(s,a)=r+\gamma\max_{a^{\prime}}Q(s^{\prime},a^{\prime}) $$

Key Hyperparameters

Parameter	Value	Description
Learning rate	1e-4	Adam optimizer
Batch size	256	Training batch
Discount $\gamma$	0.99	Reward discount
Epochs	5000	Training iterations

Conclusion

Four progressive demonstrations form four stages of one continuous pipeline: simulate, manipulate, reconstruct, generalize. Every stage—3D reconstruction, neural rendering, physics simulation, VLA policy training, reinforcement learning, real-robot deployment—runs on a single, open, consistent ROCm + PyTorch stack across AMD hardware tiers, with no hand-off to a CUDA-locked toolchain at any point. This end-to-end continuity is precisely what Physical AI research needs: closing the sim-to-real gap is fundamentally iterative, and an open, hardware-diverse, fully interoperable compute stack is foundational.