GS-Agent: Creating 4D Physical Worlds With Generative Simulation

1. Paper Overview

This paper from UMass Amherst and Genesis AI presents GS-Agent — an end-to-end multi-agent framework that integrates physics engines in the loop to generate realistic, dynamic, and controllable 4D physical worlds from natural language descriptions. Unlike methods that learn to generate 4D worlds from large-scale data, GS-Agent takes a different path: leveraging foundation model reasoning capabilities to build agentic systems that emulate human 4D world-building workflows, while automating the entire process.

Inspired by human workflows, the framework decomposes the task into entity management (3D asset curation, material tuning, placement, motion control) and rendering configuration (camera and lighting manipulation). Multiple specialized agents interact with the physics engine via code, seek multimodal feedback, and collaboratively construct 4D worlds.

2. Problem Background and Motivation

Creating dynamic, physically plausible 4D worlds has far-reaching applications across embodied AI, autonomous driving, gaming, and film production. But traditional approaches require tremendous human effort — from curating 3D assets and tuning material parameters to constructing scenes, orchestrating object motions, setting up lighting, and designing camera trajectories.

Foundation models have significantly advanced in language understanding, image synthesis, video generation, and 3D/4D scene creation, but still struggle with physical plausibility and controllability:

• Text-to-video models follow textual instructions superficially while violating fundamental physical principles (e.g., sponge fracture produces unrealistic sudden breaking).
• Purely pixel-driven generation produces visually appealing but temporally inconsistent frames that don't adhere to physical laws.
• Lack of true 3D scene reasoning (e.g., background remains identical after 180° camera rotation).

This suggests that learning 4D worlds solely from large-scale data is insufficient to produce consistent, physically grounded scenes. A complementary direction leverages LLM reasoning capabilities to build agentic systems that plan, critique, and iteratively refine world generation.

3. GS-Agent Framework in Detail

3.1 Overall Architecture

GS-Agent contains three specialized agents that collaboratively construct physically plausible 4D worlds:

graph TB
    A["Natural language"] --> B["Manager Agent
Decompose+delegate+verify"]
    B --> C["Entity Agent
Entity creation+material+motion"]
    B --> D["Render Agent
Camera+lighting+render"]
    C --> E["Physics Engine Genesis
Simulation execution"]
    D --> E
    E --> F["Multimodal feedback
Images/video/physics state"]
    F --> B
    F --> C
    F --> D
    B --> G["4D World
Simulation code+rendered video"]

3.2 Physics Engine

GS-Agent uses Genesis as the underlying physics engine. Core components include:

• Entity: Defines objects in the scene, each as $E = (\text{Morph}, \text{Material}, \text{Surface})$, where Morph specifies geometry, Material determines physical behavior (rigid/deformable/liquid), and Surface controls visual attributes.
• Solver: Selected and configured based on entity materials, e.g., rigid-body solvers, Material Point Method (MPM) for deformable objects, Smoothed Particle Hydrodynamics (SPH) for fluids.
• Renderer: Displays physics simulation results, including camera, lighting, and shadow parameters.

Entity motion follows Newton's second law:

$$\mathbf{F} = m \mathbf{a} = m \frac{d^2 \mathbf{x}}{dt^2}$$

For deformable objects using the MPM solver, deformation obeys the constitutive relation:

$$\boldsymbol{\sigma} = \mathbf{C} : \boldsymbol{\epsilon}$$

where $\boldsymbol{\sigma}$ is the Cauchy stress tensor, $\mathbf{C}$ is the elasticity tensor, and $\boldsymbol{\epsilon}$ is the strain tensor. The Entity Agent tunes material parameters including Young's modulus $E$, Poisson's ratio $\nu$, and yield stress $\sigma_y$.

3.3 Manager Agent

The Manager Agent handles user communication, step-by-step planning, task delegation, verification, and global simulation configuration. Given a natural language description, it interprets user intent, reasons about required scene components, and formulates an actionable multi-step plan.

The task delegation strategy is based on agent capability matching:

$$\text{assign}(t) = \arg\max_{a \in \{E, R\}} \text{cap}(a, t)$$

where $t$ is the sub-task, $a$ is the candidate agent (Entity $E$ or Render $R$), and $\text{cap}(a, t)$ is the capability score of agent $a$ for task $t$. After each sub-task, the Manager Agent verifies results and triggers iterative refinement until specifications are met.

3.4 Entity Agent

The Entity Agent specializes in entity creation, morphology generation, material assignment, and motion control. Key capabilities:

• Asset curation and morphology generation: Selects appropriate geometry or meshes based on descriptions.
• Physically plausible material tuning: Initializes with common-sense priors, then iteratively refines based on physics engine feedback, tuning Young's modulus, Poisson's ratio, and yield stress.
• Entity motion: Edits per-step control functions, supporting PD controllers, direct position/velocity/force commands, and programmable emitters.

The PD controller's control force is computed as:

$$\mathbf{F}_{\text{ctrl}} = K_p (\mathbf{x}_{\text{target}} - \mathbf{x}) + K_d (\mathbf{v}_{\text{target}} - \mathbf{v})$$

where $K_p$ and $K_d$ are proportional and derivative gains, and $\mathbf{x}_{\text{target}}$ and $\mathbf{v}_{\text{target}}$ are target position and velocity.

3.5 Render Agent

The Render Agent manages camera and lighting manipulation and provides visual feedback to other agents. It represents camera motion as explicit control code executed during simulation, specifying camera trajectories (e.g., orbits, dolly movements, or entity-tracking motions) and frame-capture schedules.

The camera projection transforms 3D world coordinates $\mathbf{P}_w$ to 2D image coordinates $\mathbf{p}$:

$$\mathbf{p} = \pi(\mathbf{K}, \mathbf{R}, \mathbf{t}, \mathbf{P}_w) = \mathbf{K} [\mathbf{R} | \mathbf{t}] \mathbf{P}_w$$

where $\mathbf{K}$ is the camera intrinsic matrix and $[\mathbf{R} | \mathbf{t}]$ is the extrinsic. Because trajectories are coded rather than baked into pixels, they can be refined (e.g., smoothed or reparameterized) without changing the underlying physical simulation.

4. Experimental Results

4.1 Main Quantitative Results

Key findings:

• GS-Agent significantly leads in physical plausibility (State-PIS 0.83 vs next-best 0.57) and instruction alignment (32.2 vs 30.6).
• State-PIS is a metric unique to GS-Agent — directly extracting exact 3D center-of-mass kinematics from the physics engine, inaccessible to standard text-to-video models.
• Wan2.2 is best on pure aesthetics (58.1) but only 0.46 on physical plausibility, showing pixel beauty ≠ physical reality.

4.2 User Study

With 15 participants and 270 valid responses on a 5-point Likert scale, GS-Agent was most preferred for physical plausibility (4.33), camera controllability (4.32), and content alignment (4.70), slightly trailing only Sora2 on pure aesthetics.

4.3 Emergent Capabilities

• Autonomous error detection and recovery: When an imported 3D bathtub asset was not perfectly waterproof causing leakage, GS-Agent recognized the physics failure, proposed a programmatic fix by applying rigid material patches to the geometry's corners, and automatically verified the corrected output.
• Fine-grained controllability: Users can manipulate via natural language — adjusting rendering effects (e.g., slow motion), spawning new entities (e.g., adding billiard balls), or precisely tuning physical parameters like velocity.

5. Limitations and Shortcomings

• Slightly lower aesthetics: Trails Sora2 (47.6 vs 48.5) and Wan2.2 (58.1) on pure aesthetics, because physics engine rendering quality doesn't match specialized generative models.
• Dependence on external 3D assets: Entity creation relies on pre-curated 3D asset libraries, unable to generate arbitrary geometries from scratch.
• LLM backbone dependence: Performance is limited by the underlying LLM's reasoning capability; using weaker models (e.g., Qwen3.5-27B) drops State-PIS from 0.83 to 0.62.
• Simulation speed limitations: Complex scenes (multi-body collisions, fluid interactions) may simulate slowly, impacting interactive experience.
• Evaluation metric limitations: State-PIS, though more precise, only applies to invariant quantities extractable from the physics engine, not all scenario types.

6. Summary

GS-Agent's core contribution lies in demonstrating that a multi-agent framework with physics engines in the loop can generate physically plausible, controllable 4D worlds from natural language — bypassing the physical plausibility and controllability bottleneck of purely data-driven methods. Three specialized agents (Manager, Entity, Render) emulate human 4D world-building workflows, interacting with the physics engine via code and iteratively refining.

The experimental results are convincing: State-PIS 0.83 far exceeds all baselines (next-best 0.57), and user study shows highest preference for physical plausibility (4.33) and camera controllability (4.32). The emergent autonomous error recovery capability demonstrates the flexibility of the agentic design. Slightly trailing on aesthetics is a reasonable trade-off — physical realism and controllability are more central needs for 4D world generation.