Bringing 3DGS into NVIDIA Isaac Sim: A Photoreal Import Workflow and Four Industrial-Simulation Weaknesses

Pipeline at a glance: 3DGS scan → decimate the point count → convert to USD → place on the stage and fix the orientation → put the camera on the scan trajectory.

If a 3DGS scan is going to be used for robot or autonomous-mobility simulation, the destination is NVIDIA Isaac Sim, not a game engine. This article lays out the complete procedure — from loading the scan to rendering it at photographic quality — in a form that works for any dataset. The second half covers what matters even more in practice: the fundamental weaknesses of 3DGS in industrial simulation, and the workarounds for each.

01 — Why Isaac Sim

Where you take a 3DGS scan depends on what you intend to do with it. For film footage, UE5 gives the most freedom; but if the goal is robots, autonomous mobility, or sensor simulation, NVIDIA Isaac Sim is the front-runner — because a real, scanned place can become the validation environment itself.

• Run robots in places that actually exist. Not a fictional level — the very site you scanned becomes the simulation environment.
• Coexists with physics simulation. PhysX rigid bodies, joints, and sensors all operate inside a photo-based background.
• Mass-produce training images. Generate camera images from any angle against real-world scenery.

Isaac Sim 6.0 renders gaussian splats natively. They participate in RTX path tracing, so depth of field and motion blur work on them as-is. In practice, though, bringing a scan in runs into several walls that the documentation never mentions.

02 — What You Need

• Isaac Sim 6.0 or later — roughly 40 GB once unpacked, so leave 50 GB free.
• Scan data — the 3DGS .ply.
• Conversion tool — PlayCanvas splat-transform (npm, free).
• USD conversion — ships with Isaac Sim, no extra install.

Do not install into C:\Program Files. Isaac Sim writes data while it runs, and placing it somewhere that requires administrator rights invites needless trouble.

One more point about input formats: splat-transform reads scanner-native formats directly, not just .ply. Skipping the intermediate conversion saves effort, and those vendor packages sometimes bundle a collision mesh or the camera trajectory from the scan session — both of which pay off in later steps. If you still have the original format, start from it.

03 — First Wall: Over the Draw Limit, Nothing Renders

If you are going to handle large scans, internalize this before anything else.

Isaac Sim's RTX renderer can draw at most 2²⁴ (16,777,216) points per prim. Beyond that, building the ray-tracing acceleration structure fails, and the prim displays zero points. Neither the converter nor the Python API reports an error. The result is: "conversion succeeded, point count correct — and the screen is completely black."

The nasty part is that every step finishes normally even when it has failed. The USD file contains the coordinates, scale, opacity, and color of every point, all written correctly; placed on the stage, the prim type is recognized correctly too. Only the display never happens.

This limit does not appear anywhere in the official documentation. The single clue is one line in the log: Failed to create TLAS. The spec-correct answers are "reduce the point count" or "split into multiple prims"; this article covers the former.

Check the log before you look at the viewport. After placing the prim on the stage, search the Isaac Sim log for Gaussian prim loaded. If it appears with a point count, the prim is registered as drawable. If you see Failed to create TLAS instead, nothing will display. This check keeps you from misdiagnosing a black screen as a camera problem.

04 — The Import Procedure

Decimate below the limit

Pass the target point count to splat-transform's --decimate option:

# absolute count (must stay under 2^24 = 16,777,216)
splat-transform input.ply -d 16500000 output.ply

# or a percentage
splat-transform input.ply -d 95% output.ply

This is not naive random thinning. The tool iteratively pair-merges nearby gaussians in stages, which loses less information than deleting points at random. For a reduction of a few percent, the visual degradation is essentially imperceptible.

Convert to USD and place on the stage

From Python inside Isaac Sim, run the bundled converter on the PLY. Make the output extension .usdc — .usdz stores the content as text and more than doubles the file size.

Before placing the new scan, clean up any gaussian-splat prims already on the stage. Leftover data produces the symptom "only part of the new scene is visible," which is extremely hard to isolate.

from omni.kit.converter.gsplat import convertPlyUSD
convertPlyUSD(r"output.ply", r"scene.usdc")

import omni.usd
stage = omni.usd.get_context().get_stage()

# remove any existing gaussian-splat prims
for p in list(stage.Traverse()):
    if p.GetTypeName() == "ParticleField3DGaussianSplat":
        stage.RemovePrim(p.GetPath())

# reference via OverridePrim, without specifying a type
prim = stage.OverridePrim("/World/Scene")
prim.GetReferences().AddReference(r"scene.usdc")

Use OverridePrim, not DefinePrim. If you declare a type before adding the reference, the local type declaration takes precedence over the referenced one and the gaussian-splat type is lost — the attributes arrive, but the prim type changes and nothing renders. Create the prim with OverridePrim and no type.

Fix the upside-down orientation

The converter accepts an up-axis argument, but converting Z-up scan data with a Y-up setting flips the scene upside down — buildings grow downward into the ground. If you then place a camera "looking down from the sky" without noticing, you end up viewing the underside of the ground, and the screen is black. Applying a −90° rotation about the X axis to the prim restores the original orientation:

from pxr import UsdGeom

x = UsdGeom.Xformable(prim)
x.ClearXformOpOrder()
x.AddRotateXOp().Set(-90.0)

05 — Placing the Camera

The practical trick is: do not compute the camera position yourself. Deriving a distance from the bounding box does not work, because sparse outliers drag it around — scans scatter sparse points well outside the captured area, so the computed extent is far larger than the region you actually want to see.

Instead, use the coordinates of the trajectory actually walked during the scan. Most scanners export the path as pose data; pick one point on it, put the camera at eye height (roughly 1.7 m), and face it horizontally.

For orientation, use SetLookAt().GetInverse() and assign it with AddTransformOp() — this form is immune to matrix-transpose mistakes:

from pxr import Gf, UsdGeom

eye = Gf.Vec3d(...)    # one trajectory point + eye height
target = Gf.Vec3d(...) # direction to look at

xf = Gf.Matrix4d().SetLookAt(eye, target, Gf.Vec3d(0,0,1)).GetInverse()
cx = UsdGeom.Xformable(cam_prim)
cx.ClearXformOpOrder()
cx.AddTransformOp().Set(xf)

The reason for using the trajectory is simple: the places the scanner walked are exactly where the data was recorded at the highest density. A camera placed outside the captured volume looks at surfaces that were never filmed. Even without pose data, test-shoot from a short distance first and back off gradually while watching what actually appears — that is the reliable route.

Wait for the render to converge. Beyond 10 million points, RTX convergence takes time. Before capturing, spin await app.next_update_async() for at least 150 frames — insufficient waiting saves a pitch-black image.

06 — Weaknesses in Industrial Simulation

This is the crux. For "looks," 3DGS is overwhelming — but it carries almost none of the information industrial simulation demands. The findings below were verified inside Isaac Sim by taking actual outputs.

Weakness 1: The whole scene is one blob — no object-level separation

This is the biggest constraint. A 3DGS is a single prim holding tens of millions of gaussians; there is no such unit as "this building," "this traffic light," or "this car." Unlike a 3D model, you cannot select object A versus object B, and you cannot move, hide, or swap out any specific object.

Simulation work takes operations like "move only the target object" or "swap in a different obstacle" for granted — a raw scan's 3DGS cannot do either.

Weakness 2: Only one semantic label per scene

Isaac Sim can output semantic segmentation — per-pixel images classifying "this is road," "this is building" — which is the core of training-data generation for perception AI. Gaussian splats can indeed carry a label, and it is reflected in the segmentation output.

The catch: labels attach at prim granularity. Since the whole scene is one prim (weakness 1), the scene gets exactly one label. In an actual output, buildings, road, sidewalk, and traffic lights are all painted with the same ID. Use cases like "train on the road area only" or "extract only the signs" are out of reach as-is.

Weakness 3: No physical collision

A gaussian splat is appearance data; it holds no surface information, so it cannot participate in the physics engine's collision detection. Robots pass through walls and fall through floors. Visually there is a floor; physically there is empty space.

Weakness 4: Lighting cannot be changed afterwards

3DGS bakes the lighting at capture time into color data. A scan made under overcast sky stays overcast even if you add a sun light to the scene. When you want training data spanning times of day or weather conditions, this property becomes a constraint.

07 — Working Around the Weaknesses

Every one of these has a workaround, and they all share the same idea: 3DGS handles the looks, meshes handle everything else. This is also the architecture NVIDIA itself presents.

Fix 1: Pair a collision mesh (physics)

Place a simple proxy mesh at the same position as the 3DGS and enable collision on it. The splats provide the image; the mesh provides the contact; the mesh itself stays invisible.

from pxr import UsdPhysics, UsdGeom

mesh = stage.GetPrimAtPath("/World/CollisionMesh")
UsdPhysics.CollisionAPI.Apply(mesh)      # enable collision
UsdGeom.Imageable(mesh).MakeInvisible()  # hide it visually

There are three sources for the mesh. The simplest is a mesh the scanner exported alongside the splat. Without one, splat-transform can generate a collision mesh directly from the 3DGS. Or approximate floors, walls, and obstacles with simple boxes — if the use case is "can the robot get through," boxes are often enough.

For a concrete follow-up that builds exactly this two-layer environment — using the scanner's raw mesh as the collider and patching only its ground holes with measured LiDAR — see the authors' companion article on setting up a 3DGS simulation environment.

Fix 2: Split into regions and label each (semantics)

If labels attach at prim granularity, splitting the prims splits the labels. Cut the splat spatially, export each piece as a separate USD, and label them individually. Do this before importing into Isaac Sim: select regions in SuperSplat (PlayCanvas's free editor) and export them one by one, or cut mechanically by coordinates.

The effort scales with the granularity of the split. A coarse "road / buildings / everything else" classification is realistic; hand-labeling individual objects is not.

The more reliable route: put labels on the mesh side. When fine semantics are needed, overlaying a labeled mesh as a separate layer is the practical option — reuse the collision mesh from Fix 1, split it by part, and label each part. Segmentation output comes from the mesh layer while the image comes from the splat layer. Public datasets already ship this structure, distributing the splat USD and the collision USD separately and composing them at use time.

Fix 3: Accept the lighting, or change tools

Relighting the 3DGS itself is outside Isaac Sim's scope. If you need weather or time-of-day variation, either convert the rendered images afterwards with a generative image model, or scan the same place several times under different conditions. UE5 has relighting-capable plugins, so material produced there is another option.

Choosing the Right Use Cases

With all this in mind, where 3DGS fits — and where it does not — separates quite cleanly:

Use case	3DGS alone	Notes
Photographic-quality background / appearance validation	◎	Its greatest strength — real sites reproduced as they are
Camera-image generation (as background)	○	Any angle, but no per-pixel ground-truth labels
Robot movement / collision validation	×	Pairing with a collision mesh is mandatory
Segmentation training data	×	Requires overlaying a labeled mesh
Object-level manipulation / swapping	×	Impossible without splitting; combining CG assets is faster
Lighting-condition variation	×	Baked in — needs a different route

Summarizing: 3DGS is the technology for bringing real environments in as photographic-quality backgrounds. The objects that move inside them, and everything that needs ground-truth labels, you still prepare as meshes, the traditional way. With that division of roles in place, 3DGS is a very powerful option indeed.

08 — Summary

The import procedure is a single straight path: decimate below 2²⁴ → convert to .usdc → reference via OverridePrim → fix the orientation → put the camera on the trajectory. The stumbles sit at the two ends: over the point limit, nothing shows and no error is raised; a camera outside the scan volume sees nothing but broken surfaces. Master those two and the rest goes smoothly.

For industrial use, treat 3DGS as exactly what it is — a technology for bringing real environments in as photographic-quality backgrounds — nothing more, nothing less. Object-level separation, semantic labels, and collision are all things the splat itself does not have; when you need them, overlay meshes as separate layers and divide the roles. The flip side is that "fidelity of appearance" is something no other technique can replace.

Splitting work between Isaac Sim and UE5 by use case is also realistic; for output as film footage, the authors cover the workflow in their UE5 × XGRIDs SDK article.

Source: LocaHun 3D Tech Blog — NVIDIA Isaac Simに3DGSを持ち込む