image-to-3d-pipeline: A Fair Benchmark and Browser Explorer for Image-to-3D Models

Not another model, but a fair ruler

Single-image-to-3D has a dozen open-source models and no obvious winner. image-to-3d-pipeline answers that not by training one more network, but by building the harness that lets you compare them honestly: take a set of AI-generated renders of a fictional submersible as the only input, reconstruct candidate meshes with several models, score every candidate against a fixed Blender inspection protocol, and ship the winner into a browser-based WebGL explorer you can actually walk around. Built by Dreamers Inc, the code is Apache-2.0 licensed, JavaScript is the primary language, and the repo ships three sample source renders plus one ~1.5 MB finished GLB so you can inspect the output before running any inference at all.

The authors are clear-eyed about what the input actually is: view-conditioned artwork, not camera-calibrated photography. The object-only set is fifteen 1536 x 768 PNGs, and several hidden and underside details disagree between views. A photogrammetry pipeline therefore cannot be expected to recover a faithful closed asset just because it has many inputs, and that premise is what forces every score downstream to be structured visual inspection rather than pixel error.

Four stages: mask, reconstruct, evaluate, explore

Stage	Script	What it does
Preprocess	tools/preprocess_object_masks.py	rembg (u2net) cutouts, long side capped at 1536, alpha channel feathered with a 0.35 Gaussian blur, plus checkerboard previews for human checking
Reconstruct	tools/run_trellis_multiview.py	Multi-image TRELLIS in stochastic or multidiffusion mode, exporting raw-mesh.ply, gaussian.ply, ship.glb and a turntable video comparing appearance against normals
Evaluate	tools/run_reconstruction_eval.sh	Writes mesh-facts.json, runs the fixed Blender inspection orbit, then builds textured and clay contact sheets
Explore	05-web-explorer/	Three.js plus three-mesh-bvh loads the winning GLB into a navigable underwater scene, with no WASM physics runtime and no external service

The reconstruction runner reads like a lab notebook rather than a demo. Before it runs it enforces a hard constraint: the image count may not exceed the sampler step count, because TRELLIS's multi-image path needs at least as many steps as images, and it exits with an error rather than quietly degrading. Afterwards it writes the SHA-256 and byte size of every input image, plus model, mode, seed, steps, simplify ratio, texture size, hostname, PyTorch and CUDA versions, GPU name, the four stage timings (model load, inference, turntable, GLB export) and peak CUDA memory in GiB into run-config.json and run-metrics.json. Defaults are seed 42, 12 steps, simplify 0.90, texture size 1024, sparse-structure cfg_strength 7.5 and SLAT cfg_strength 3.0. Any single run can therefore be reproduced, challenged and aligned against another.

The evaluation protocol: turning "looks good" into axes you can score

The evaluation side is all about holding variables fixed. render_reconstruction_eval.py renders in Blender 4.3 EEVEE with ten views hardcoded in the source: azimuths 0/45/90/135/180/225/270/315 at 10 degrees elevation, plus az045_el35 and az225_elm15 to cover bow, stern and underside. Each candidate mesh is normalised by scaling its longest bound to 4.8 units, shot on a 55mm lens at radius 8.0, lit by one fixed three-area-light rig (key 1150, fill 700, warm rim 800), colour-managed with AgX - Medium High Contrast at 768px. Every view renders twice: once with the original materials, once swapped to a single uniform clay material, because the clay pass is what stops texture from hiding bad geometry. A render-manifest.json records renderer, resolution and the angle of each view.

Scoring uses a 0-2 rubric across five axes: all-angle silhouette, source feature retention, texture continuity, novel-view credibility and asset integrity. Four guardrails sit on top of it: the GLB must import in Blender, a texture node must exist, there must be no catastrophic missing surfaces, and remote GPU OOM or failure must be logged rather than silently retried. The rubric and the guardrails live in 07-experiments/HUNYUAN_2MV_EXPERIMENT_PROTOCOL.md, so every new candidate is measured with the same ruler.

Five candidates, ranked, with their costs

Rank	Candidate	Inputs	Browser asset	Main defect
1	TRELLIS V2 stochastic	15 masked object-only views, 20 steps	1.56 MB / 11,866 faces	Front and stern machinery remain inferred; fine seams are mostly texture, not geometry
2	TRELLIS V2 multidiffusion	Same 15 views, 20 steps	1.61 MB / 12,444 faces	Over-smoothed canopy and underside; some view-specific details disappear
3	TRELLIS V1 multidiffusion	10 rembg-masked scene images, 12 steps	1.58 MB / 11,114 faces	Blurred, flattened stern and a generic upper surface
4	TRELLIS V1 stochastic	Same 10 images, 12 steps	1.85 MB / 17,790 faces	Fuses the vessel into an incorrect manta-like near-symmetry
5	TripoSR single-view	Individual scene images	0.8-2.3 MB	Thin relief shell, blank or hallucinated back; unusable as the ship

The verdict is written with unusual restraint. TRELLIS V2 stochastic wins the identity-versus-detail balance, keeping the long hull, canopy, side turbine recess and fin language, so it becomes the explorer's placeholder asset, but it is "a plausible visual reconstruction, not an engineering-faithful digital twin". The web runtime can accept a better GLB without architectural change. Single-view TripoSR is judged unusable outright, and that is the pipeline's most persuasive counterexample: fast is not the same as shippable.

The explorer: collision and flight in a browser

The explorer is a self-contained Vite project with exactly two runtime dependencies, three 0.179.1 and three-mesh-bvh 0.9.1, on Node 22.12 or newer. Collision involves no physics engine at all: each hull mesh gets a MeshBVH built with maxLeafTris: 10, then a player-radius sphere is corrected out of penetration along the surface normal. Movement is smoothed by exponential interpolation, with a target speed of 6.2 in explore mode and 5.6 in pilot mode, pitch clamped to ±1.42 radians, vertical position clamped between -1.45 and 21, and horizontal radius pulled back in past 58. The wheel drives vertical motion, and the world is an inside-facing 360 underwater panorama plus a repeatable sand microtexture whose lighting brightens with height toward the surface.

URL parameters are the experiment switches: ?mesh=trellis2 for the default candidate, ?mesh=v2-multidiffusion for the smoother V2 alternative, ?mesh=v1-multidiffusion for the original scene-backed baseline, ?mode=pilot for chase-camera piloting of the real asset, and ?projection=starboard for a deliberately isolated one-image projection experiment the authors explicitly label a controlled comparison, since a true upgrade would need calibrated multi-angle source coverage. The repo also carries Playwright tests covering asset loading, keyboard movement, a solid-hull collision probe, console errors, pilot mode and ten viewport sizes; mobile layout is supported, but touch navigation is out of scope for this milestone.

What to try next: model research and protocol

07-experiments/MODEL_RESEARCH_2026-08-07.md takes apart the question "would a newer model be better?" and answers it candidate by candidate, with the core advice being: do not replace the local winner merely because another model advertises higher resolution. Microsoft TRELLIS.2 (4B, 512^3 to 1536^3 output, requiring Linux, an NVIDIA GPU with at least 24 GB and a separate pilot environment) is the best controlled next single-reference comparison, but must not be called multi-image. Tencent Hunyuan3D 2.1 (3.3B Shape plus 2B Paint, with a documented six-view texture stage) is better used as a texture-only test on fixed geometry. The MapAnything / VGGT / MASt3R family of metric multi-view reconstruction is the only category that genuinely treats every view as an observation, but it needs a small pose/depth gate defined first, aborting before meshing if estimated cameras or projected features disagree, rather than spending GPU time on an asset that merely averages conflicting illustrations. NVIDIA 3D Object Reconstruction expects calibrated stereo video or photographic imagery, an assumption the inputs do not meet, so it is not an efficient first test.

The Hunyuan3D 2MV experiment already has a written protocol. It allows exactly four RGBA object-only anchors (port profile, bow-on, stern-on, starboard orthographic) matching Hunyuan's front/left/back/right input contract, and it is designed as four runs: H0 the existing baseline, H1 the end-to-end reconstruction swap, H2 the texture stage only on H0 geometry, and H1R a separately labelled Paint-feasibility run where H1 shape is reduced to 40,000 faces. The authors note explicitly that H1R must not be compared as pure texture-only evidence. Within a named run, model, anchor count, seed, mesh resolution and renderer may not change.

Licensing boundaries and reproduction notes

This repository's code is Apache-2.0, but the four third-party tools that setup.sh fetches each carry their own licence, so they are cloned into a gitignored tools/vendor/ rather than vendored into the repo: you accept their terms at the moment you fetch them.

Tool	License	Pinned commit
TRELLIS (microsoft)	MIT	442aa1e
TripoSR (VAST-AI-Research)	MIT	107cefd
stable-fast-3d (Stability AI)	Stability AI Community License, commercial use restricted	ff21fc4
Hunyuan3D-2 (Tencent Hunyuan)	Tencent Hunyuan community license, territorially restricted	f8db630

Every source image is an AI-generated synthetic render of a fictional vessel; no real vehicle, vessel, person or client is depicted. Large artifacts are excluded wholesale: Python environments, model checkpoints, intermediate PLY/GLB outputs and the explorer's full runtime asset set. 05-web-explorer/README.md explains how to supply assets, and the single GLB in examples/ can be copied straight into public/assets/ to get it running.

git clone https://github.com/dreamers-laboratory/image-to-3d-pipeline && cd image-to-3d-pipeline
./setup.sh                      # clones the four reconstruction tools at pinned commits into tools/vendor/
cd 05-web-explorer
cp ../examples/mesh/submersible-v2-stochastic.glb public/assets/ship-trellis2-starboard.glb
npm install && npm run build && npm run preview   # open http://127.0.0.1:4173/

The real value here is not the 1.5 MB submersible mesh but the fact that "which image-to-3D model is better" stops being an argument and becomes a reproducible record: fixed inputs, fixed renderer, fixed viewpoints, a fixed rubric, hashes and timings on disk, and one boundary statement written into the docs. What you get is a plausible visual reconstruction, not an engineering-faithful digital twin. For anyone producing simulation assets, digital twins or 3D content for robotics, that evaluation skeleton is worth copying far more than any single model's output.