img2threejs: Rebuilding Reference Images as Code-Only Procedural Three.js Models

What it is

img2threejs is an agent skill that runs inside coding agents such as Claude Code, Codex and OpenCode. Give it one reference image of an object or character and it rebuilds the subject as a code-only, procedural Three.js model: a TypeScript factory that assembles a THREE.Group from primitives, procedural shaders and generated geometry, plus an ObjectSculptSpec JSON recording the component tree, materials, sockets and the review history of every pass. It explicitly refuses three routes: photogrammetry, mesh extraction, and downloaded art packs. The artifact is therefore diffable, reviewable line by line, and version-controllable.

The project shipped v2.0.0 in September 2026 under Apache-2.0 and has passed fifteen thousand GitHub stars. Every model in the live gallery at img2threejs.io is generated code running in the browser: no mesh files, no downloads.

The eight-pass sculpting pipeline

Reconstruction is not one-shot. It advances through a fixed order: blockout, structural-pass, form-refinement, material-pass, surface-pass, lighting-pass, interaction-pass, optimization-pass. Each pass has its own acceptance criteria and unlocks only after the previous pass produced a real render, a comparison sheet and a passing vision score. The code generator emits only the currently unlocked pass, so the model never re-reads or rewrites the whole model on every iteration.

Two design choices came from real failures. Geometric gates run before the browser renders: bald patches and self-intersections are found on point sets, so no pass budget is spent rendering geometry that was already wrong. And routing can refuse: when confidence that the subject is a weapon versus a character falls below 0.82, the pipeline resolves to request-input and asks for more information instead of guessing.

Gates that make "looks close" insufficient

The strict-quality gate blocks code generation until the spec is deep enough for the object's complexity: a compound object cannot ship a single-root component tree. The chirality gate requires every left/right pair to be a sagittal mirror rather than a rotated copy, because rotation preserves handedness and negating two axes yields the same hand twice. For hair subjects, scalp exposure is a hard gate evaluated on geometry, since a bald patch is an interior defect that outline metrics cannot see.

Review requires a real render, one packaged side-by-side sheet, a passing vision score, and banded interior difference. The interior metric exists because of a measured failure: silhouette IoU reads roughly eleven percent of a figure's cells and scored a face-deleted version identically to the finished one. Material and lighting gates demand independent PBR channels and real lights, never albedo aliased into roughness.

Division of labor: scripts enforce, the model judges

The forge directory holds around ninety Python modules, all pure standard library: PNG read and write are hand-rolled with struct and zlib, with no PIL, numpy or Playwright. Scripts handle validation, gating, spec authoring, PBR evidence extraction and comparison-sheet packaging, and they never score visuals. Model tokens go to exactly one job: looking at a single reference-versus-render sheet and deciding pass or fail. The factory generator is fail-closed: on a failed gate it returns BLOCKED with the spec artifact, failure metrics, causes and the next action, and writes no code.

Material and character subsystems

Material identity is an executable sub-pipeline: named component region, verified crop, resolution against a versioned Three.js material registry, reference-derived PBR evidence with bounded priors, wiring into ObjectSculptSpec, multi-angle and microscope captures after generation, a per-region comparator with bounded feedback, and finally a blocking material gate. The registry never decides from colour alone; ambiguous regions stay in probe or request-input.

Characters route through an anatomy-aware track: head-unit proportions, facial landmarks, pose. The skeleton is derived from the component tree so bones cannot drift from geometry; skinning weights come from geodesic distance measured through the solid, keeping rigid roles out of smooth skinning. Hair is a five-stage subsystem with a hard scalp-exposure gate. Likeness maximization and multi-view silhouette carving are opt-in: the former fits a parametric template to landmarks, de-lights and camera-matches the reference before projecting it; the latter intersects at least two orthographic binary silhouettes into a welded voxel mesh and records unseen areas as low confidence instead of inventing hidden detail.

The v2.0 plugin ecosystem

Domain knowledge left the main repository in v2.0 and became installable plugins managed by the img2 harness (install, add, doctor). plugin-cs2 supplies CS2 weapon-skin reconstruction with family adapters and review gates; plugin-character carries the full character workflow including Stage R rigging and animation gates; plugin-img2glb adds an image-to-GLB emission target through a hosted TRELLIS space; plugin-hello-cube is the minimal template for writing your own. Plugin quality floors merge raise-only, and a profile whose plugin is missing fails loudly, naming what is absent, rather than silently downgrading.

Honest limits

A single image cannot reveal hidden sides or guarantee exact geometry, and the docs say so plainly: unseen faces are inferred by mirroring visible ones and flagged as approximate; characters are stylized reconstructions, not photoreal likeness; "this cannot reach the requested fidelity from this image" is a valid, expected result. The GLB-reference route treats a GLB purely as a measurement instrument, extracting sizes, proportions, band widths, base colour, roughness and metalness, while the shipped artifact remains procedural TypeScript and no .glb or .bin is fetched at runtime.

Why it matters

Mainstream image-to-3D ships meshes: heavy files that are hard to edit and hard to version. img2threejs ships code: the same reference image yields a reviewable, diffable TypeScript factory with a runtime hierarchy exposing pivots, sockets and colliders, ready to bind a skeleton and animate. For game props, web visualization and digital content pipelines that is a different asset form. The roadmap moves one theme per release: the character split in v2.1, environments in v2.2, engine exporters in v2.3, auto rigging in v2.4, and playable worlds generated from reference images in v3.0.