OPEN SOURCE DEEP DIVE
SpatialLM: Turning Point Clouds into Structured Indoor Scenes
SpatialLM turns monocular-video, RGB-D, or LiDAR point clouds into walls, doors, windows, and semantically labeled 3D boxes for robot scene understanding.
SpatialLM treats indoor perception as a language-model problem with a geometric front end. The input is an unstructured point cloud. The output is not a segmentation mask, a depth map, or a free-form description of the room. It is a structured scene written in a constrained code-like syntax: walls, doors, windows, and oriented three-dimensional boxes with semantic labels. That makes the project relevant to robotics because the result can be passed to a mapping stack, a scene graph, a simulator, or a task planner without asking another model to guess what the geometry means.
The repository is more specific than the broad label "3D large language model" suggests. SpatialLM combines a point-cloud encoder, a projector into a language model's hidden space, and a Qwen 2.5 or Llama 3.2 decoder. The point features occupy the same sequence as text embeddings. The model then generates room structure as text that is parsed back into typed geometric objects. This review follows repository commit 8913c44d84a450c53e9340b13317f8cf7144a738.
Why a point cloud is not yet a room model
A raw point cloud records where surfaces were observed. It does not say which points form one wall, where a doorway begins, whether a rectangular volume is a bed or a cabinet, or which direction a sofa faces. Classical perception pipelines can recover parts of this information with separate segmentation, fitting, detection, and fusion stages. SpatialLM instead trains one model to emit a compact structural description directly.
The distinction matters for embodied systems. A navigation stack needs traversable floor and wall boundaries. Manipulation or interaction policies need object categories, center positions, yaw, and extents. A digital twin needs synchronized architectural and object layers. SpatialLM does not provide motion planning, grasping, or robot control, but it can produce a representation that those systems can consume. Its central contribution is the bridge from dense, noisy geometry to a small set of semantic geometric primitives.
| Perception output | What it preserves | What a robot still has to infer |
|---|---|---|
| Raw point cloud | Observed surface geometry and color | Walls, rooms, object identity, orientation, task relevance |
| Semantic point labels | Per-point category predictions | Object instances, metric boxes, room topology |
| SpatialLM layout | Named architectural elements and oriented object boxes | Free space, dynamics, affordances, manipulation plans |
| Task planner | Actions and goals grounded in a scene model | Execution feedback and physical control |
The input path begins before the model
SpatialLM accepts point clouds produced from monocular video reconstruction, RGB-D capture, or LiDAR. The repository ships inference examples but does not turn a camera stream into a metric point cloud by itself. For a custom RGB video, the documented workflow uses a SLAM system such as SLAM3R or MASt3R-SLAM to reconstruct geometry, then aligns the scene and corrects its scale before inference.
The alignment requirement is strict in the current release. The point cloud must be z-up and should follow the ScanNet convention in which walls are broadly aligned with the x-y planes. A cloud that has the correct shape but the wrong global rotation will present doorways and furniture in misleading directions. The example documentation recommends Manhattan-frame estimation or manual tools such as Blender when automatic alignment is unreliable.
Scale is equally important. SpatialLM was trained with the convention that one coordinate unit represents one meter, although training-time augmentation prevents the model from depending on exactly one scale. A monocular reconstruction can be internally consistent while remaining arbitrarily scaled. The repository suggests estimating room height from walls and rescaling accordingly, or comparing reconstructed keyframes with metric depth estimates. Both operations sit outside the model and remain engineering responsibilities.
Once a PLY file reaches inference, the model performs its own normalization. It shifts coordinates to positive values, normalizes colors, applies optional voxel-size cleanup, and converts the cloud into a grid-sampled sequence. The default inference path allows statistical outlier removal and voxel downsampling before the final feature construction. This reduces noisy reconstruction output to a bounded set of encoded points, but it cannot repair a systematically wrong pose or scale.
How geometry enters a language model
The prompt contains a point-start token, a point placeholder, and a point-end token followed by the task instruction and a reference output template. Internally the model embeds the text, runs the point cloud through a 3D encoder, projects the resulting features into the decoder's hidden dimension, removes the placeholder, and inserts the point tokens between the start and end markers. The language model then attends to geometric features and text tokens in one causal sequence.
SpatialLM 1.0 uses a SceneScript encoder. SpatialLM 1.1 moves to the Sonata Point Transformer V3 encoder, doubles the point-cloud resolution, and offers 0.5B Qwen and 1B Llama variants. The repository exposes both encoder families through the same causal language model interface. During training, labels for the inserted point feature positions are masked out, so the loss is applied to the generated structural text rather than asking the encoder to reproduce point coordinates.
| Stage | Implementation in the repository | Effect |
|---|---|---|
| Geometry encoder | SceneScript for 1.0, Sonata Point Transformer V3 for 1.1 | Turns spatial neighborhoods into learned point features |
| Projector | Linear layer or a two-layer MLP into the LM width | Makes geometric features compatible with text embeddings |
| Sequence assembly | Point features inserted between exactly one start and end token | Lets one autoregressive decoder attend to geometry and instructions |
| Language decoder | Qwen 2.5 0.5B or Llama 3.2 1B backbone | Generates the structured scene in the requested task mode |
| Parser | spatiallm.Layout converts generated text into typed objects | Restores normalized coordinates to metric scene values |
The phrase "multimodal" is therefore literal at the token level. The model does not convert the point cloud to an image and then ask a vision-language model to describe it. Point embeddings are injected into the language model sequence. SpatialLM also replaces the model itself rather than only preprocessing data for a general-purpose LLM, which is why it belongs in an LLM capability view as well as a 3D perception view.
The output is a small scene program
The generated format defines walls with endpoints, height, and thickness. Doors and windows refer back to a wall identifier and carry a position, width, and height. Object boxes carry a class, center position, yaw angle, and three axis extents. Coordinates are quantized during generation and then undiscretized using the model configuration before the layout is returned.
Wall(ax, ay, az, bx, by, bz, height, thickness)
Door(wall_id, position_x, position_y, position_z, width, height)
Window(wall_id, position_x, position_y, position_z, width, height)
Bbox(class, position_x, position_y, position_z,
angle_z, scale_x, scale_y, scale_z)
This grammar is intentionally narrow. It captures rooms and furniture with compact semantics, but it does not encode non-planar architecture, articulated objects, material properties, uncertainty, or temporal change. Wall-referenced openings are convenient for editing and floor-plan reconstruction. Oriented boxes are convenient for rigid indoor objects. Neither representation is a substitute for a full mesh or a physically detailed simulator scene.
The three inference modes expose different slices of the same output space. all predicts architecture and object boxes, arch predicts only walls, doors, and windows, and object predicts only boxes. For box-producing modes, SpatialLM 1.1 can be conditioned on a user-provided subset of 59 furniture categories. The task prompt then changes from a generic request for boxes to a request for the selected categories, allowing the same weights to serve narrower perception requests.
| Mode | Generated content | Typical downstream use |
|---|---|---|
all | Walls, doors, windows, and object boxes | Full structured room reconstruction and digital twins |
arch | Walls, doors, and windows | Floor-plan recovery, navigation boundaries, architectural editing |
object | Oriented boxes, optionally filtered by requested categories | Scene inventories and category-specific robot perception |
Data and training are sized for a specialist team
The released SpatialLM dataset contains 12,328 synthetic indoor scenes and 54,778 rooms, each paired with 3D annotations. The benchmark test set contains 107 point clouds reconstructed from monocular RGB video with MASt3R-SLAM. That test distribution is deliberately harder than clean RGB-D scans because it includes reconstruction noise, holes, and occlusions produced by the capture pipeline.
The fine-tuning example adapts the Qwen 0.5B model to ARKitScenes with a new category set and randomized gravity-aligned scene orientations. Data is converted into a ShareGPT-style supervised fine-tuning format. The training code is a deliberately reduced LLaMA-Factory subset, not a general experiment platform. The repository estimates approximately 60 GB of CUDA memory for full fine-tuning and states that LoRA and quantized training are not yet implemented.
That makes fine-tuning a serious infrastructure requirement rather than a quick notebook step. Teams with smaller GPUs can still run inference and prepare data, but adapting the model to a new sensor geometry or ontology should be budgeted as a dedicated training project. The ARKitScenes example also shows that custom categories are learned through new annotations, not created reliably by editing the inference category list alone. The built-in category vocabulary is part of what the released weights understand.
Benchmarks show a strong specialist model, not uniform perception
On Structured3D layout estimation, the fine-tuned SpatialLM 1.1 Qwen 0.5B model reaches 94.3 F1 at 0.25 IoU and 93.5 at 0.5 IoU. It improves over the reported SceneScript fine-tuned result and RoomFormer checkpoint on this benchmark. The benchmark evaluates walls, doors, and windows after training on the project's synthetic data and further fine-tuning on Structured3D.
| Structured3D layout estimation | RoomFormer | SceneScript, fine-tuned | SpatialLM 1.1 Qwen 0.5B, fine-tuned |
|---|---|---|---|
| F1 at 0.25 IoU | 83.4 | 90.4 | 94.3 |
| F1 at 0.5 IoU | 81.4 | 89.2 | 93.5 |
On ScanNet 3D object detection with 18 categories, SpatialLM reaches 65.6 F1 at 0.25 IoU. That is slightly above the reported V-DETR result of 65.1 at the loose threshold, but below V-DETR at 0.5 IoU, where SpatialLM scores 52.6 and V-DETR scores 56.8. The honest conclusion is not that one architecture dominates everywhere. SpatialLM is competitive at coarse localization while the dedicated detector retains an advantage at tighter boxes.
| ScanNet 3D detection, 18 classes | V-DETR | SceneScript, fine-tuned | SpatialLM 1.1 Qwen 0.5B, fine-tuned |
|---|---|---|---|
| F1 at 0.25 IoU | 65.1 | 49.1 | 65.6 |
| F1 at 0.5 IoU | 56.8 | 36.8 | 52.6 |
Zero-shot video results vary sharply by category
The repository's most operationally relevant evaluation is the 107-cloud video test set, because it measures behavior on noisy reconstruction rather than laboratory-clean scans. The aggregate layout results are moderate: at 0.25 IoU, the Llama 1B model reports 68.9 for walls, 49.1 for doors, and 47.0 for windows; the Qwen 0.5B model reports 68.2, 47.4, and 51.4 respectively.
Object performance is highly uneven. The Qwen model reaches 95.2 F1 for beds and 69.1 for sofas, but only 32.3 for chairs, 11.2 for cabinets, and 0.0 for refrigerators in the Llama result. These categories differ in shape regularity, occlusion frequency, training coverage, and box definition. A robotics team should not transfer a single headline score to its own object set. It should measure recall and box tightness for the categories that affect the robot's task.
| Zero-shot category | SpatialLM 1.1 Llama 1B | SpatialLM 1.1 Qwen 0.5B |
|---|---|---|
| Bed | 96.8 | 95.2 |
| Sofa | 66.9 | 69.1 |
| Nightstand | 62.8 | 67.0 |
| Coffee table | 56.4 | 64.9 |
| Chair | 20.8 | 32.3 |
| Cabinet | 15.2 | 11.2 |
| Side table | 14.6 | 9.7 |
| Refrigerator | 0.0 | 16.7 |
What it adds to a robotics stack, and what it does not
SpatialLM is most useful as a structured perception layer. A mobile robot can use its architectural output as an initial room hypothesis, then refine free space with live range measurements. A manipulation system can use the object boxes to propose regions of interest, then run segmentation and pose estimation on denser local observations. A simulator or digital-twin pipeline can convert the typed layout into editable primitives faster than it can reconstruct those primitives from a mesh.
Several limits follow directly from the representation and the repository's own documentation. It handles indoor point clouds, not outdoor terrain. It expects metric scale and a meaningful z-up orientation. It can miss objects in cluttered or non-standard layouts, especially when scenes differ from the Chinese residential apartment distribution emphasized in training. It returns static boxes, so movable, articulated, transparent, or deformable objects need additional sensing. It does not estimate uncertainty per element, and it does not expose a planner or controller.
Latency is also a system-level property. The repository provides inference code, not a tuned real-time serving stack. Encoder choice, point resolution, voxel cleanup, model size, dtype, prompt length, and generated token count all affect runtime and memory. A production deployment would need to benchmark those choices on the target GPU and decide whether periodic room-scale inference is sufficient or whether a faster local update path is required between full reconstructions.
Licensing requires reading the model family, not only the repository badge
The repository is not one uniform license boundary. The Qwen variant derives from Qwen 2.5, the Llama variant from Llama 3.2, and the two language backbones carry different terms. SpatialLM 1.0's SceneScript encoder and SpatialLM 1.1's Sonata encoder or model weights are identified as CC BY-NC 4.0. Pointcept-derived code is Apache 2.0, and TorchSparse is MIT.
| Component | License or source noted by the repository | Practical consequence |
|---|---|---|
| Qwen 2.5 backbone | Apache 2.0 origin | Permissions follow the Qwen license and applicable repository terms |
| Llama 3.2 backbone | Llama 3.2 license | Review use restrictions and distribution obligations separately |
| SceneScript encoder | CC BY-NC 4.0 | The non-commercial constraint is material for product planning |
| Sonata encoder and SpatialLM 1.1 weights | CC BY-NC 4.0 | Do not assume the code license covers commercial model use |
| Pointcept code | Apache 2.0 | Code licensing does not erase restrictions on model weights |
| TorchSparse | MIT | Separate dependency with its own notice requirements |
The strongest reason to study SpatialLM is the integration boundary it makes concrete. Dense reconstruction is no longer only an input to a hand-built perception pipeline. A point encoder can feed geometric tokens into a generative model that emits typed architecture and objects. The method is still a research system with strict input assumptions, uneven category performance, and a non-commercial model-weight license. Within those limits, it provides a practical reference for turning spatial observations into a representation that software and robots can act on.