LightNav-0: Eliciting VLM Spatial Intelligence for Generalist Embodied Navigation

One model, four robots: a generalist navigation brain

LightNav-0 is a compact generalist embodied-navigation model from the Light Origins team. The core idea is to elicit the spatial intelligence of a pretrained vision-language model: it is instantiated from Qwen3-VL-4B-Instruct and adds no navigation-specific modules — no waypoint predictor, no task-specific action head, no per-embodiment experts. Only the vocabulary is extended (indexed pointing tokens + RVQ action tokens), so both the spatial reasoning trace and the action codes are decoded through the backbone's original autoregressive LM head.

Instruction following, open-vocabulary object navigation and embodied visual tracking share one token interface, and a single checkpoint transfers zero-shot to humanoid, quadruped, wheeled and aerial robots in unseen scenes — fully autonomous, no teleoperation.

Method: dual-channel pointing + RVQ action tokenizer

At each decision step the model consumes a timestamped egocentric RGB history and a natural-language instruction, interleaved in a single causal sequence, and emits:

• Dual-channel pointing — an affordance point (a feasible local direction or free-space waypoint) and an object point (the task goal), each one image-grid token. This is an explicit spatial reasoning trace that grounds the plan in pixels before any action is generated.
• Three RVQ action tokens, which decode to 10 future SE(2) waypoints — a common geometric interface handed to each embodiment's own low-level controller.

Task semantics come entirely from the instruction; there is no task-identification token, and the same backbone, token interface and objective serve every navigation task.

Temporally aware history compression: navigation needs both recent geometric detail and long-horizon context. LightNav-0 compresses history by recency, following the shape of the Ebbinghaus forgetting curve: the sampling rate decays exponentially with frame age while the spatial pooling stride grows exponentially, so distant observations contribute fewer and coarser tokens while the current observation keeps the finest detail. Timestamp tokens preserve ordering after pooling. The compressor runs after the vision transformer under configurable pixel budgets (256K/576K/1M), bounding context length without collapsing history into one fixed-resolution summary.

RVQ action tokenizer: a 10-step SE(2) trajectory is quantized by a coarse 256-entry codebook plus two residual 256-entry codebooks, resolving roughly 0.9 m, 7 cm and 4 cm. Any non-empty token prefix decodes into an executable coarse trajectory, and each residual level refines geometric precision — the same three tokens express both gross motion and centimetre-scale path shape.

Benchmarks: one checkpoint leads everywhere

One shared checkpoint, no per-benchmark fine-tuning; every number comes from a single forward RGB stream — no depth, odometry or panoramic rig. Highlights:

• Instruction following (VLN-CE): R2R val-unseen SR 68.5% / SPL 62.8%, RxR SR 73.6% / SPL 64.5% — ahead of NaVILA, StreamVLN, DualVLN, CorrectNav and Qwen-RobotNav-8B.
• ObjectNav (six settings): MP3D 53.3%, HM3D v2 79.5%, OVON unseen classes 47.0% — first on all six.
• Embodied visual tracking (EVT-Bench): SR 82.6% on the distractor-heavy DT setting, also passing every panoramic/multi-camera system in the paper.
• INSIGHT-Bench (deployment-oriented: 1,097 episodes, 210 indoor/outdoor scenes, one shared 120° forward RGB interface): SR 43.7% vs NaVid 26.9% and Uni-NaVid 24.3%.

The scaling analysis shows three distinct behaviours: model scaling saturates (2B→4B lifts R2R SR/SPL by 8.6/7.4 points, 8B mostly doesn't help); data scaling is monotonic but diminishing; environment coverage is the one axis that keeps paying — going from 1/8 of training environments to all of them adds 16.7 SR on R2R and 21.1 on RxR. Scene diversity, not parameters or sheer hours, is the reliable lever.

Engineering and deployment: trained entirely in simulation

The model is trained entirely in simulation ("so it scales") and transfers to real robots via scalable real2sim2real — across robots, tasks and scenes. The repo ships the full stack:

• Inference / serving: `lightnav-predict` runs instruction navigation on a video clip in one command; `lightnav-serve` streams frames over WebSocket, and several robots can share one server (micro-batched sessions).
• Simulation playground: `mujoco_demo/` bundles a MuJoCo TurtleBot in a ProcTHOR scene — no ROS, no Habitat, no GPU needed on the client.
• Real-robot deployment: `robot_deploy/` is a complete ROS 2 on-robot stack — camera driver, WebSocket client, MPC waypoint tracker and a web control panel — with Unitree Go2 and LimX TRON 1 adapters plus a bring-your-own-robot interface.
• Evaluation: VLN-CE / ObjectNav and EVT-Bench harnesses ship in the repo; INSIGHT-Bench episodes and evaluation code are released separately.

Model weights are on Hugging Face (LightOriginsHQ/LightNav-0, fine-tuned from Qwen3-VL-4B-Instruct); code and weights are Apache-2.0. Paper: arXiv 2608.30935, "LightNav-0: Eliciting VLM Spatial Intelligence for Generalist Embodied Navigation".