VLAct: Beyond Data Scaling via Representation-Centric Continued Pre-training for VLA

Positioning: Treating the VLM Backbone as a First-Order Design Variable for VLA

VLAct is the open-source representation-centric continued pre-training framework and model zoo from the starVLA team, accompanying the paper "Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models" (arXiv:2608.27550). It tackles a central question in today's VLA landscape: given a fixed robot-data budget, how do you distill scarce trajectories into maximally reusable visual-action knowledge?

Robot trajectories cannot be scraped from the web the way text can — every trajectory must be produced through embodied execution, and the space a policy must generalize over (scenes, objects, goals, embodiments, contact dynamics) is combinatorial and continuous. Even the largest robot datasets are sparse samples of that space. Data scaling remains essential, but it is not the only axis: with the same data budget, how trajectories are distilled into the backbone determines the downstream ceiling. VLAct treats continued pre-training as representation learning rather than action fitting, and produces a generalist action backbone built on Qwen3-VL-4B.

Core Findings: Three Failure Modes of Naive VLA Continued Pre-training

Through controlled pilot experiments, the paper isolates three failure modes that arise when you simply keep training a VLM backbone on robot data — the foundation of the entire project:

#	Failure mode	Evidence
1	Prior erosion — robot data is far narrower than web-scale corpora, so end-to-end updating overwrites broadly useful vision-language features	Updating the full backbone scores only 78.9 on LIBERO-Plus vs. 82.6 with shallow-layer protection
2	Decoder lock-in — a single pre-training head specializes the backbone to that head's decoding geometry	OFT pre-training lifts OFT fine-tuning 61.7 → 75.8, but drags PI fine-tuning from 60.5 down to 55.1
3	Discretization loss — discrete action tokens teach coarse structure but lose fine-grained temporal and amplitude information	FAST → FAST reaches only 45.2 on LIBERO-Plus, while FAST → GR00T reaches 76.7

VLAct's design principle: address all three failure modes during continued pre-training. Downstream, you discard the pre-training heads, attach a freshly initialized action head of your choice, unfreeze the full backbone, and fine-tune normally. Against the matched Qwen3-VL-OFT baseline, the VLM backbone weights are the only thing that changes — downstream head, initialization, data, optimizer, and budget are identical — so the gains are cleanly attributable to the learned backbone representation.

Method: A Three-Pronged Continued Pre-training Recipe

1. Preserve the VLM Prior

Freeze the vision encoder and the lower half of the LLM layers (shallow-layer protection), and mix image-caption data into every minibatch: $L_{total} = L_{action} + 0.5 \cdot L_{VLM\text{-}CE}$. Lower layers carry broad visual and spatial processing; captions supply dense supervision over objects, attributes, relations, and scene context, keeping the trainable layers near their original operating regime.

2. Multi-Head Co-Supervision

Against decoder lock-in, VLAct trains OFT + PI + GR00T action heads simultaneously (sharing a latent representation), combined with a partially unified action layout (disjoint action layout + padded action dims masking), keeping the backbone neutral across decoding geometries. This is the meaning of the name "VLAct": one backbone, many action decoders.

3. Wrap-Aware Losses

For angular periodicity in joint space, shortest-angular-joint loss and endpoint wrap loss ($L_{wrap}$) prevent actions like 359°→1° from being penalized as large errors.

Results: Small Model, Big Scores

With only Qwen3-VL-4B scale and open data, the continued-pretraining backbone reaches:

Benchmark	Score	Notes
LIBERO-Plus	82.6%	PI head
VLA-Arena	54.8%	PI head
RoboTwin 2.0	92.5%	OFT head
RoboCasa-GR1	49.5%	with only 20% of training data, transferring to an unseen robot
RoboDojo leaderboard	6th of 35 policies by success rate	ARX X5 real robot; ahead of every explicitly designated world-action model

The cross-embodiment transfer is particularly notable: on an unseen robot, just 20% of the data yields near-full-data performance — direct evidence of the value of a reusable backbone.

Engineering and Usage

The repository ships the complete continued pre-training pipeline (single-node 8-GPU script and multi-node Slurm script) plus downstream fine-tuning/evaluation launchers for six benchmarks: LIBERO-Plus, VLA-Arena, RoboTwin 2.0, DOMINO, RoboCasa, and RoboDojo. Recipe components map cleanly to code locations:

Recipe component	Where it lives
Shallow-layer protection (vision encoder + LLM layers 0–17)	--trainer.freeze_modules
Caption-mixed co-training	--datasets.vlm_data.dataset_use, --trainer.loss_scale.vlm
Multi-head co-supervision	--framework.heads oft,gr00t,pi, --framework.head_loss_weights
Partially unified action layout	--framework.disjoint_action_layout, --framework.mask_padded_action_dims
Wrap-aware loss	--trainer.shortest_angular_joint_loss*, --trainer.endpoint_wrap_loss_weight

The model zoo on HuggingFace releases the raw continued-pretraining backbone (the recommended starting point for new embodiments, datasets, or decoders) plus downstream checkpoints for RoboDojo, RoboTwin, DOMINO, VLA-Arena, and LIBERO-Plus. Note: the continued-pretraining checkpoint is not a directly deployable policy — initialize an incompatible downstream action head from scratch, and keep config.yaml and dataset_statistics.json alongside the checkpoint.

Takeaway

VLAct's value is not yet another bigger VLA, but turning "how to refine scarce robot data into a good backbone" into reproducible science: the ablations over three failure modes are clean, multi-head co-supervision is an elegant answer to decoder lock-in, and 4B-scale results competitive with much larger models validate the route's efficiency. For teams fine-tuning VLAs on their own robots, starting from the released continued-pretraining backbone is currently one of the best value choices. The code is open-sourced under the MIT license.