Per-Group Error, Not Total MSE: Fine-Tuning Vision-Language-Action Models for 11-DoF Mobile Manipulation

Per-Group Error, Not Total MSE: Fine-Tuning VLA Models for 11-DoF Mobile Manipulation

Paper: Per-Group Error, Not Total MSE: Fine-Tuning Vision-Language-Action Models for 11-DoF Mobile Manipulation
Authors: Pau Montagut Bofi, Mario García Blasco, Tessa Pulli, Markus Vincze
Institution: TU Wien
Links: arXiv:2606.00253 | GitHub Code

One-Sentence Summary

This paper reveals that when fine-tuning VLA models on mobile manipulators with heterogeneous joint spaces (arm, gripper, head, wheeled base), the checkpoint with lowest total MSE is not the best on the real robot — per-group error analysis shows SmolVLA's bottleneck is the mobile base (slowest convergence), while π₀.₅ expert-only fine-tuning's deployment bottleneck is arm degradation, validated by 60 real-robot trials confirming per-group error as a more reliable deployment signal than total MSE.

Background and Motivation

Deploying pretrained VLA models on mobile manipulators is where published benchmarks meet their limits. Most VLAs (π₀, SmolVLA, OpenVLA) are trained on fixed arms with 6-7 DoF Cartesian actions, while platforms like the Toyota HSR require controlling 11 joints spanning arm, gripper, head, and wheeled base — a combination absent from most pretraining corpora.

The heterogeneity of these joint groups — from binary gripper to continuous base motion — matters beyond architecture: when 11 dimensions are aggregated into a single MSE, easy-to-predict groups can mask groups that fail. A checkpoint can look competitive on total MSE yet behave poorly on the robot because a deployment-critical group is underfit. Which group that is depends on the model and training regime.

The core contribution: total MSE is not a reliable criterion for selecting deployment checkpoints, while per-group error decomposition reveals the true training dynamics and deployment bottleneck. This is demonstrated through: (1) a two-phase fine-tuning pipeline for SmolVLA with native 11D output; (2) comparative analysis against π₀.₅ (3.3B) including expert-only fine-tuning; (3) error decomposition by joint group; (4) Toyota HSR real-robot validation.

Figure 1: End-to-end pipeline. Pretraining SmolVLA on HSR teleoperation data, task-specific fine-tuning on pick-up task (producing HSR-SmolVLA), offline + real-robot evaluation.

Method

Models. SmolVLA is a 450M-parameter lightweight VLA with configurable action dimensionality, natively compatible with HSR's 11-DoF joint space without architectural changes. π₀.₅ is a 3.3B-parameter stronger pretrained baseline using flow-matching action heads, with mobile manipulation in its pretraining corpus.

Training. SmolVLA uses two-phase fine-tuning: Phase 1 pretrains on 109k HSR teleoperation episodes for 20k steps to introduce the base modality; Phase 2 fine-tunes on 3,971 pick-up task episodes for 50k steps. π₀.₅ expert-only fine-tuning trains only the action head (backbone frozen) from the 80k baseline checkpoint.

Per-Joint-Group Error Analysis. The 11 joints are divided into four groups $\mathcal{G}=\{\text{arm},\text{gripper},\text{head},\text{base}\}$, each with independent MSE:

$$\text{MSE}_{g}=\frac{1}{|g|}\sum_{j\in g}(\hat{a}_{j}-a_{j})^{2},\quad g\in\mathcal{G}$$

where $\hat{a}_{j}$ and $a_{j}$ are predicted and ground-truth values for joint $j$. This decomposition makes each group's convergence dynamics visible rather than masked by aggregate MSE.

flowchart TD
  A[VLA Pretrained Model] --> B{Select Model}
  B --> C[SmolVLA 450M]
  B --> D[pi_0.5 3.3B]
  C --> E[Phase 1: 109k HSR episodes pretrain 20k steps]
  E --> F[Phase 2: 3971 task episodes finetune 50k steps]
  D --> G[Expert-only finetune: action head only]
  D --> H[Baseline 80k/100k]
  F --> I[Per-Group Error Decomposition]
  G --> I
  H --> I
  I --> J[Arm MSE]
  I --> K[Gripper MSE]
  I --> L[Head MSE]
  I --> M[Base MSE]
  J --> N[Total MSE != Deployment Performance]
  M --> N
  N --> O[Real-Robot Validation 60 trials]
  O --> P[Per-Group Error More Reliable]

SmolVLA Results

Units ×10⁻³. Total MSE lowest at 40k (1.61); 50k slight regression indicates overfitting.

SmolVLA's total MSE drops 80% during fine-tuning (5.10→1.61), but this average hides critical differences: gripper error falls fastest (8.51→0.85, ~10×), head remains minimal (0.05), arm converges by 20k (0.91→0.88), while base is the last group to converge (7.24→2.14) and sets the ceiling on total MSE. The base is SmolVLA's fine-tuning bottleneck — the underlying checkpoint was pretrained on LeRobot community data dominated by fixed-arm teleoperation with no mobile-base exposure. Phase-1's 109k HSR episodes introduce the base modality, but 20k+50k steps are insufficient to bring base error down to gripper or head levels.

Figure 2: Per-joint-group MSE over SmolVLA training. Gripper error falls fastest; base error is last to converge and sets the ceiling on total MSE.

π₀.₅ Baselines and Expert-Only Fine-Tuning

Units ×10⁻³. Expert 3k has lowest total MSE (0.95) but arm degrades from 0.30 to 0.59.

The π₀.₅ 80k baseline achieves total MSE 1.04, outperforming SmolVLA's best (1.61) by 35.4%. Continuing to 100k degrades performance (1.21, +16%), indicating overfitting even in the pretrained model. Expert-only fine-tuning from 80k improves total MSE to 0.95 at 3k steps (-9.0%), primarily through gripper improvement (4.31→2.21). However, arm MSE degrades (0.30→0.59), suggesting optimization is dominated by joint groups with larger error magnitudes (gripper and base): reducing their error trades off accuracy on the already well-predicted arm. At 5k steps, gripper MSE spikes to 4.21, indicating rapid overfitting.

Real-Robot Validation

60 real-robot trials (20 per model) on Toyota HSR pick-up tasks. π₀.₅ 80k achieves perfect 20/20 (4.0/4); the two fine-tuned models perform worse: expert 3k at 3.75/4 (15/20 lifts), HSR-SmolVLA at 3.5/4 (12/20). Mann-Whitney tests confirm π₀.₅ 80k significantly outperforms expert 3k ($p=0.010$) and HSR-SmolVLA ($p=0.001$).

Key finding: expert 3k has the lowest total MSE (0.95 vs 1.04) and lower base-group error (1.72 vs 1.85), yet loses to the untouched baseline on the robot. This disconnect aligns with offline arm-group error (0.59 vs 0.30), not base-group error or total MSE. In 5 failed trials, the robot correctly approached and closed the gripper around the object but failed to lift cleanly — consistent with an arm policy that lost calibration. HSR-SmolVLA failures involved accurate navigation but ~20 cm grasping miscalibration, consistent with its higher residual arm error ($0.88\times10^{-3}$ vs π₀.₅'s $0.30\times10^{-3}$).

The per-group MSE decomposition $ ext{MSE}_{g}=rac{1}{|g|}\sum_{j\in g}(\hat{a}_{j}-a_{j})^{2}$ is simple but reveals a structural problem. When per-group error magnitudes differ by 1-2 orders of magnitude (e.g., SmolVLA gripper 8.51 vs head 0.11), the standard unweighted total MSE objective:

$$\mathcal{L}_{ ext{total}}=\sum_{g\in\mathcal{G}}rac{|g|}{n} ext{MSE}_{g}$$

is dominated by large-magnitude groups. In SmolVLA at 40k, base MSE is 2.14 while arm is 0.88 and gripper 0.85 — base contributes ~53% of total MSE while arm only ~22%. This means gradient descent naturally prioritizes reducing base error, potentially at the cost of arm precision stagnating.

The arm degradation in π₀.₅ expert fine-tuning can be understood as capacity reallocation. With only the action head trainable, limited capacity is redirected to large-magnitude groups (gripper 4.31, base 1.85), degrading the already-good arm (0.30). An uncertainty-weighted objective:

$$\mathcal{L}_{ ext{weighted}}=\sum_{g\in\mathcal{G}}rac{1}{2\sigma_{g}^{2}} ext{MSE}_{g}+\log\sigma_{g}$$

where $\sigma_{g}$ is a learnable uncertainty per group, would automatically assign higher weight to small-magnitude groups, protecting them. This is left as future work.

The per-group error imbalance ratio is:

$$ ho=rac{\max_{g} ext{MSE}_{g}}{\min_{g} ext{MSE}_{g}}$$

At SmolVLA pretrained 0k, $ ho=8.51/0.11pprox77.4$; at 40k finetune, $ ho=2.14/0.05pprox42.8$. Though reduced after fine-tuning, the ratio remains large, justifying per-group reporting.

Real-robot scoring uses the Mann-Whitney U test. The U statistic for two independent samples is:

$$U=\sum_{i=1}^{n_1}\sum_{j=1}^{n_2}\mathbb{1}(X_i>Y_j)+0.5\cdot\mathbb{1}(X_i=Y_j)$$

with $n_1=n_2=20$ trials per model. π₀.₅ 80k vs expert 3k yields $p=0.010$, vs HSR-SmolVLA $p=0.001$, confirming significant baseline advantage. The two fine-tuned models' $p=0.128$ honestly reports non-significance.

Overfitting appears in both models. SmolVLA total MSE rises from 1.61 (40k) to 1.63 (50k); π₀.₅ expert gripper MSE spikes from 2.21 (3k) to 4.21 (5k). Per-group monitoring can detect when a group's validation error starts rising while training error continues falling — π₀.₅ expert arm degradation at 3k (0.30→0.59) is the direct cause of deployment performance loss.

Figure 3: Real-robot evaluation tasks on Toyota HSR. The robot must approach and grasp a ceramic mug or Cheez-It cracker box, 20 trials per model.

Discussion and Conclusion

The most concrete lesson: total MSE is not a reliable criterion for selecting deployment checkpoints, and the dominant residual-error group varies by model and training regime. SmolVLA's training bottleneck is the base group (slowest convergence); π₀.₅ expert fine-tuning's deployment bottleneck is the arm group (degraded under fine-tuning). The shared mechanism is familiar in multi-output regression: when per-group loss magnitudes are imbalanced by 1-2 orders of magnitude, an unweighted objective trades capacity away from small-magnitude groups. Uncertainty weighting or gradient balancing could protect the arm while improving gripper and base.

Three guidelines for teams deploying VLAs on heterogeneous platforms: (1) report per-group error alongside total MSE; (2) identify the dominant group in each regime before trusting it as a deployment proxy; (3) keep fine-tuning short when task data is scarce — both models enter overfitting.

Limitations

Limited trial scale. 60 trials (20 per model) suffice for statistical comparison of baseline vs fine-tuned ($p\leq0.010$) but cannot resolve the gap between the two fine-tuned models ($p=0.128$). Generalization to other tasks and embodiments remains future work.

Offline MSE is frame-level. It does not capture errors that compound over trajectories. The SmolVLA vs π₀.₅ comparison involves different pretraining corpora and is not a controlled architecture study — it reflects the practical choice between a small-team checkpoint and a larger pretrained baseline.