A Motion-Aware Vector Quantization Framework with Centroid Reuse for Efficient VLA Inference

1. Overview: Motion-Aware VQ with Centroid Reuse for Efficient VLA Inference

VQVLA, proposed by Zhuoran Song, Haozhe Jiang et al. from SJTU and CASIA (July 2026, MICRO 2026), is an algorithm-hardware co-design framework that accelerates VLA inference by exploiting weight similarity and execution dynamics. Its two core contributions: (1) MotionVQ—a motion-aware vector quantization scheme that dynamically adjusts quantization precision based on the robot's execution state, reducing memory access while preserving task success rate; (2) a merged-centroid vectorized GEMM paradigm that operates directly on the codebook-index representation, eliminating redundant multiplications through spatial aggregation and temporal reuse of centroids. Experiments show VQVLA achieves 6.5×, 2.8×, 1.9×, 3.3×, and 4.3× speedup over A100 GPU, Dadu-Corki, LUT-DLA, CodeGEMM, and ShiftAddLLM respectively, with negligible accuracy degradation.

2. Core Problem: Memory and Computation Bottlenecks of VLA Inference

The paper decomposes VLA inference latency into four stages: tokenizer, ViT, transformer backbone, and action head. Evaluating OpenVLA, OpenVLA-OFT, and RDT on A100 GPU, the stages average 2.7%, 16.6%, 80.6%, and 0.1%—the transformer backbone is the absolute bottleneck, for two reasons: (1) High off-chip memory bandwidth demand: LLaMA2-7B contains 14GB parameters; limited on-chip buffer causes frequent off-chip access; (2) High computational intensity: each layer uses 8 GEMMs with $N \times 4096 \times 4096$ and $N \times N \times 4096$ operations ($N$ = input token count, typically 200-600). Existing accelerators like Dadu-Corki treat VLA as full-precision workloads, not exploiting quantization redundancy.

3. Motivation: Correlation Between Motion Magnitude and Spatial Proximity

Analysis of robot motion behavior reveals a clear correlation: when motion magnitude is small ($x \leq 0.8$), the arm is typically near target objects ($y \leq 0.16$), requiring precise adjustment; when motion magnitude is large ($x > 0.8$), the arm is far from objects ($y > 0.16$), with less constrained movement. Operations are categorized into two states:

$$\text{state} = \begin{cases} \text{execution} & \text{if } D \leq T_d \text{ (fine control, affects success rate)} \\ \text{transition} & \text{if } D > T_d \text{ (coarse motion, no direct task impact)} \end{cases}$$

Execution state needs high precision, transition state can use low precision—this is the basis for MotionVQ's dynamic precision.

4. MotionVQ Algorithm

MotionVQ has two stages: state prediction and precision adaptation. State prediction computes motion magnitude $D$:

$$D = \sqrt{A_x^2 + A_y^2 + A_z^2}$$

where $(A_x, A_y, A_z)$ is the 3D action from the last VLA inference output. $D$ is compared to threshold $T_d$: $D \leq T_d$ is execution state, $D > T_d$ is transition state. Precision adaptation dynamically selects between two offline-generated quantization configs: high-precision uses a 256-centroid codebook, low-precision uses 64 centroids:

$$\text{precision} = \begin{cases} \text{VQ}[256,2,256] \text{ (4.125 bits, execution)} & D \leq T_d \\ \text{VQ}[128,2,64] \text{ (3.125 bits, transition)} & D > T_d \end{cases}$$

This enables a state-aware trade-off between memory efficiency and task success rate. Let high/low codebook sizes be $C_h$ and $C_l$; the average bitwidth is:

$$\bar{b} = p_{\text{exec}} \cdot \lceil \log_2 C_h \rceil + (1-p_{\text{exec}}) \cdot \lceil \log_2 C_l \rceil$$

where $p_{\text{exec}}$ is the execution-state proportion. More transition states yield lower average bitwidth.

5. Merged-Centroid Vectorized GEMM Paradigm

This kernel operates directly on the codebook-index representation, avoiding dequantization. It exploits spatial and temporal locality of centroids to eliminate redundant computation:

5.1 Spatial Merging

Exploits centroid repetition within a column (spatial-level hot centroids). Identical indices in the index matrix mean corresponding weight vectors map to the same centroid; multiple multiplications of the same hot centroid with different inputs can be consolidated. If positions $k_1, \dots, k_s$ share index $idx$, the corresponding inputs $I_i^{k_1}, \dots, I_i^{k_s}$ each multiply the same centroid $C = \text{codebook}[idx]$. By the distributive law:

$$\sum_{j=1}^{s} I_i^{k_j} \times C \;\longrightarrow\; \left(\sum_{j=1}^{s} I_i^{k_j}\right) \times C$$

Sum first, then a single multiplication—drastically reducing multiplication count. A spatial centroid location table is built per weight group for efficient implementation.

5.2 Temporal Reusing

Exploits centroid repetition across time steps (temporal-level hot centroids). If the same centroid is accessed across consecutive steps, it can be cached on-chip for reuse, avoiding repeated off-chip loading.

6. Architecture Design

The designed accelerator efficiently supports dynamic precision selection and centroid-reuse computation, with a matrix multiplication engine (spatial merging PE array and temporal reusing PE array), state predictor, and index processing engine. Implemented in Verilog, synthesized with Synopsys Design Compiler at 28nm, 500MHz, scaled to 7nm for comparison with A100. Off-chip memory access energy estimated at 3.9 pJ/bit.

7. Experimental Evaluation

7.1 Algorithm Evaluation

High-precision set VQ[256,2,256] (4.125 bits), low-precision set VQ[128,2,64] (3.125 bits). VQVLA's average success rate drops only 2.5% (acceptable). Memory access reduced 79.4% on average, multiplications reduced 54.8% on average. Savings vary by model/task: OpenVLA on LIBERO-Object reduces 71.4% multiplications, RDT on ManiSkill-StackCube reduces 52.2%.

Metric	Average Result	Notes
Success rate drop	2.5%	Acceptable accuracy loss
Memory access reduction	79.4%	Low-precision set in transition
Multiplication reduction	54.8%	Spatial merging + temporal reuse

7.2 Architecture Evaluation

Compared against A100 GPU, Dadu-Corki, LUT-DLA, CodeGEMM, and ShiftAddLLM:

Baseline	Speedup	Type
NVIDIA A100 GPU	6.5×	GPU
Dadu-Corki	2.8×	VLA-specific accelerator
LUT-DLA	1.9×	VQ accelerator
CodeGEMM	3.3×	GPU quantization method
ShiftAddLLM	4.3×	Reparameterization framework

VQVLA leads across all baselines, achieving 2.8× over the strongest VLA-specific accelerator Dadu-Corki.

8. Limitations

• 2.5% success rate loss: Though acceptable, it still impacts precision-sensitive execution-state tasks; the threshold $T_d$ requires task-specific tuning.
• Centroid repetition dependency: Spatial merging efficiency depends on hot-centroid repetition frequency in the index matrix; weight distributions unfavorable to clustering yield limited savings.
• Fixed two precision configs: Only high/low tiers; finer state-precision mapping could be better but increases hardware complexity.
• 28nm synthesis: Though scaled to 7nm for comparison, not validated on a real 7nm chip; area and power are estimates.
• Only VLA backbone evaluated: Tokenizer and ViT stages not quantized/accelerated; end-to-end speedup may be lower than backbone-only.

9. Conclusion

VQVLA is an algorithm-hardware co-design framework that uses MotionVQ motion-aware vector quantization to dynamically switch precision by execution state (256 centroids for execution, 64 for transition), and a merged-centroid vectorized GEMM kernel operating directly in the compressed domain to eliminate dequantization and redundant multiplications (spatial merging sums before multiplying, temporal reusing caches hot centroids). It achieves 6.5× over A100 GPU, 2.8× over the strongest VLA-specific accelerator Dadu-Corki, with only 2.5% success rate drop, 79.4% memory access reduction, and 54.8% multiplication reduction. The core insight: robot motion is stateful—fine motions need precision, coarse motions can be compressed, and quantized weights are naturally clusterable for reuse—the combination of motion awareness and centroid reuse makes VLA inference both fast and accurate.

flowchart LR
    A["Last VLA output action A"] --> B["Compute motion magnitude D=√(Ax²+Ay²+Az²)"]
    B --> C{"D ≤ Td?"}
    C -->|yes execution| D["High-precision codebook VQ[256,2,256]
4.125 bits"]
    C -->|no transition| E["Low-precision codebook VQ[128,2,64]
3.125 bits"]
    D --> F["Codebook+index representation"]
    E --> F
    F --> G["Merged-centroid vectorized GEMM"]
    G --> H["Spatial merging: sum then multiply centroid"]
    G --> I["Temporal reusing: cache hot centroids"]
    H --> J["VLA inference output"]
    I --> J
    J --> A