FM-VLA: Force-based Memory for Vision-Language-Action Models in Contact-Rich Manipulation

Background and Motivation

Vision-language-action (VLA) models have achieved remarkable generalization across diverse manipulation tasks by leveraging internet-scale vision-language pretraining. However, most existing VLA architectures rely solely on the current observation, treating decision-making as a memoryless mapping $\pi(a_t \mid o_t, l)$. While this Markovian assumption suffices for many tasks, it breaks down in real-world long-horizon scenarios where the correct action depends on past interactions rather than the instantaneous observation alone.

Consider this scenario: a robot must press a button exactly N times. Each press yields negligible visual displacement, but the force sensor records a distinct, sharp impulse. Or: a robot must find a hidden block under one of two visually identical upside-down cups—after inspecting each cup, the scene returns to its original appearance, so the policy must remember which cup was already checked. These tasks are inherently non-Markovian; the correct action depends on accumulated past states and interactions.

Prior work has explored two memory paths. The first is visual memory: MemoryVLA maintains a memory bank over past observations, while MEM combines short-term video memory with long-term text summaries. But these methods fail when visual changes are subtle or observations are severely occluded—repeated button presses may produce no observable visual change. The second is force augmentation: ForceVLA and TA-VLA integrate force signals as a first-class modality into VLA, but they focus on using current or short-window force readings to improve action prediction, not on using force as a mechanism for tracking interaction progress. They capture local interaction states—whether contact is established or how much force is applied—but cannot accumulate the long-horizon temporal information required for non-Markovian decision-making.

FM-VLA is designed to bridge this gap. The key insight is that many interaction-relevant signals—contact events, force magnitude, the count of repeated actions—are naturally captured by force sensor measurements, providing a more direct and unambiguous representation of interaction dynamics than vision. The core challenge: how to transform high-frequency, noisy, unstructured raw force signals into memory representations that effectively guide action generation?

Preliminaries

Understanding FM-VLA requires several key concepts. First, six-axis force/torque sensors: mounted on robot wrists, measuring 3-axis force and 3-axis torque at 100Hz, with each reading $f_\tau \in \mathbb{R}^6$. Second, variational autoencoders (VAE): generative models that compress high-dimensional data into a low-dimensional latent space and reconstruct it, trained via reconstruction loss and KL regularization. Third, flow-matching action experts: the module in VLA models responsible for generating action sequences, learning a vector field from noise to actions to produce continuous action chunks.

Another key concept is non-Markovian decision-making: traditional VLAs assume the current observation contains all decision-relevant information (Markov property), but in contact-rich tasks, the correct action depends on the accumulated history of contact events, requiring explicit memory mechanisms.

Method: FM-VLA Architecture

FM-VLA is built on $\pi_{0.5}$, which consists of a vision-language model (PaliGemma with SigLIP vision encoder) and a flow-matching action expert. The VLM processes the current image and language instruction, and its internal features condition the flow-matching action expert via cross-attention to generate action chunks. FM-VLA's core innovation is the force-based memory module that encodes long-horizon force history into memory tokens injected into the action expert.

Architecture Overview

flowchart TB
    subgraph VLM["VLM (PaliGemma + SigLIP)"]
        IMG[Current Image] --> VL[VL Features]
        LANG[Language] --> VL
    end
    subgraph FM["Force Memory Module"]
        FH[Wrist F/T History
100Hz, 6-axis] --> EMA[First-order EMA]
        EMA --> PAD[Random Noise Pre-padding]
        PAD --> VAE[VAE Encoder
Frozen]
        VAE --> ZF[Force Memory Tokens
K=8, d_h]
        SH[Joint State Window
Last 1s] --> PROJ[Linear Projection
Zero-init]
        PROJ --> ZS[State Memory Token
1, d_h]
    end
    VL --> AE[Flow-Matching Action Expert]
    ZF --> AE
    ZS --> AE
    AE --> OUT[Action Chunk Output]

The diagram above shows the complete FM-VLA architecture. Force/torque history is smoothed by EMA and noise-pre-padded, then compressed by the frozen VAE encoder into K=8 force memory tokens. The joint-state short window is mapped by a linear projection to a single state memory token. Both groups of tokens are injected into the flow-matching action expert's suffix.

Problem Formulation

FM-VLA learns a policy $\pi(a_t \mid o_t, l, h_t)$ mapping the current observation $o_t$, language instruction $l$, and temporal history $h_t$ to an action $a_t$. Unlike the memoryless policy $\pi(a_t \mid o_t, l)$, this introduces two complementary proprioceptive streams. The first is a long-horizon wrench history $\{f_\tau\}_{\tau=1}^{t}$, where each $f_\tau \in \mathbb{R}^{d_f}$, $d_f = 6$, stacks 3-axis force and 3-axis torque—this stream captures accumulated contact events over an entire episode. The second is a short-window joint-state history $\{s_\tau\}_{\tau=t-W+1}^{t}$, where each $s_\tau \in \mathbb{R}^{d_s}$ concatenates joint positions and gripper states (e.g., $d_s = 16$ for a 7-DoF bimanual setup)—this stream captures recent proprioceptive dynamics. Together they form the temporal-history representation:

$$h_t = \big[\,\underbrace{\mathrm{Enc}_\phi(\{f_\tau\}_{\tau=1}^{t})}_{\text{wrench history } Z_f \in \mathbb{R}^{K \times d_h}} \;\;\|\;\; \underbrace{\mathrm{Proj}_\psi(\{s_\tau\}_{\tau=t-W+1}^{t})}_{\text{state history } z_s \in \mathbb{R}^{d_h}} \,\big]$$

where $\mathrm{Enc}_\phi$ is a pretrained VAE encoder compressing the unbounded wrench history into K fixed-dimensional latent tokens, and $\mathrm{Proj}_\psi$ is a lightweight linear projection mapping the joint-state window to a single token, learned end-to-end with the VLA.

Wrench History Processing

Raw force sensor readings are noisy with high-frequency content. The paper employs two preprocessing steps. First-order exponential moving average (EMA) smoothing: a causal filter removing most noise while preserving signal onsets and peaks:

$$\tilde{f}_{\tau} = \alpha f_{\tau} + (1 - \alpha)\tilde{f}_{\tau-1}$$

where $\alpha$ controls the smoothing degree. Randomized noise pre-padding: the paper notices that wrench history length leaks temporal episode progress, letting the model shortcut on sequence length rather than utilizing signal temporal structure. To remove this cue, each history is prepended with a random-length prefix of low-amplitude Gaussian noise (uniformly sampled up to 10s) during training, randomizing the padding length.

Force Memory Encoder (VAE)

The force memory VAE is based on a Perceiver-IO architecture operating on K learnable latent query tokens. Given a wrench history $F = [f_1, \ldots, f_T] \in \mathbb{R}^{T \times d_f}$, the signal at each time-step is first quantile-normalized based on entire-dataset statistics, then projected through an input MLP and integrated with Fourier positional encoding to yield wrench tokens. The encoder leverages cross-attention to extract wrench features into the latent queries, interleaved with self-attention blocks. Finally, a per-latent linear head outputs posterior parameters:

$$\mu_k, \log \sigma_k^2 = \text{Head}_{\text{VAE}}\big(\text{Enc}_\phi(F)_k\big), \quad z_k = \mu_k + \sigma_k \odot \epsilon_k, \quad \epsilon_k \sim \mathcal{N}(0, I)$$

yielding a latent representation $Z \in \mathbb{R}^{K \times d_z}$ of K tokens. The decoder reverses this process via cross-attention layers, where time-step-specific Fourier-encoded queries attend to the latent tokens to produce the reconstructed sequence $\hat{F} \in \mathbb{R}^{T \times d_f}$.

Short State History

Unlike the wrench stream, the proprioceptive role of the joint-state stream is well captured by very recent states—the action expert only needs to know "where the arms are and where they are heading." Therefore, the paper avoids a second VAE on the state side, using a lightweight projection layer with no pretraining. At each control step $t$, a short window $S_t \in \mathbb{R}^{W \times d_s}$ of the most recent joint-state frames is extracted by sub-sampling at a fixed stride (covering the last second of motion), flattened, and projected to a single state history token via a zero-initialized linear layer.

Memory Token Injection

Both proprioceptive memory streams enter the policy exclusively through the action-expert suffix. From the frozen force encoder, only the posterior mean $\mu_f \in \mathbb{R}^{K \times d_z}$ is taken and projected from the VAE latent dimension $d_z$ to the action-expert hidden dimension $d_h$ via a zero-initialized linear layer, yielding force memory tokens $Z_f \in \mathbb{R}^{K \times d_h}$. The action-expert sequence layout is:

$$\underbrace{[a_k^{(1)}, \ldots, a_k^{(H)}]}_{\text{noisy-action tokens}} \;\|\; \underbrace{[Z_f^{(1)}, \ldots, Z_f^{(K)}]}_{\text{wrench memory}} \;\|\; \underbrace{[z_s]}_{\text{state window}}$$

Wrench tokens are appended immediately after noisy-action tokens, and the state-window token is last. Placing both memory streams in the post-position keeps noisy-action tokens at the same RoPE positions they had during base-policy pretraining, avoiding disruption of existing action generation capabilities.

Two-Stage Training

Stage 1: Force-VAE pretraining. The force memory VAE is trained jointly on wrench histories from all tasks using a masked-ELBO objective. The loss consists of two parts: a masked reconstruction term over valid frames and a free-bits-regularized KL on each latent dimension:

$$\mathcal{L}_{\text{VAE}} = \frac{1}{\sum_\tau m_\tau \cdot d_f} \sum_{\tau=1}^{T} m_\tau \|f_\tau - \hat{f}_\tau\|^2 + \beta \cdot \frac{1}{K d_z} \sum_{k,j} \max(D_{\text{KL}}^{(k,j)}, \lambda)$$

where $m_\tau \in \{0,1\}$ masks padding frames, $D_{\text{KL}}^{(k,j)}$ is the per-dimension KL divergence between the posterior and standard normal prior, $\beta$ controls regularization strength, and $\lambda$ is the per-dimension free-bits floor—switching off the KL gradient on dimensions that already encode less than $\lambda$ nats to prevent posterior collapse.

Stage 2: VLA finetuning. The force encoder is frozen and switched to evaluation mode (taking only the posterior mean), and finetuned end-to-end with the VLA policy. The VLA is trained with the standard flow-matching loss, with force memory tokens as additional conditioning signals injected into the action-expert suffix.

Experimental Results

Experimental Setup

Data collection and experiments are conducted on an AgiBot G1 bimanual humanoid robot with two 7-DoF arms and two 1-DoF grippers. Each wrist is equipped with a 6-axis force/torque sensor at 100Hz. Policy inputs include 6-DoF wrist wrench, three RGB streams (head + two wrist cameras), and a proprioceptive state vector.

Three contact-rich bimanual tasks are designed. (1) Find a Block Under Two Cups: the robot sequentially lifts two visually identical upside-down cups to find a hidden wooden block. Since the scene returns to its original appearance after each cup is placed back, the policy must remember which cup was already inspected. 200 demonstrations. (2) Push Buttons: the robot presses a button exactly $N \in \{1,2,3\}$ times. The button's minimal travel distance produces negligible visual displacement but a distinct wrench impulse—counting requires force history. 350 demonstrations. (3) Wipe Dishes: the robot grasps a sponge and wipes a bowl's interior for a specified number of passes $N \in \{1,2,3\}$. Visual changes are marginal; force history is the dominant cue. 200 demonstrations.

Main Results

Table 1: Success rates (%) of different methods on all tasks.

FM-VLA consistently outperforms all baselines with 83.3% average success rate. The advantage is particularly pronounced on the Buttons and Wipe tasks, which require precise temporal reasoning over force signals. The memoryless $\pi_{0.5}$ struggles significantly (27.8% average), failing entirely on the wiping task. TA-VLA performs similarly poorly—its short moving window of force captures instantaneous contact but cannot retain the long-term event counts required. Notably, the visual-memory baseline $\pi$-MEM shows moderate improvements on Cups and Wipe but fails heavily on Buttons (33.3% vs. our 72.2%), confirming that visual memory is insufficient for tasks lacking clear visual state changes.

Ablation Studies

Modality: What to remember? Force-only drops to 25.9% average—the policy lacks short-term spatial awareness before making contact, leading to erratic pre-contact motions. State-only drops to 40.7%—achieving 100% on Cups but only 11.1% on Buttons, as state cannot replace force history for recording contact events. The combination reaches 83.3%, proving force and state memory are complementary and both indispensable.

Table 2: Modality and architecture ablation results (%).

Architecture: Why VAE? Replacing the VAE encoder with a GRU recurrent encoder (33.3%) and a Q-Former cross-attention module (57.4%) both significantly underperform. The GRU suffers from vanishing gradients over long 100Hz sequences, losing early contact events (Wipe at only 5.6%); the Q-Former overfits to instantaneous peaks instead of holistic temporal structure. The VAE is pretrained on a continuous wrench reconstruction objective, forcing the latent space to encode macroscopic structure—force magnitudes, onset timings, contact counts—in a few tokens, making task-relevant signals easy for the action expert to extract.

Capacity: How many tokens? The VAE latent token count is ablated over $\{4, 8, 16, 32\}$ on the Wipe task. 4 tokens form an informational bottleneck; 16 and 32 tokens unexpectedly degrade performance—as the pretrained $\pi_{0.5}$ action expert observes at most 50 tokens during training, 32 extra force tokens exceed this limit and disrupt action generation. 8 tokens achieve the peak success rate.

Inference Efficiency

Table 3: Inference latency on RTX 4090.

FM-VLA has 64ms latency, adding only 3ms over the base model. In contrast, $\pi$-MEM adds 39ms due to multiple RGB frames input to the vision/video encoder, reaching 190ms at K=16. The low-dimensional nature of force memory enables FM-VLA to achieve far superior performance while maintaining high efficiency.

Limitations

The paper acknowledges two limitations. First, the VAE latent space introduces a fixed bottleneck of 8 tokens; for very long-horizon tasks requiring memory over hundreds of contact events, hierarchical or adaptive compression may be needed. Second, the VAE is trained on force data from the demonstration dataset; pretraining on large-scale robot datasets with diverse force/torque recordings could further improve performance.

From an independent assessment, another limitation lies in the narrowness of task design. The three tasks, while carefully designed to expose non-Markovian properties, are artificially constructed memory-dependent scenarios. The value of force memory in real-world contact-rich tasks—such as assembly, insertion, or deformable object manipulation—remains untested. Furthermore, the task-agnostic claim of the VAE is insufficiently validated: although Stage 1 is jointly trained on all tasks to produce a general-purpose wrench latent space, the limited data from only three tasks leaves the transferability of this latent space to unseen task types untested.

Conclusion and Outlook

FM-VLA demonstrates that force-based memory is an effective approach for non-Markovian contact-rich manipulation. Through a two-stage paradigm—first training a VAE to compress force history into compact representations, then injecting the frozen encoder into the VLA action expert—the model gains accumulated event memory even when visual observation is limited or ambiguous, a capability neither vision-based memory nor single-token force conditioning provides. On three contact-rich bimanual tasks, FM-VLA achieves 83.3% average success rate, significantly outperforming baselines while introducing only 3ms additional inference overhead.

This work opens a new direction for multimodal memory in VLA models: force is not just an auxiliary signal for improving current actions, but a first-class memory modality for tracking long-horizon interaction progress. As force sensors become standard on robot platforms and large-scale force datasets accumulate, force-based memory may become a standard component of contact-rich manipulation systems.