PAPER DEEP DIVE
Memory as Plans: World-Action Modeling with Memory-Grounded Planning
Most robot policies predict actions from the current observation or a short fixed window, yet long-horizon manipulation is non-Markovian: the evidence a decision needs may already be out of view. MaP-WAM treats memory as planning-time evidence instead of executor input. Completed segments are stored as structured multimodal records pairing a language instruction with sparse visual context (8 uniformly sampled frames per segment); a fine-tuned Qwen3.5-4B language planner proposes the next segment-level language plan, and a causal world model initialized from WAN-2.2-5B turns the long-term visual context into a matching visual plan. The two form a memory-grounded plan that conditions a World-Action-Progress executor, a Mixture-of-Transformers extension of a video DiT that jointly predicts action chunks, future visual latents, and execution progress, with progress as a first-class modality. Because the executor only sees a fixed-length plan prefix, its context stays constant as history grows, and block-causal attention makes both planning and execution KV-cacheable; plan-observation alignment calibrates recursively predicted progress against visual-plan frames, and a progress gate (threshold 0.95) triggers segment transitions that write resampled real observations back into memory. On RMBench MaP-WAM reaches 83.3% average success over nine memory-dependent tasks (best baseline LingBot-VA 77.1%), and 78.0% on two real Franka Research 3 tasks (88% Find Button, 68% Press Buttons), while a full-context executor runs out of memory at 1700 history frames and WAP holds an approximately constant 827 ms per action chunk. Limitations: segment structure is taken from benchmark annotations rather than discovered automatically, and plan-observation alignment uses a lightweight training-free pixel-difference metric.
Paper Metadata
Title: Memory as Plans: World-Action Modeling with Memory-Grounded Planning
Authors: Sizhe Zhao, Haozhe Xie, Weiyu Zhao, Chenchu Zhang, Huan Wang, Chenyang Wang, Qinglin Liu, Shengping Zhang (Harbin Institute of Technology, Nanyang Technological University, Shandong University, HIT (Weihai) Qingdao Research Institute)
Links: arXiv:2609.11561 · project page · code repository github.com/aipixel/MaP-WAM (repository created; training/inference code and checkpoints are listed as Coming Soon in the release plan and were not public at the time of this read)
One-line summary: MaP-WAM stops feeding long histories to the executor; it compresses completed segments into a memory-grounded language-plus-visual plan, executes that plan with a fixed-context World-Action-Progress model, and closes the loop with progress-gated segment transitions.
Background and Motivation
Most robot policies, whether vision-language-action models or world-action models, formulate action prediction as a Markovian problem: condition on the current observation or a short fixed window and regress the next action chunk. The approximation works for reactive manipulation but fails on memory-dependent tasks, where the evidence a decision needs may already be out of view. The color of a button was seen once before the covers came down; the initial positions of two blocks were recorded once before the swap. None of that is in the current frame.
Existing memory mechanisms fall into three families. Language memory compresses history into a summary or an intermediate subtask instruction: compact and interpretable, but it discards fine-grained visual and spatial evidence. Continually updated memory keeps latent states or memory banks that can be overwritten as compression repeats. Growing windows concatenate raw observations into the context; causal world models such as LingBot-VA make this natural, but frame-wise history makes inference latency and GPU memory grow with trajectory length, forcing a hard trade-off between history coverage and execution efficiency.
The paper's key observation is that long-horizon visual history is not needed as a direct input to the executor at every control step. Its main job is to determine the next segment-level plan and the desired visual evolution; once the plan exists, execution can simply follow it. This decouples reading history from control into two modules: memory-grounded planning and plan-conditioned execution.
The resulting system, MaP-WAM, maintains a structured multimodal episodic context made of the task instruction, completed segment instructions, and sparse visual context. At each planning stage a vision-language model proposes the next segment-level language plan and a causal world model generates the matching visual plan from the long-term visual context. A World-Action-Progress (WAP) model then executes the plan on a fixed-length plan prefix, jointly predicting action chunks and execution progress. The executor context never grows with history, and block-causal attention makes both planning and execution key-value cacheable.
Following a plan raises a new problem: the number of control steps a segment needs is unknown in advance, depending on task difficulty and stochastic execution. The paper's answer is to model progress as a first-class modality. WAP generates a progress sequence together with actions, a progress gate decides when a segment is complete, and plan-observation alignment uses the visual plan as a temporally indexed reference to calibrate the drift of recursive progress prediction. These pieces form the third contribution.
Preliminaries
Conditional flow matching. For a target $y$ and condition $c$, flow matching interpolates $y$ with Gaussian noise $\epsilon\sim\mathcal{N}(0,I)$ at flow time $\rho\in(0,1)$ to build $y^{\rho}=(1-\rho)y+\rho\epsilon$, then trains a velocity field $v_{\theta}$ to point from data toward noise:
$$\mathcal{L}_{\mathrm{FM}}(y,c)=\mathbb{E}\left[\left\|v_{\theta}(y^{\rho},c,\rho)-(\epsilon-y)\right\|_{2}^{2}\right]$$
Both the visual planner and the executor in MaP-WAM are trained with this objective: the visual plan, future visual latents, action chunks, and progress sequences each enter as a flow-matching target sharing one condition.
Mixture-of-Transformers. MoT gives each modality its own expert parameters inside a shared Transformer backbone, letting heterogeneous tokens exchange information through joint self-attention. MaP-WAM extends a pretrained video DiT with an action expert and a progress expert so that visual, action, and progress modalities denoise together in a single forward pass.
Segment structure. A demonstration trajectory is partitioned into semantically coherent segments, each with one language instruction $\text{l}_{k}$. RMBench supplies segment boundaries during training; at deployment the boundaries are decided online by the progress gate. This structure is the shared skeleton for memory organization and progress annotation.
Method
Formulation and decomposition. A generic memory-dependent policy writes $\pi(\text{a}_{t+1:t+h}\mid\text{f}_{\leq t},\text{s}_{t},\text{l})$, and the difficulty is that critical information may live only in historical observations $\text{f}_{<t}$. MaP-WAM splits this into a planner that produces a memory-grounded plan from the episodic context available before segment $k$,
$$\pi_{\mathcal{P}}(\text{C}_{k}\mid\text{C}_{<k})$$
and an executor that predicts short-horizon actions within the plan's horizon:
$$\pi_{\mathcal{E}}(\text{a}_{t+1:t+h}\mid\text{C}_{k},\text{f}_{t},\text{s}_{t},\text{l}),\quad[t,t+h]\subseteq\mathcal{H}(\text{C}_{k})$$
The horizon $\mathcal{H}(\text{C}_{k})$ is not fixed by design; it is determined online by progress-gated transitions. Because the inputs of $\pi_{\mathcal{E}}$ are independent of history length, the executor context stays constant as the task proceeds, which is the starting point of every efficiency claim in the paper.
Structured multimodal episodic context. Each completed segment $i$ contributes a record $\text{C}_{i}=\{\text{l}_{i},\text{G}_{i}\}$: its language instruction plus sparse visual context, a fixed-length sequence of frames uniformly sampled from the real execution trajectory ($N=8$, obtained by dividing the segment into eight equal temporal bins and keeping the last frame of each). The initial observation is stored separately as $\text{G}_{0}=\text{f}_{0}$. Before planning segment $k$ the prefix holds $1+8(k-1)$ frames, temporally compressed by the WAN-VAE into $1+2(k-1)$ latent timesteps, so memory grows deliberately slowly.
Language planner. Given the global instruction $\text{l}$, completed segment instructions $\text{l}_{<k}$, and a compact keyframe set $\text{f}^{\star}$ extracted from $\text{G}_{<k}$ (the initial frame plus the last frame of each completed segment), a fine-tuned VLM predicts the next language plan:
$$\pi_{\mathcal{P}}^{l}(\text{l}_{k}\mid\text{l}_{<k},\text{l},\text{f}^{\star})$$
The implementation is Qwen3.5-4B with LoRA (rank 32, alpha 64); the prompt template feeds the image sequence and the completed subtask list and asks for the instruction of subtask $k$. The language plan carries only the semantic objective; object identity and spatial relations are delegated to the visual plan.
Causal world model as visual planner. The visual planner $\pi_{\mathcal{P}}^{v}$ is a causal world model that generates the visual plan $\text{G}_{k}$ conditioned on the long-term visual context $\text{G}_{<k}$, the language plan $\text{l}_{k}$, and the global instruction, trained with the standard flow-matching objective:
$$\mathcal{L}_{\mathcal{P}}^{v}=\mathcal{L}_{\mathrm{FM}}(\text{G}_{k},(\text{G}_{<k},\text{l}_{k},\text{l}))$$
Tokens are organized into segment-wise blocks with a block-causal self-attention mask: block $i$ attends only to blocks $j\leq i$, preventing future leakage across segments. In cross-attention, completed evidence blocks attend only to the global instruction $\text{l}$ while the target block attends to the segment language plan $\text{l}_{k}$. The hidden states of the completed prefix therefore depend on neither the target block nor $\text{l}_{k}$, so their key-value states can be computed once and reused at later planning stages within the same episode.
World-Action-Progress executor. Each training segment $\text{f}_{i:j}$ is annotated with normalized progress $\text{p}_{t}=\frac{t-i}{j-i}$ for $t\in[i,j]$, a continuous coordinate aligning execution states with the visual plan. WAP extends a pretrained video DiT with an action expert (1.02B parameters, hidden dimension 1024) and a progress expert (207M parameters, hidden dimension 256) in a Mixture-of-Transformers layout. Given the segment plan $\text{C}_{k}=\{\text{l}_{k},\text{G}_{k}\}$, current observation $\text{f}_{t}$, proprioceptive state $\text{s}_{t}$, and current progress $\text{p}_{t}$, it jointly predicts future visual latents, the action chunk, and the progress sequence. The visual plan is encoded as a static clean prefix, the current observation as clean state tokens, and future visual, action, and progress tokens as noisy targets. The structured attention mask lets dynamic tokens attend to the plan and state while the plan prefix never sees dynamic tokens, so it stays cacheable for the whole segment; following FastWAM, action and progress tokens do not attend to future visual tokens and vice versa, making future visual prediction an auxiliary training objective that can be dropped at inference.
All three branches share the flow-matching condition $c=(\text{G}_{k},\text{l}_{k},\text{f}_{t},\text{s}_{t},\text{p}_{t})$:
$$\mathcal{L}_{\mathcal{E}}^{m}=\mathcal{L}_{\mathrm{FM}}\left(y_{m},c\right),\quad m\in\{v,a,p\}$$
and the total loss is $\mathcal{L}_{\mathcal{E}}=\lambda_{v}\mathcal{L}_{\mathcal{E}}^{{v}}+\lambda_{a}\mathcal{L}_{\mathcal{E}}^{{a}}+\lambda_{p}\mathcal{L}_{\mathcal{E}}^{{p}}$ with $\lambda_{v}=\lambda_{a}=\lambda_{p}=1.0$ in experiments; the action and progress branches share one sampled flow timestep while the future video branch uses an independent one. The progress condition $\text{p}_{t}$ plays a concrete role: an explicit temporal anchor that disambiguates visually similar observations belonging to different semantic stages, such as pressing versus releasing, without widening the observation window.
Plan-observation alignment for progress calibration. Ground-truth progress is unavailable at deployment, so the progress condition is updated recursively from WAP's own predictions and drifts over long horizons. The visual plan supplies a temporally indexed reference: at the start of each segment the reference sequence is the segment's initial observation plus the eight generated plan frames, with progress increasing linearly from 0 to 1 across the nine frames. Before each action-chunk inference, the two reference frames nearest to the current estimate $\text{p}_{t}$ are compared with the observation obtained after executing the current chunk using mean absolute pixel difference in RGB, and the progress condition is set to the average of $\text{p}_{t}$ and the progress index of the visually closer reference frame. The metric is training-free and cheap, yet it anchors execution back to the planned visual evolution.
Progress-gated segment transition. MaP-WAM averages predicted progress over the latest actions to obtain a completion score $s$; once $s$ exceeds the threshold $\tau=0.95$, execution of the current plan terminates. The real execution observations are uniformly resampled into sparse visual context $\text{G}_{k}$, which replaces the generated visual plan in the appended record $\{\text{l}_{k},\text{G}_{k}\}$, keeping memory grounded in real observations rather than generated predictions, and planning is invoked for the next segment.
flowchart TD
A["Episodic context C_1..k-1
language records + 8-frame sparse visuals"] --> B["VLM language planner Qwen3.5-4B
predict segment plan l_k"]
A --> C["Causal world model CWM
WAN-2.2-5B flow matching"]
B --> C
C --> D["WAP executor on fixed plan prefix
joint action + progress denoising"]
D --> E["Plan-observation alignment
pixel-diff calibrates p_t"]
E --> D
D --> F{"mean progress score above tau = 0.95 ?"}
F -- "no: execute next action chunk" --> D
F -- "yes: segment complete" --> G["Resample real observations
append record l_k + G_k to memory"]
G --> A
The diagram mirrors Algorithm 1: planning is invoked once per segment transition while WAP is queried repeatedly inside a segment; the plan prefix is prefilled and cached at segment start and refreshed only when the progress gate triggers a new plan.
Experiments
Figure 1: language memory loses fine-grained evidence and growing windows grow expensive with context; MaP-WAM keeps sparse visual memory while the executor context stays fixed.
Simulation on RMBench. RMBench is a simulation benchmark for long-horizon memory-dependent manipulation with five M(1) tasks (decisions depend on one past observation) and four M(n) tasks (multiple). Following the benchmark protocol, MaP-WAM is trained with 50 official expert demonstrations per task and evaluated over 100 rollouts per task with global seed 0; the planners are trained multi-task while WAP is trained single-task following MEM-0. The table below shows MaP-WAM reaching an 83.3% average success rate, above the strongest baseline LingBot-VA at 77.1%, while Mem-0 reaches 42.0% and the memory-free DP, pi-0.5, and X-VLA sit around 10%.
| Task | TMC | DP | pi0.5 | X-VLA | Mem-0 | WLA-0 | LingBot-VA | MaP-WAM |
|---|---|---|---|---|---|---|---|---|
| Observe and Pick Up | M(1) | 1% | 9% | 9% | 4% | - | 3% | 19% |
| Rearrange Blocks | M(1) | 0% | 13% | 13% | 89% | - | 100% | 66% |
| Put Back Block | M(1) | 0% | 11% | 18% | 90% | - | 100% | 100% |
| Swap Blocks | M(1) | 11% | 24% | 16% | 67% | - | 99% | 97% |
| Swap T | M(1) | 20% | 15% | 3% | 14% | - | 88% | 96% |
| Battery Try | M(n) | 10% | 16% | 26% | 28% | 45% | 41% | 82% |
| Blocks Ranking Try | M(n) | 10% | 6% | 1% | 18% | 23% | 100% | 94% |
| Cover Blocks | M(n) | 0% | 0% | 2% | 68% | 84% | 79% | 100% |
| Press Button | M(n) | 0% | 0% | 0% | 0% | 74% | 84% | 96% |
| Average | - | 5.8% | 10.4% | 9.8% | 42.0% | - | 77.1% | 83.3% |
Two groups of tasks carry most of the signal. Press Button and Swap T require distinguishing visually near-identical press/release phases and remembering the initial positions of two T-shaped blocks; MaP-WAM scores 96% on both, exactly where the progress condition and sparse visual memory are supposed to bite. Observe and Pick Up demands discriminating among dozens of object types while jointly reasoning over temporally separated observations and spatial relations from only 50 demonstrations; MaP-WAM lifts the best baseline from 9% to 19%, still low in absolute terms but a clear directional gain. Conversely, on Rearrange Blocks MaP-WAM (66%) trails LingBot-VA (100%) and Mem-0 (89%); the task has weak memory demands, and a fixed plan prefix is a constraint relative to full-context conditioning.
Figure 2: overview. Memory-grounded planning produces the language and visual plan; WAP predicts actions and progress on a cached plan prefix; plan-observation alignment calibrates progress; real observations are written back to memory at segment transitions.
Real robot on Franka Research 3. Real-world experiments use a 7-DoF Franka Research 3 arm with third-person and wrist-mounted RealSense D435i cameras. In Find Button the robot first observes button colors, the buttons are then occluded by covers, and a human presents a color instruction on a whiteboard; the robot must open the matching cover. In Press Buttons the robot reads two numbers on a whiteboard, presses the left and middle buttons the indicated number of times, then presses the right confirmation button. With 50 trajectories per task for training and 50 independent trials per task for evaluation, MaP-WAM reaches 88% and 68% (78.0% average); baselines score non-zero on Find Button only through random selection and fail entirely on Press Buttons.
| Real-robot task | pi0.5 | FastWAM | MaP-WAM |
|---|---|---|---|
| Find Button | 26% | 20% | 88% |
| Press Buttons | 0% | 0% | 68% |
Ablation 1: visual planning. On Observe and Pick Up and Swap T the paper compares two variants: w/o visual plan removes the CWM and conditions WAP only on the language plan; w/o visual memory replaces the CWM with a standard current-observation-conditioned world model. The former drops to 2% and 14%, the latter to 1% and 13%, against 19% and 96% for the full model. The qualitative comparison shows that without visual memory the planner generates plans with wrong object identities, colors, or positions: visual guidance must be grounded in long-horizon episodic evidence, not the current frame.
Figure 3: visual planning ablation. With visual memory the CWM produces plans with correct objects and spatial configuration; a current-observation world model produces wrong objects, colors, or positions.
Ablation 2: progress modeling. On three harder M(n) tasks the paper compares three variants. Replacing progress with a binary completion classification token yields 37.0% on average. Keeping progress prediction but removing the current progress $\text{p}_{t}$ from the conditioning inputs collapses Press Button from 96% to 18% (the executor can no longer separate pressing from releasing) and averages 54.0%. Keeping recursive progress but disabling plan-observation alignment hurts Blocks Ranking Try most, dropping 94% to 38%, because that task averages 423.10 steps for the swap subgoal versus 116.32 steps elsewhere and autoregressive drift accumulates; the average is 73.7%. The full model reaches 96.7%, isolating the three roles of progress: modality, condition, and closed-loop calibration.
| Progress design | Blocks Ranking | Cover Blocks | Press Button | Average |
|---|---|---|---|---|
| classification | 40% | 71% | 0% | 37.0% |
| w/o progress condition | 57% | 87% | 18% | 54.0% |
| w/o progress calibration | 38% | 85% | 98% | 73.7% |
| MaP-WAM | 94% | 100% | 96% | 96.7% |
Inference efficiency. On the planning side, sparse visual context retains only two latent timesteps per completed segment after WAN-VAE compression, and block-causal attention makes completed evidence a cacheable prefix: planner latency stays around 450-470 ms with KV caching even after 16 consecutive segments, versus roughly 720 ms without, and planning is invoked only at segment transitions. Execution dominates latency. Against a Full Context variant that conditions the same executor on the growing frame-wise history, uncached full-context latency grows rapidly; with KV caching it still costs about 4x the zero-history latency at 1,500 frames and exceeds 80 GB of GPU memory at 1,700 frames, causing an out-of-memory failure. WAP caches a fixed plan prefix and holds an approximately constant 827 ms per action chunk across the same range. The memory capability is bought without paying execution efficiency, which is the hardest selling point against growing-window designs.
Figure 4: latency. (a) visual planner latency versus completed segments; (b) per-chunk executor latency versus history frames, where full context runs out of memory at 1,700 frames while WAP stays near 827 ms.
Limitations
Stated by the authors, first: MaP-WAM builds on the segment structure provided by existing benchmarks to organize memory and progress; RMBench annotations determine training-time segment boundaries, and extending planning to unsegmented demonstrations via automatic segment discovery is left as future work.
Stated by the authors, second: plan-observation alignment uses a lightweight training-free matching metric (RGB pixel difference); learned similarity measures may improve calibration robustness in visually complex scenes.
Additional assessment: WAP is trained single-task following the MEM-0 protocol, so the executor is not shared across tasks and the paper does not separate the gain from memory structure from the gain of per-task fitting; on Rearrange Blocks the fixed plan prefix clearly underperforms LingBot-VA and Mem-0, and Observe and Pick Up remains at 19% absolute, showing costs on weak-memory and fine-discrimination tasks; the real-robot study covers two tasks with 50 demonstrations each, limiting how far 78.0% generalizes; and the language planner sees only the compact keyframe set $\text{f}^{\star}$, which may miss mid-segment visual details a future plan depends on.
Conclusion and Outlook
MaP-WAM's contribution is a reassignment of responsibilities rather than a stronger backbone: memory serves planning, planning produces a plan, the plan conditions execution, and progress closes the loop among the three. The split resolves the two classic tensions of memory mechanisms, language memory losing visual evidence and growing windows losing execution efficiency, at the cost of depending on segment structure and progress annotation. For practitioners, a constant executor context plus KV caching on both planning and execution is a viable route to bring long-horizon memory into real-time control. For researchers, automatic segment discovery and learned plan-observation similarity are the two extensions the authors themselves name. Once code and checkpoints follow the release plan, reproducing 83.3% on RMBench and 78.0% on the real robot will be the first test of this memory paradigm.
Golden Line
"Don't replay the past. Plan from it." The project page's one-line statement of the core idea: the value of history lies not in being watched again, but in being converted into a plan for what to do next.
