Skip to content
RobotWorld
Back to Papers

PAPER DEEP DIVE

Paper世界模型World Model

The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation

Multi-turn long-horizon planning is critical for foundation model agents, yet how to fundamentally improve it remains unclear. Existing models are trained on uncontrollable and opaque Internet data, making it difficult to identify how planning ability is acquired, shaped, and integrated. To address this challenge, we introduce a unified and controlled multi-turn environment that enables precise control. It allows systematically study long-horizon planning across three stages. (1) Planning ability acquisition during pre-training. We study data format, distribution, and quality. Explicit world model construction through CoT state transition modeling yields stronger long-horizon generalization. Atomic skills alone are insufficient for compositional generalization, whereas a litte long-horizon data works. Moreover, suboptimal trajectories severely impair performance because errors amplify over long horizons. (2) Planning ability shaping via GRPO and OPD post-training. Through mutual information, we distinguish general planning patterns from task-specific planning knowledge. For planning patterns, we identify three application regions of post-training: unnecessary, effective, and unsupported. OPD has a broader effective region than GRPO under low-quality and long-horizon settings, as it provides more consistent update directions. For planning knowledge, distilling unseen procedures from a teacher with different knowledge may impair student's prior world modeling without fully establishing new knowledge. (3) Planning ability integration through MOPD post-training. We show that multi-teacher on-policy distillation (MOPD) integrates capabilities by converging to shared planning-pattern across environments. Compatible patterns enable cross-environment generalization, partially shared patterns support continual learning, while completely conflicting patterns cause severe interference.

Tianyi Men, Zhuoran Jin, Kang Liu, Jun ZhaoJuly 27, 20266 min read
中文

The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation

Institutions: CASIA, UCAS  |  arXiv: 2607.24720v1


One-Sentence Summary

This work proposes a unified controlled multi-turn environment to systematically study how long-horizon planning ability is acquired (pre-training), shaped (GRPO/OPD post-training), and integrated (MOPD multi-teacher distillation), revealing the roles of world model internalization, mutual information differences between planning patterns and knowledge, and multi-teacher compatibility for cross-environment generalization.


Background and Motivation

As foundation models advance toward agentic systems, multi-turn long-horizon planning becomes critical. Unlike single-turn QA, long-horizon planning requires decomposing complex tasks, tracking environment states, modeling state transitions, and composing atomic skills through long-horizon interactions. Existing benchmarks (OSWorld 2.0, EdgeBench, DeepPlanning) show models have basic short-horizon capability but struggle significantly with long-horizon tasks.

The core problem: existing models are trained on uncontrollable, opaque Internet data, making it difficult to identify how planning ability is acquired, shaped, and integrated. This work poses three progressive questions across three training stages.

Figure 1: Research overview

Figure 1: Three-stage research framework — large-scale pre-training, RL post-training (OPD/GRPO), and multi-teacher consolidation (MOPD).

Controlled Planning Environment

A unified controlled multi-turn environment supports precise control over environment numbers, levels, task length, data quality, planning knowledge, and planning patterns. Skill graph construction converts item composition rules (AND/OR logic) into DAGs, mapped to planning tasks with difficulty controlled by search depth and branching factor.

Figure 2: Controlled environment

Figure 2: Controlled multi-turn environment — skill graph construction, graph-to-Gym mapping, difficulty control.

Stage 1: Planning Ability Acquisition during Pre-training

World Model Internalization

The world model is defined by a state transition function:

$$\mathcal{T}: \mathcal{S} \times \mathcal{A} \rightarrow \mathcal{S}$$

Given state $s_t$ and action $a_t$, it predicts $s_{t+1} = \mathcal{T}(s_t, a_t)$. Skill internalized world modeling plans via DFS over an internal simulated state space. Each atomic skill $k \in \mathcal{K}$ is a transition function $f_k: \mathcal{S} \rightarrow \mathcal{S}$:

$$s_{t+1} = f_{k_{t,m_t}} \circ \cdots \circ f_{k_{t,1}}(s_t)$$

Models with world models (w/ WM) consistently outperform direct answering baselines (w/o WM) across all difficulty levels. Chain-of-thought state transition modeling yields stronger long-horizon generalization than direct action prediction.

Figure 3: WM vs no-WM performance

Figure 3: Final performance comparison between students with and without world modeling.

Atomic Skill Compositional Generalization

Models trained only on atomic skills cannot achieve compositional generalization. Adding a small amount of long-horizon data substantially improves planning ability, showing that long-horizon composition requires explicit long-horizon training signals.

Impact of Suboptimal Trajectories

Suboptimal trajectories severely impair performance due to error accumulation. With per-step error rate $\epsilon$ over $T$ steps:

$$P_{\text{fail}}(T) = 1 - (1-\epsilon)^T \approx 1 - e^{-\epsilon T}$$

Even small $\epsilon$ causes $P_{\text{fail}} \to 1$ as $T$ grows, explaining why suboptimal data is particularly harmful for long-horizon planning.

Stage 2: Planning Ability Shaping via GRPO and OPD

Mutual Information Framework

Planning patterns ($P$) and planning knowledge ($K$) are distinguished via mutual information:

$$I(T;P) = H(P) - H(P|T), \quad I(T;K) = H(K) - H(K|T)$$

Since $P$ is abstract and reusable (reflection, backtracking), $I(T;P)$ is low. Since $K$ is task-specific, $I(T;K)$ is high. Thus $I(T;P) < I(T;K)$, which profoundly affects post-training effectiveness.

Planning Patterns: RL Applicability Regions

RegionCharacteristicRL Effect
A: UnnecessarySimilar performance across patternsRL selection pointless
B: EffectiveClear performance differencesRL discovers better patterns
C: UnsupportedBetter patterns exist but RL can't find themRL may fail

GRPO has a narrower effective region. Under low-quality data and long horizons, OPD has a broader effective boundary because RL's sparse credit assignment introduces both correct and incorrect gradients, while OPD provides consistent update directions when the teacher is ideal.

Figure 5: GRPO vs OPD applicability

Figure 5: Applicability boundaries and capability upper bounds of multi-turn GRPO and OPD.

Planning Knowledge: Distillation Limitations

When teacher and student have different procedural knowledge, direct OPD distillation of unseen multi-path knowledge cannot guarantee effective transfer. The student may enter an intermediate state where old knowledge is impaired while new knowledge is not fully established:

$$\Delta\theta_{\text{OPD}} \propto \nabla_\theta \sum_{k,t} D_{\text{KL}}(p_{k,t} \| q_{k,t}^{\text{teacher}})$$

Stage 3: MOPD Multi-Teacher Integration

Multi-teacher on-policy distillation integrates capabilities via sequential training. The $m$-th teacher's loss:

$$\mathcal{L}_{\text{Agent-OPD}}^{(m)}(\theta) = \mathbb{E}_{\tau \sim \pi_\theta} \left[ \sum_{k=1}^{K} \sum_{t=1}^{T_k} D_{\text{KL}}(p_{k,t} \| q_{k,t}^{(m)}) \right]$$

MOPD integrates by converging to shared planning-pattern distributions. Three cases: compatible patterns enable cross-environment generalization, partially shared support continual learning, and fully conflicting cause catastrophic forgetting.

Token-level KL divergence measures student-teacher closeness:

$$\text{KL}^{\sigma,\gamma}(\tau) = \frac{1}{\sum_{k=1}^{K} T_k} \sum_{k=1}^{K} \sum_{t=1}^{T_k} \sum_{v \in \mathcal{V}_{k,t} \cup \{\text{tail}\}} P_{k,t}^{\sigma}(v) \log \frac{P_{k,t}^{\sigma}(v)}{Q_{k,t}^{\gamma}(v)}$$ Figure 6: MOPD framework

Figure 6: MOPD analysis framework — non-shared and conflicting planning patterns across environments.

graph TD
    A["Pre-training
Ability Acquisition"] --> B["World Model Internalization"] A --> C["Atomic Skill Composition
Needs long-horizon data"] A --> D["Trajectory Quality
Suboptimal data harmful"] B --> E["RL Post-training
Ability Shaping"] C --> E D --> E E --> F["Planning Pattern P
Low mutual information"] E --> G["Planning Knowledge K
High mutual information"] F --> H["OPD broader effective region
GRPO limited by sparse credit"] G --> I["Distilling unseen knowledge
May impair existing modeling"] H --> J["MOPD Multi-Teacher Integration"] I --> J J --> K["Compatible: Cross-env generalization"] J --> L["Partial: Continual learning"] J --> M["Conflicting: Catastrophic forgetting"]

Key Findings

FindingKey ConclusionEvidence
World modelw/ WM consistently outperforms w/o WMavg@8 and pass@8 across 3 difficulties
Compositional gen.Atomic skills insufficient, need long-horizon dataSmall amount of LH data helps substantially
Suboptimal data4 Opt:8 Sub much worse than 4 OptError accumulation amplification
OPD vs GRPOOPD better under low-quality data + long horizonMore consistent gradient directions
Knowledge distillationUnseen knowledge distillation may impairKL dynamics show intermediate state
MOPD compatibilityCompatible → generalize, conflicting → forgetCross-environment success rates

Limitations

  1. Experiments use synthetic controlled environments, with a gap to real Internet data distribution
  2. Model scale is 100M parameters; conclusions may differ at larger scales
  3. Knowledge distillation intermediate-state analysis relies mainly on KL divergence
  4. MOPD conflict detection and resolution mechanisms need further study

Conclusion and Outlook

This work systematically studies multi-turn long-horizon planning across three stages: pre-training requires world model internalization, long-horizon data, and high-quality trajectories; RL post-training shows OPD has a broader effective region for planning patterns but knowledge distillation has limitations; MOPD integrates via shared distribution convergence, with compatible patterns enabling generalization and conflicting causing forgetting. These findings provide practical guidance for developing stronger agentic foundation models.

Key Insight: Errors in long-sequence decision-making accumulate and amplify — even with low per-step error rate, the overall failure rate of long-horizon trajectories approaches 1.

Related Papers

Pre-training Visual Dexterity in Simulation

Pre-training Visual Dexterity in Simulation

Large-scale pre-training has made robot policy fine-tuning increasingly data-efficient, but this progress has largely been driven by datasets and embodiments built around simple parallel-jaw grippers. Dexterous, multi-fingered hands remain comparatively data-starved because real teleoperation is costly to scale, while human hand video is off-embodiment and requires lossy pose estimation and retargeting. We introduce Simulation Pre-training for Dexterity (SPD), a pre-training framework for dexterous manipulation that uses data entirely collected in simulation. In SPD, humans manipulate virtual objects inside a VR headset, enabling on-embodiment trajectories and robot-free collection. With the help of five operators, we collect 75 hours of multi-task dexterous manipulation over one week, and use it to pre-train a causal transformer on a sequence modeling objective. We study the benefits of simulation pre-training on real-world tasks by fine-tuning on 1-2 hours of physical demonstrations on a 56-DoF bimanual dexterous setup. We find that our approach outperforms training behavior cloning policies from scratch, showing that simulation teleoperation is a viable pre-training source for real-world dexterous manipulation. We perform ablation studies, measuring the benefits of history conditioning and short action chunks for reactive control.

灵巧操作灵巧手预训练Aug 16, 2026
FA-RDP: A Frequency-Adaptive Reactive Diffusion Policy for Contact-Rich Manipulation

FA-RDP: A Frequency-Adaptive Reactive Diffusion Policy for Contact-Rich Manipulation

In contact-rich manipulation, action multimodality and reactivity dominate different stages of a single episode. Before contact, multiple trajectories might be equally valid, making it important to preserve diverse action modes. After contact, geometric constraints and force limits narrow the solution space, while successful execution demands rapid responses to force feedback. However, standard diffusion policies use a fixed inference frequency and sampling steps throughout the episode, forcing a fundamental compromise: low-frequency, multi-step sampling better preserves pre-contact multimodality but responds slowly to force feedback, whereas high-frequency sampling improves reactivity but tends to collapse distinct pre-contact modes. To resolve this tradeoff, we present FA-RDP, a frequency-adaptive reactive diffusion policy. A shared multi-frequency visual-force Transformer predicts action chunks at both low and high frequencies, while a learned multimodality indicator dynamically selects multi-step low-frequency sampling before contact and one-step high-frequency sampling as action ambiguity decreases. We further introduce Manifold Consistency Distillation (MCD), which reparameterizes the diffusion network to predict actions on the robot action manifold while retaining DDPM-based residual supervision. Experiments on three contact-rich manipulation tasks show that FA-RDP achieves the highest success rate while preserving diverse pre-contact trajectory modes. Code and videos are available at https://fa-rdp.github.io.

PaperJul 30, 2026
PhiZero: A World Model Built Around Physical Language

PhiZero: A World Model Built Around Physical Language

We introduce PhiZero, a physical world model built around physical language, a compact discrete representation of world-state transitions. Existing physical world models typically predict future videos directly in pixel space, leaving the underlying world dynamics implicit within high-dimensional visual predictors. Motivated by humans' ability to abstract predictive structure from visual experience and organize it in natural language for explicit reasoning, we learn physical language from in-the-wild videos through self-supervision and use it to explicitly reason about how the physical world evolves. Accordingly, PhiZero adopts a reason-then-render paradigm: it first infers future world evolution as a physical-language sequence and then renders the inferred transitions into videos. Extensive experiments across generation and understanding benchmarks validate the ability of PhiZero to model physically coherent world evolution. We further show its potential for realistic and interactive world modeling, fine-grained action-conditioned simulation, and zero-shot motion transfer.

PaperJul 30, 2026
Booster Lab: A Data-Centric Pipeline for Learning Deployable Humanoid Locomotion Policies

Booster Lab: A Data-Centric Pipeline for Learning Deployable Humanoid Locomotion Policies

Humanoid robot motion learning requires not only task-oriented control policies but also physically feasible and natural behaviors that can be transferred to real robots. However, robot-feasible motion data are often scarce: raw human demonstrations may be incompatible with the robot morphology, open-source clips vary in quality, and simulation-collected robot trajectories still require feasibility checking. To address these challenges, we propose a data-centric training and deployment pipeline that integrates motion data curation, real-to-sim model adaptation, AMP-based reinforcement learning, and sim-to-real deployment. We validate the framework on the Booster T1 robot and further provide preliminary cross-platform validation on Booster K1.

步态优化人形机器人AMPJun 26, 2026