When to Plan: Learning to Select Between Reactive Control and Deliberative Planning

Paper: When to Plan: Learning to Select Between Reactive Control and Deliberative Planning
Link: arXiv:2607.16421v1, 2026 (RLC 2026)
Affiliation: Prediction and Action Lab (PAL), University of Wisconsin–Madison
Code: ❌ No public code released

1. Abstract

This paper studies the meta-reasoning problem: how an RL agent can learn to dynamically allocate computation between a fast reactive policy and a slower decision-time planner. The reactive policy produces an action from a single forward pass but only performs well within its training distribution; the planner produces good action sequences from any state but at additional computational cost. The authors formalize this as a meta-MDP using the options framework, training a meta-policy that selects between reactive action and variable-depth planning based on uncertainty signals from the reactive policy. Across 5 motion planning and navigation tasks, the adaptive meta-policy outperforms all fixed-compute baselines (always-react, always-plan) in time to reach goal states. Ablations confirm that reactive uncertainty and observation history are the most critical components.

2. Background and Motivation

Agents in large, open-ended environments face a core tradeoff: fast reactive control vs slow deliberative planning. Reactive policies (single forward pass) are cheap but distribution-constrained; planners (world-model-based) produce near-optimal actions across the full state space but are computationally expensive. Planning is most valuable in states outside the reactive policy's training distribution.

Limitations of existing methods:

• Event-triggered replanning: Assumes agents always plan, only learning when plans become invalid, not considering computation minimization for faster task completion;
• Variable-length options: Optimize action efficiency rather than computation allocation, without explicitly modeling the reactive-planning cost tradeoff;
• Model-free/model-based control arbitration: Relies on hand-designed switching rules or uncertainty heuristics, ignoring model-based planning computation; or assumes equal inference time for planning and reactive control.

Cognitive science (dual-process theories, bounded rationality) suggests humans deliberate when expected value exceeds cost, directly motivating the formulation of planning as a time-costly option.

3. Core Method

3.1 Meta-MDP Formalization

The meta-reasoning problem is formalized as a meta-MDP: $\tilde{\mathcal{M}}=(\tilde{\mathcal{S}},\mathcal{O},\tilde{\mathcal{P}},\tilde{r},\gamma)$, where $\tilde{\mathcal{S}}$ is the meta-state space, $\mathcal{O}$ is the options space ($o\in\mathcal{O}$ is a possibly temporally-extended choice action), $\tilde{\mathcal{P}}$ is the meta-transition function, $\tilde{r}$ is the meta-reward, and $\gamma$ is the discount factor. The meta-state space need not correspond directly to the world-environment state space $\mathcal{S}$.

Two types of options:

• Planning option $\pi_k$: Uses the planner to generate a horizon-$k$ plan $p=[a_0,a_1,\dots,a_{k-1}]$, executed open-loop. Planning cost $c_p(k)=-\alpha k$ ($\alpha$ = per-step planning time penalty), execution cost $c_a=-1$ (per step), total reward $\tilde{r}_{\tilde{t}}=c_p(k)+k\cdot c_a=-\alpha k-k$.
• Reactive option $\pi_r$: $c_p=0$, total reward $\tilde{r}_{\tilde{t}}=c_a=-1$. Near-optimal in-distribution $\mathcal{S}_{ID}$: $V^{\pi_r}(s)\approx V^*(s)$, but far worse out-of-distribution $\mathcal{S}_{OOD}$: $V^{\pi_r}(s)\ll V^*(s)$.

3.2 Meta-Observation Design

The meta-policy operates on an internal state $\tilde{s}$ computable without search. The observation concatenates three components:

$$\hat{s}_{\tilde{t}}=[u_{\tilde{t}},d_{\tilde{t}},\tilde{a}_{\tilde{t}-1}]$$

where $u_{\tilde{t}}$ is the reactive policy uncertainty score, $d_{\tilde{t}}$ is current distance to goal, and $\tilde{a}_{\tilde{t}-1}$ is the previous meta-action. An $n$-step history is also provided:

$$\tilde{s}_{\tilde{t}}=[\hat{s}_{\tilde{t}},\hat{s}_{\tilde{t}-1},\dots,\hat{s}_{\tilde{t}-(n-1)}]$$

The action space includes a reactive option and N planning options (different horizons):

$$\mathcal{O}=\{\pi_{\mathrm{r}},\pi_{k_{1}},\pi_{k_{2}},\dots,\pi_{k_{N}}\}$$

3.3 Reactive Uncertainty: Ensemble Method

The reactive policy is implemented as an ensemble of $M$ neural networks. For continuous control, each network outputs an action mean $\mu_i\in\mathbb{R}^{D}$; the ensemble action and uncertainty are:

$$a=\frac{1}{M}\sum_{i=1}^{M}\mu_{i},\qquad u(s)=\sum_{j=1}^{D}\mathrm{Var}_{i}\!\left[\mu_{i,j}\right]$$

For discrete action spaces, each network outputs a categorical distribution $\mathbf{p}_i$, averaged $\bar{\mathbf{P}}=\frac{1}{M}\sum_{i=1}^{M}\mathbf{p}_i$; the ensemble action and uncertainty are:

$$a=\arg\max_{j}\,\bar{P}(j),\qquad u(s)=-\sum_{j=1}^{J}\bar{P}(j)\log\bar{P}(j)$$

High uncertainty corresponds to out-of-distribution inputs where planning may be beneficial.

flowchart TD
    A[Environment State s] --> B[Reactive Policy Ensemble
M=4 Neural Networks]
    B --> C[Action a_r + Uncertainty u_t]
    C --> D[Meta-Observation s_tilde
= u_t, d_t, a_prev + History]
    D --> E[Meta-Policy π_Ω
PPO Trained]
    E --> F{Select Option}
    F -->|Reactive π_r| G[Execute Reactive Action One Step]
    F -->|Planning π_k| H[A* Planner
horizon=k]
    H --> I[Execute k-step Plan Open-Loop]
    G --> J[Environment Step]
    I --> J
    J --> A

3.4 Training Procedure

• Meta-policy: PPO (Stable Baselines3, default hyperparameters), 4-step observation history;
• Reactive policy: Pretrained by behavior cloning from planner-generated trajectories on in-distribution tasks, $M=4$ networks;
• Planner: A* planner with a given accurate world model, constrained to maximum horizon;
• Joint-training setting: State-action pairs from planning options recorded to replay buffer; after each rollout, ensemble members trained by behavior cloning; meta-policy tracks reactive policy improvement.

4. Key Experiments

4.1 Environment Suite

5 environments, 20 tasks each (half in-distribution, half OOD), 30 seeds per task, 300 evaluation episodes per seed:

Figure 1: Environment suite. Box Push, DoorKey, Fetch, Maze, Navigation — varying in state space type, task structure, and reactive policy difficulty.

• Box Push: Simplified Sokoban, 6×6 grid, 3 boxes, irrecoverable incorrect moves, symbolic observations;
• DoorKey: MiniGrid multi-stage task (pickup key → unlock door → reach goal), symbolic observations;
• Fetch: Modified Gymnasium Robotics Fetch Reach with obstacles, 7-DOF arm joint-space control, planner operates in Cartesian space;
• Maze: Grid maze, 5 start/goal pairs per obstacle configuration, image observations;
• Navigation: Continuous navigation, continuous displacement control, image observations.

Left: Box Push environment (simplified Sokoban, irrecoverable errors); Right: Fetch environment (7-DOF arm with obstacles).

Left: Maze environment (grid maze); Right: Navigation environment (continuous navigation).

4.2 Main Results: Meta-Policy vs Fixed-Compute Baselines

Average episode return (includes acting and planning cost; higher is better):

The meta-policy achieves the best (or within CI of best) in all 5 environments. Box Push has irrecoverable errors — the meta-policy learns to avoid reactive control and short plans; Fetch converges to >75% reactive control, suggesting partial generalization but OOD still needs planning.

4.3 Ablation: Meta-Observation Components

Removing one observation component at a time (average episode return):

History and Uncertainty are the most critical components. Removing History degrades Box Push 4.5× and Fetch 1.7× — without progress tracking, it's unclear if the reactive agent moves toward the goal. Removing Uncertainty also drops Box Push and Fetch significantly — without directly determining in/out-of-distribution, the meta-agent cannot perform optimally.

4.4 Environment Characteristics Influence

Analysis by varying three environment characteristics:

• In-distribution task proportion: As in-distribution proportion increases, the meta-policy shifts toward reactive control, with reactive action proportion roughly tracking in-distribution proportion. At all-in-distribution, converges to fully reactive — correctly interpreting reactive competence.
• Planning cost $\alpha$: As $\alpha$ increases, meta-policy relies more on reactive control. Even though reactive policy produces suboptimal OOD actions, time saved by avoiding planning suffices for better returns in some cases. Short/medium/long plan ratios remain roughly constant — the meta-policy learns when to plan, not how long to plan.
• Environment stochasticity $\epsilon$: No noise to low noise ($\epsilon=0.1$): reactive percentage rises from ~45% to ~70% (long planning drops from ~20% to ~5%). Long plans are most affected by noise since errors compound over plan duration. At high noise, short/medium planning slightly increases — still useful for OOD states.

4.5 Joint Training: Tracking Improving Reactive Policy

Joint training on Box Push, DoorKey, Maze (Fetch, Navigation excluded due to compute). Planning-option state-action pairs added to replay buffer; after each rollout, ensemble members trained by behavior cloning. As training progresses:

• Meta-policy shifts toward reactive control across all environments, converging to fully reactive execution;
• Reactive uncertainty signal converges to low disagreement — ensemble uncertainty reflects reactive policy strength;
• This creates a learning curriculum: the meta-policy calls planning in OOD states (best immediate reward + generates training data), the reactive policy trains on OOD states, uncertainty decreases, and the meta-policy recognizes improved competence and stops calling planning.

5. Limitations and Future Work

• Perfect world model assumption: The method assumes the planner uses an accurate world model, which rarely holds in real-world settings. Cognitive science suggests humans account for model uncertainty when deciding whether to plan; extending to learned world models with state uncertainty is a natural next step.
• Single-agent static environments: Environments limited to single-agent and static during plan creation. Extending to dynamic environments and interruptible plans would test generalization of the reactive-planning tradeoff.
• Hand-designed features: Meta-observation relies on hand-designed features. More expressive uncertainty estimates (Bayesian or flow-based) or learned observation representations could improve high-dimensional generalization.
• Tracking worsening reactive policy: The meta-policy can track improving reactive policies, but whether it can track worsening ones (insufficient capacity or catastrophic forgetting) is an open question.
• Broader applications: The reactive-planning tradeoff is structurally similar to LLM fast single-pass generation vs slow chain-of-thought reasoning, and VLA fast lightweight (robot) vs large slow (cloud) models.

6. Conclusion

This paper presents an RL method for learning adaptive computation allocation through a meta-reasoning policy. The approach conditions on reactive policy uncertainty and high-level state features to select between a fast reactive option and planning options — where planning improves action quality at the cost of time. Across a suite of motion planning and navigation tasks, the meta-policy outperforms all fixed-compute heuristic baselines when optimizing for time-to-goal. Ablations confirm that reactive uncertainty and observation history are the most critical components of the meta-state. The meta-policy adapts to environment characteristics: increasing planning cost shifts allocation toward reactive control, and increasing in-distribution coverage shifts allocation away from planning. Finally, conditioning on uncertainty signals enables the meta-policy to converge to exclusively reactive control in a joint-training setting where the reactive policy continues to learn. These results show that high-level state and reactive competence signals are sufficient for a meta-reasoning policy to effectively allocate computation across environments and training regimes.