Offline RL with Hierarchical Action Chunking

1. Paper Overview

This paper proposes HiQC (Hierarchical Implicit Q-Chunking) — an offline goal-conditioned RL algorithm that combats the "curse of horizon" through dual temporal decomposition. In offline goal-conditioned RL, learning general-purpose policies from static datasets is a promising goal, but scaling to long-horizon tasks remains challenging — value estimation errors in TD learning compound through long Bellman bootstrap chains, making value estimates for distant goals unreliable.

HiQC's core idea combines high-level latent planning with low-level action chunking: the high-level planner predicts subgoals in latent space, while the low-level controller executes action chunks of length $k$. By conditioning the low-level critic on full action chunks for unbiased $k$-step value backups, HiQC compresses the horizon at both planning and execution levels. On the OGBench benchmark, HiQC achieves the highest aggregate performance among all compared methods, with its largest gains on long-horizon navigation tasks like humanoid-giant.

2. Problem Background and Motivation

Offline RL derives optimal policies from static datasets, avoiding expensive online interaction. But the major obstacle is the curse of horizon: popular offline RL algorithms rely on TD learning, minimizing the error between value estimates and bootstrapped Bellman targets. In the offline setting, extrapolation errors on out-of-distribution actions introduce bias, and as task horizon increases, these biases compound through long recursive Bellman backup chains, rendering value estimates for distant goals unreliable.

Existing approaches each have shortcomings:

• n-step returns can mitigate bias accumulation, but as $n$ grows, variance also increases — especially in stochastic environments or when the data distribution deviates from the policy.
• Hierarchical RL (HRL) reduces horizon by decomposing tasks into high-level subgoal sequences, but standard hierarchies rely on low-level policies executing single actions to reach subgoals, leaving the low-level policy susceptible to the curse of horizon over the subgoal duration.
• Action chunking generates temporally coherent action sequences in imitation learning, and within TD learning allows the critic to perform unbiased n-step backups over the chunk length, but only reduces the horizon by a constant factor, which may be insufficient for long-horizon tasks.

HiQC's key insight: neither hierarchy alone nor chunking alone is sufficient — a dual horizon reduction is needed.

3. HiQC Method in Detail

3.1 Two-Level Architecture

HiQC employs a two-level temporal decomposition:

graph TB
    A["Task goal g"] --> B["High-level planner
Latent space planning"]
    B --> C["Subgoal z
every c steps"]
    C --> D["Low-level controller
Action chunking"]
    D --> E["Action chunk a
length k steps"]
    E --> F["Environment"]
    F --> D
    F --> B

The high-level planner predicts subgoals $z$ in latent space every $c$ steps, while the low-level controller executes action chunks of length $k$ to reach the subgoal. The key design is that the low-level critic performs unbiased $k$-step value backups over full action chunks, rather than single-step backups.

3.2 High-Level Value Function

The high-level value function is trained via Implicit V-Learning using $c$-step returns:

$$V_H(s) = \mathbb{E}\left[\sum_{t=0}^{c-1} \gamma^t r_t + \gamma^c V_H(s_{c})\right]$$

The high level bootstraps every $c$ steps, compressing the horizon from $T$ to $T/c$. The high-level policy learns subgoal prediction through Advantage-Weighted Regression (AWR):

$$\pi_H(z \mid s, g) \propto \exp\left(\alpha_H \, A_H(s, z, g)\right)$$

where $A_H$ is the high-level advantage function and $\alpha_H$ is the temperature. The subgoal $z$ is represented in a learned latent space $\phi(s)$.

3.3 Low-Level Action Chunking Critic

The low-level critic is HiQC's core innovation — it performs $k$-step Bellman backups rather than single-step:

$$Q_L(s, \mathbf{a}_{0:k}) = \mathbb{E}\left[\sum_{t=0}^{k-1} \gamma^t r_t + \gamma^k Q_L(s_k, \mathbf{a}'_{0:k})\right]$$

where $\mathbf{a}_{0:k}$ is the action chunk of length $k$. Since all actions in the chunk come from the dataset (offline setting), this $k$-step backup is unbiased — introducing no extrapolation error from out-of-distribution actions. The low-level policy learns via AWR over the action chunk space:

$$\pi_L(\mathbf{a}_{0:k} \mid s, z) \propto \exp\left(\alpha_L \, A_L(s, \mathbf{a}_{0:k}, z)\right)$$

The low-level policy is parameterized with Conditional Flow Matching (CFM) rather than a standard Gaussian policy, better capturing the multimodal distributions inherent in trajectory data.

3.4 Theoretical Analysis: Bootstrap-Chain Bound

Under a bounded per-backup error model (Assumption 1) with per-step backup error $\epsilon$, the paper derives value error bounds. For standard HIQL, the high level needs $T/c$ backups (contributing $T/c \cdot \epsilon_H$), and the low level needs $c$ single-step backups per subgoal interval (contributing $c \cdot \epsilon_L$):

$$\mathcal{E}_{\text{HIQL}}(T, c) \leq \frac{T}{c}\,\epsilon_H + c\,\epsilon_L$$

HiQC's low level uses $k$-step chunk backups, needing only $c/k$ chunk backups per subgoal interval:

$$\mathcal{E}_{\text{HiQC}}(T, c, k) \leq \frac{T}{c}\,\epsilon_H + \frac{c}{k}\,\epsilon_L$$

Optimizing $c$ yields the minimum error bound:

$$\min_c \mathcal{E}_{\text{HiQC}} \leq 2\sqrt{\epsilon_H \epsilon_L}\,\sqrt{\frac{T}{k}} = \mathcal{O}\!\left(\sqrt{\frac{T}{k}}\right)$$

while HIQL's optimal bound is $\mathcal{O}(\sqrt{T})$. Chunking reduces the number of low-level backups from $c$ to $c/k$, improving scaling from $\mathcal{O}(\sqrt{T})$ to $\mathcal{O}(\sqrt{T/k})$.

4. Experimental Results

4.1 OGBench Main Results

Key findings:

• Significant long-horizon navigation advantage: On humanoid-giant, HiQC achieves 33% while HIQL only 10%, and flat methods fail almost completely. On pointmaze-giant, HiQC reaches 80%.
• antmaze-giant ties HIQL: Both at 68%, because antmaze uses the smallest chunk size $k=2$, limiting horizon compression.
• Mixed manipulation results: HiQC is best on cube-triple (9%), second on scene-play (65%), but DQC outperforms on puzzle-4x6 (16% vs 1%) and scene-play (76% vs 65%).
• Highest aggregate: HiQC's overall success rate of 53% surpasses HIQL (42%) and DQC (26%).

4.2 Ablation Studies

Flow Matching vs Gaussian Policy: CFM policy consistently achieves higher success rates across 9 environments, validating that flow matching better captures multimodal distributions, preventing the "averaging" artifacts of Gaussian policies that disrupt temporally extended actions.

Chunk Size Sensitivity: As $k$ increases, performance first improves then plateaus; the choice of $c$ also affects performance, requiring trade-offs between high-level planning frequency and low-level execution precision.

5. Limitations and Shortcomings

• Simplified theoretical model: The bootstrap-chain model abstracts away function approximation, sampling error, and inter-level interactions during learning; Assumption 1 may not hold uniformly when values are represented by deep neural networks. The theory should be viewed as intuition rather than an end-to-end guarantee.
• Manipulation task disadvantage: Only 1% on puzzle-4x6 (DQC achieves 16%); HiQC's latent subgoal bottleneck may fail to encode discrete object states (e.g., puzzle button states).
• Training stability: scene-* tasks require reduced learning rate ($10^{-6}$) and special settings for stable training, indicating the chunked low-level critic is harder to optimize in these domains.
• Chunk size selection: antmaze using $k=2$ limits compression; optimal $k$ and $c$ selection relies on domain knowledge without an automated method.
• Offline-only evaluation: No evaluation of online fine-tuning or sim-to-real transfer.

6. Summary

HiQC's core contribution lies in demonstrating that dual temporal decomposition — hierarchical subgoals at the planning level and action chunking at the execution level — effectively combats the curse of horizon in offline goal-conditioned RL. By having the low-level critic perform unbiased $k$-step backups over full action chunks, HiQC compresses the recursion depth of value propagation from $\mathcal{O}(\sqrt{T})$ to $\mathcal{O}(\sqrt{T/k})$. On OGBench, the overall success rate reaches 53%, with humanoid-giant improving from 10% to 33%.

However, the advantage concentrates in long-horizon navigation; on manipulation tasks, DQC's decoupled critic horizon design is superior. This suggests future work could explore combining HiQC's hierarchy with DQC's decoupled critic horizon, as well as automated hyperparameter selection. The paper honestly acknowledges that the theoretical analysis is structural intuition rather than an end-to-end guarantee, and the manipulation disadvantage may stem from the information bottleneck of latent subgoals.