Dreamer-CPC: Message Learning with World Models for Decentralized Multi-agent Reinforcement Learning

Dreamer-CPC: Message Learning with World Models for Decentralized Multi-agent Reinforcement Learning

Paper: Dreamer-CPC: Message Learning with World Models for Decentralized Multi-agent Reinforcement Learning
Link: arXiv:2607.19809　|　Basis: DreamerV3 + Collective Predictive Coding (CPC)　|　Environments: Observer / CatchApple

One-line summary: Integrates CPC-based message learning into DreamerV3's RSSM world model. Each agent infers and exchanges messages from latent states reflecting observation/action history, achieving 4-5× the episode return of IPPO-CPC in the CatchApple task where observations are temporarily missing.

Background and Motivation

In multi-agent reinforcement learning (MARL), inter-agent communication improves performance under partial observability. Representation learning-based approaches enable decentralized agents to learn messages grounded in their own observations, but they rely only on current observations and cannot convey information accumulated over time. When task-relevant information is temporarily occluded or agents must act based on predictions of future states, generating messages from current observations alone is insufficient.

Existing methods mostly adopt centralized training with decentralized execution (CTDE), coupling agents through centralized optimization. While these improve engineering performance, they diverge from decentralized learning observed in natural agents. Collective Predictive Coding (CPC) formulates symbol emergence as decentralized Bayesian inference in a joint generative model, providing a theoretical foundation. MARL-CPC demonstrated that coordination-useful messages can emerge even in non-cooperative settings. But all these methods generate messages based only on current observations.

Dreamer-CPC's core idea: integrate CPC message learning into a world model that learns latent dynamics from observation/action histories, extending message learning beyond current observations. Each agent learns messages from RSSM latent states in DreamerV3, enabling communication grounded not only in current observations but in the history accumulated by the world model.

Method Details

1. Problem Formulation: Partially Observable Markov Game

The multi-agent RL problem is formulated as a partially observable Markov game $\mathcal{G}=\langle\mathcal{I},\mathcal{S},\{\mathcal{A}^{k}\},\{\mathcal{X}^{k}\},\mu_{0},p_{\text{tr}},p_{\text{obs}},\{\rho^{k}\},\gamma\rangle$. Each agent $k$ selects action $a_t^k$ using local history. The discounted return is $G_t^k=\sum_{\ell=0}^{\infty}\gamma^{\ell}r_{t+\ell}^k$, with objective:

$$J^{k}(\pi^{k};\pi^{-k})=\mathbb{E}\left[G_{0}^{k}\right]$$

2. World Model: RSSM

Each agent $k$ maintains its own world model learning temporal dynamics from local observation/action history and received messages. The RSSM has deterministic recurrent state $h_t^k$ and stochastic latent state $z_t^k$. At each step: $h_t^k=f_{\phi^k}(h_{t-1}^k,z_{t-1}^k,a_{t-1}^k)$, prior $\hat{z}_t^k\sim p_{\phi^k}(\cdot|h_t^k)$, posterior $z_t^k\sim q_{\phi^k}(\cdot|h_t^k,x_t^k)$. Observation, reward, and continuation are predicted from $h_t^k$ and $z_t^k$.

3. Message Module and Message-Augmented RSSM

Each agent $k$ maintains a recurrent message module with state $\xi_t^k$ summarizing previously inferred message history: $\xi_t^k=g_{\psi^k}(\xi_{t-1}^k,\mathbf{m}_{t-1})$. The message $m_t^k$ is modeled stochastically with prior $\hat{m}_t^k\sim p_{\psi^k}(\cdot|\xi_t^k)$ and posterior $m_t^k\sim q_{\psi^k}(\cdot|\xi_t^k,h_t^k,z_t^k)$.

Messages are incorporated into two RSSM components: the joint predicted message $\hat{\mathbf{m}}_t$ feeds into the recurrent state transition, and the joint inferred message $\mathbf{m}_t$ conditions observation prediction:

$$h^{k}_{t}=f_{\phi^{k}}(h^{k}_{t-1},z^{k}_{t-1},a^{k}_{t-1},\hat{\mathbf{m}}_{t})$$

$$\hat{x}^{k}_{t}\sim p_{\phi^{k}}(\cdot|h^{k}_{t},z^{k}_{t},\mathbf{m}_{t})$$

To achieve decentralized learning, Dreamer-CPC applies stop-gradient to messages from other agents, allowing each agent to use others' messages for prediction without backpropagating through their parameters.

Figure 1: Dreamer-CPC framework. Each agent infers messages from RSSM latent states and exchanges them; received messages are incorporated into subsequent state transitions and observation predictions.

4. Joint Learning Loss

The world model and message module are jointly trained by minimizing a unified loss. The variational posterior factorizes as:

$$q_{\phi^{k},\psi^{k}}(z_{1:T}^{k},m_{1:T}^{k})=\prod_{t=1}^{T}q_{\phi^{k}}(z_{t}^{k}|h_{t}^{k},x_{t}^{k})\,q_{\psi^{k}}(m_{t}^{k}|\xi_{t}^{k},h_{t}^{k},z_{t}^{k})$$

The loss includes prediction likelihood terms and KL regularization aligning priors to posteriors.

5. Actor-Critic Learning with Message Exchange During Imagination

Actor $\pi_{\theta^k}$ and Critic $v_{\eta^k}$ follow DreamerV3's actor-critic procedure. The key difference is message exchange during imagination: at each imagination step, every agent infers a message from its world-model state and exchanges it. Received messages are incorporated into subsequent RSSM transitions. Actor and critic losses are computed on the resulting imagination rollout, with stop-gradient ensuring received messages are detached during optimization.

graph TD
  A[Each agent's local obs/action history] --> B[RSSM: h_t, z_t]
  B --> C[Message module: posterior m_t from ξ_t, h_t, z_t]
  C --> D[Exchange messages: receive others' m]
  D --> E[Incorporate into RSSM transition: h_t = f(h,z,a,m̂)]
  D --> F[Observation prediction: x̂ ~ p(h,z,m)]
  E --> G[Imagination rollout + Actor-Critic training]
  F --> G
  style C fill:#f5a623,stroke:#b97316,color:#fff
  style D fill:#4a90d9,stroke:#2c5f8a,color:#fff
  style G fill:#7ed321,stroke:#4a8a14,color:#fff

Experimental Results

Observer Environment

Observer is a non-cooperative information-sharing task where the global state can be accurately inferred by aggregating all agents' observations at each step. The obs-shared condition reached IQM 53.63. Among settings without direct observation sharing, Dreamer-CPC achieved the highest performance: $5\times 4$ messages reached IQM 39.85 (95% CI [34.22, 40.82]), $20\times 1$ reached 19.72. IPPO-CPC reached only 7.31 ($20\times 1$) and 6.81 ($5\times 4$). No-communication IPPO and no-comm DreamerV3 remained near zero (1.74 and 0.48).

Method	Observer IQM	CatchApple IQM
obs-shared DreamerV3	53.63	38.21
Dreamer-CPC (5×4)	39.85	28.53
Dreamer-CPC (20×1)	19.72	20.47
IPPO-CPC (5×4)	6.81	5.52
IPPO-CPC (20×1)	7.31	4.98
IPPO (no comm)	1.74	4.66
no-comm DreamerV3	0.48	5.93

CatchApple Environment

CatchApple is a newly introduced task where agents' observations are temporarily missing at certain time steps — the global state cannot be accurately inferred even by aggregating all observations. Agent 1 remains stationary observing a falling object's trajectory; Agent 2 controls a catcher but cannot directly observe the object's future landing column. Since Agent 2 observes the object's column only after the capture decision, it must predict the future landing column from Agent 1's message and move in advance. Dreamer-CPC achieved IQM 28.53 ($5\times 4$) and 20.47 ($20\times 1$), 4-5× that of IPPO-CPC (5.52/4.98). IPPO-CPC remained comparable to no-communication IPPO (4.66) and no-comm (5.93), showing that current-observation-based message learning completely fails when observations are missing.

Figure 2: Learning curves for Observer and CatchApple. Dreamer-CPC substantially outperforms baselines in both environments.

Message Channel Configuration Analysis

Message Config	Observer Dreamer-CPC	CatchApple Dreamer-CPC	Observation
5×4 (four independent 5-way)	39.85	28.53	Superior to 20×1, higher diversity
20×1 (single 20-way)	19.72	20.47	Concentrated information capacity

The $5\times 4$ configuration (four independent 5-way one-hot messages) outperforms $20\times 1$ (single 20-way) in both environments, suggesting multiple independent message channels are more effective than a single large vocabulary — independent channels may allow simultaneous encoding of different information dimensions rather than compressing all information into a single discrete symbol.

Figure 3: CatchApple environment. Agent 1 observes the falling object; Agent 2 must predict the landing position from messages and move in advance.

联合动作空间

$$ \mathcal{A}=\prod_{k\in\mathcal{I}}\mathcal{A}^{k} $$

Limitations

Author-stated: Experiments are validated only in two relatively simple grid-world environments; applicability to more complex continuous control scenarios is unverified.

Analysis: Dreamer-CPC's decentralized design comes at the cost of synchronous message exchange — at each imagination step, all agents must synchronously infer and exchange messages, which may become a bottleneck as agent count grows. Stop-gradient achieves decentralization but blocks gradient-based cooperative optimization between agents, potentially limiting the upper bound of joint message quality optimization. Joint training of the world model and message module increases optimization complexity and hyperparameter sensitivity. Each agent maintaining an independent world model incurs linear compute/memory growth with agent count. The 4-5× advantage in CatchApple is partly because the IPPO-CPC baseline completely fails when observations are missing — no-communication baselines also perform very low, making the relative advantage appear larger. The obs-shared upper bound (38.21) still exceeds Dreamer-CPC (28.53), indicating message communication remains less effective than direct observation sharing. The non-cooperative setting, while biologically more plausible, may be less efficient than CTDE methods in cooperative engineering applications.

Conclusion and Outlook

Dreamer-CPC integrates CPC-based message learning into an RSSM-based world model. Each agent learns stochastic messages through a recurrent message module, and received messages are used for predicting observations, rewards, and continuation signals. Stop-gradient on other agents' messages achieves decentralized learning without direct gradient paths between agents. Experiments show Dreamer-CPC outperforms IPPO-CPC and no-communication baselines in non-cooperative settings. The CatchApple results demonstrate that communication learned through a world model can support effective coordination even when task-relevant observations are temporarily unavailable. The core methodological contribution is extending message learning from current observations to world model latent dynamics — messages reflect not just "what is seen now" but "what is predicted based on history." This temporal extension enables agents to compensate through communication when current observations are insufficient, providing a new paradigm for coordination in partially observable decentralized multi-agent systems.

The value of a message lies not in conveying "what is seen now" but in conveying "what is predicted based on history" — when current observations are insufficient, the world model's latent dynamics become the information source for communication. This is the core of Dreamer-CPC's success in observation-missing scenarios.