World Models

Dreamer-CPC: Message Learning with World Models for Decentralized Multi-agent Reinforcement Learning

In multi-agent reinforcement learning (MARL), inter-agent communication is effective for improving performance under partial observability. Representation learning-based approaches enable decentralized agents to learn messages grounded in their own observations, but they rely only on current observations and cannot convey information accumulated over time. We propose Dreamer-CPC, a decentralized model-based MARL method that integrates message learning based on Collective Predictive Coding (CPC) into the world model of DreamerV3. Each agent independently maintains a world model and a message module, and infers and exchanges messages from the latent states of the world model that reflect the history of past observations and actions. We evaluated Dreamer-CPC in two environments: Observer, a non-cooperative information-sharing task, and CatchApple, a newly introduced task in which task-relevant observations are temporarily missing. In both environments, Dreamer-CPC outperformed IPPO-CPC, an existing CPC-based method that generates messages from current observations, as well as no-communication baselines. In particular, in CatchApple, Dreamer-CPC achieved 4 to 5 times the episode return of IPPO-CPC, demonstrating effective coordination where other methods fail due to missing observations. These results suggest that communication grounded in the latent dynamics of world models can support decentralized decision-making when current observations alone are insufficient.

Taisuke Takayama, Naoto Yoshida, Tadahiro Taniguchi

The World Model Remembers, the Actor Forgets: Dream Rehearsal for Continual Model-Based RL

Model-based reinforcement-learning agents of the DreamerV3 family forget catastrophically when trained on task sequences, even when an unbounded replay buffer preserves every earlier experience. We ask a question the continual-RL literature has assumed an answer to but never measured: which component forgets? Under never-clear replay, pre-registered component-level probes (n=3 seeds throughout) show that the world model retains essentially everything measurable about old tasks – reward discrimination (retention ratio ~1.0), value estimates, and termination structure – while the actor's behavior collapses. Forgetting in this regime is a channel problem, not a memory problem. We demonstrate this by intervention: with the world model frozen and identical imagined rollouts, reinforcement learning in imagination fails to recover a lost skill (0/3 seeds), while supervised self-imitation on the world model's own graded dreams recovers it on 3/3 seeds with zero environment interaction. Interleaved during training, this graded dream rehearsal yields a task-label-free, parameter-constant continual learner: 3/3 four-task chains retained where plain replay passes 0/3, 3/3 eight-task chains, and consistent gains over matched real-episode cloning (paired difference +0.13, bootstrap 95% CI [0.07, 0.24], complete seed separation). The dream-grading step is load-bearing: we characterize two scoring failure modes, provide an offline selection gauge that caught both before they contaminated results, and give a realized-first grading rule that closes them. All experiments were pre-registered with committed protocols; every refuted hypothesis is reported.

Gurp Nijjer

Koopman Dreamer: Spectrally Constrained Latent Dynamics for Stable World-Model Imagination

Latent world models improve sample efficiency in continuous control by optimizing policies over imagined latent trajectories, but common neural transitions offer limited direct control over modal persistence and error accumulation in long rollouts. We propose Koopman Dreamer, a Dreamer-style world model with a spectrally constrained deterministic latent dynamics core. Its Koopman-inspired backbone uses two-dimensional rotation–scaling blocks with bounded radii to represent damping, rotation, and near-periodic modes. Linear and low-rank bilinear action terms capture global and state-dependent control effects, while stochastic-state modulation supplies local correction information. To reduce the mismatch between posterior-conditioned training and prior-only imagination, the model combines posterior-conditioned EMA teacher targets with one-step consistency, multi-step rollout, and open-loop observation-prediction objectives. We further derive a multi-step rollout-error bound that separates amplification by the spectral backbone and bilinear interaction from the additive effects of stochastic-state mismatch and modeling residuals, clarifying the trade-off between error attenuation and long-term information retention. Experimental results on proprioceptive continuous-control tasks from the DeepMind Control Suite and UAV-LiDAR autonomous navigation demonstrate that Koopman Dreamer improves the stability of long-horizon latent rollouts and achieves stronger closed-loop control performance on tasks that rely on high-quality multi-step imagination.

Jiaqi Li, Xinglong Zhang, Haibin Xie

NavVerse: Benchmarking Indoor-to-Outdoor Embodied Navigation in Continuous Robot Simulation

NavVerse from UMich CURLY Lab and Tsinghua builds a physics-enabled navigation benchmark on Isaac Sim with 100 indoor, 50 outdoor and 50 indoor-to-outdoor hybrid scenes and 10,000 ObjNav/PlaceNav/VLN episodes evaluated through a unified waypoint interface and PID execution layer. Zero-shot experiments across RL, VLA and modular baselines show the strongest VLA drops from 17.65% outdoor to 3.64% hybrid on PlaceNav, identifying the indoor-to-outdoor transition as the major bottleneck; the benchmark further contributes collision-rate, obstacle-clearance and navigable-surface safety diagnostics plus embodiment-dynamics controls.

Junzhe Wu, Yue Hu, Zeyu Han

AlayaWorld: Long-horizon playable video world generation (15B)

AlayaWorld team releases a 15B-parameter interactive playable video world generation model at 24FPS 720p with open weights. Uses Depth Anything and point clouds as spatial memory for long-horizon world generation.

ABot-World-0: Infinite interactive world rollout on a single desktop GPU

An action-conditioned video world model for real-time long-horizon closed-loop interaction. Trained on multi-source data (AAA games, simulation engines, internet videos) with progressive distillation and LongForcing to mitigate drift. Streams 720P video at 16 FPS on a single RTX 5090 with 1.2s first-frame latency and ~19GiB peak VRAM.

Patch Policy: pretrained ViT beats OpenVLA-OFT with 0.7% params

Patch Policy uses pretrained ViT plus a small transformer to beat OpenVLA-OFT with only 0.7% of its params, trainable on a single RTX 5090, demonstrating cable insertion at 2mm tolerance.

Agentic Real2Sim: reconstructing a physics simulator from a single video

Agentic Real2Sim lets a vision-language agent reconstruct an editable, executable physics simulator from a single real-world video, instead of learning physics directly from video.

Google Genie 3 Simulates the Real World from Street View Images

Google Genie 3 can fully simulate the real world from street view images alone, leveraging Googles most detailed physical reality map ever built.

Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents

Real-to-sim conversion for robotic interaction with objects remains labor-intensive because it requires more than visual reconstruction: a streamlined real2sim process must recover scene geometries and object states, infer physical parameters, and assemble actors, objects, cameras, poses, and trajectories into a runnable physical simulation. Today this process still depends on manual tuning of visual foundation models, mesh cleanup, coordinate-frame alignment, and brittle workflow glue across visual perception tools and simulators. We introduce Agentic Real2Sim, a framework for generalized physical world modeling with vision-language agents, converting a real-world recording of object-robot interaction into a simulatable episodic twin which preserves observations, geometries, robot interactions, and object states. We evaluate Agentic Real2Sim on rigid-object manipulation, deformable-object interaction, and humanoid motion scenes, spanning domains that are usually handled by separate Real2Sim pipelines, marking a first step toward scalable conversion. The framework's agentic decisions can be driven by an open-weight VLM backend at a small fraction of the cost of frontier models, while attaining comparable conversion success rate. We aim to use the resulting real-world-aligned twins for downstream robotics tasks, specifically policy learning and evaluation. The project site is available at https://agentic-real2sim.github.io/.

Guanxiong Chen, Qianjun Xia, Jiawei Peng

AutoE2E: Open-Source End-to-End AI for Self-Driving

Autoware Foundation releases AutoE2E, an open-source end-to-end AI for self-driving, showing early model training results with green predictions and purple ground truth.

NVIDIA Cosmos 3 Edge: 4B Edge World Model for Robots

NVIDIA releases Cosmos 3 Edge, a 4B open model for real-time robotics and edge vision AI, combining vision, language, video and action to let robots understand scenes and predict what comes next.