PAPER DEEP DIVE
PhiZero: A World Model Built Around Physical Language
We introduce PhiZero, a physical world model built around physical language, a compact discrete representation of world-state transitions. Existing physical world models typically predict future videos directly in pixel space, leaving the underlying world dynamics implicit within high-dimensional visual predictors. Motivated by humans' ability to abstract predictive structure from visual experience and organize it in natural language for explicit reasoning, we learn physical language from in-the-wild videos through self-supervision and use it to explicitly reason about how the physical world evolves. Accordingly, PhiZero adopts a reason-then-render paradigm: it first infers future world evolution as a physical-language sequence and then renders the inferred transitions into videos. Extensive experiments across generation and understanding benchmarks validate the ability of PhiZero to model physically coherent world evolution. We further show its potential for realistic and interactive world modeling, fine-grained action-conditioned simulation, and zero-shot motion transfer.
Paper Information
| Field | Details |
|---|---|
| Title | PhiZero: A World Model Built Around Physical Language |
| Authors | Shuyao Shang, Yuqi Wang, Ruopeng Gao, Xu Chen, Tieniu Tan, Lue Fan, Zhaoxiang Zhang (CASIA NLPR) |
| Paper | arXiv:2607.28624 |
| Project | Phi-Zero.github.io |
One-line summary: PhiZero proposes a world model built around "physical language"—a compact discrete state-transition representation learned through self-supervision from in-the-wild videos, reasoning future world evolution as a physical-language sequence before rendering to video, achieving the best physical fidelity (IQ-Score 41.2) on the Physics-IQ benchmark.
Background and Motivation
Video world models can generate future videos with high visual fidelity and are increasingly regarded as promising physical world simulators. However, for such models to support Physical AI, their central objective must go beyond visual fidelity toward learning how the physical world evolves. Mainstream approaches train primarily through direct pixel-space prediction, leaving the underlying dynamics implicit within high-dimensional visual representations and often resulting in physically inconsistent outcomes.
The authors draw inspiration from human intelligence. Rather than memorizing individual visual outcomes, humans abstract patterns from visual experience into generalizable knowledge about how the world evolves. Natural language serves as a primary medium for organizing and expressing such knowledge, providing a symbolic space for explicit reasoning. The success of language models demonstrates the scalability of language as a substrate for learning and reasoning.
However, when the modeling target shifts from the digital world to the physical world, natural language is often too coarse to faithfully represent the complex state transitions observed in visual experience. The authors ask: can we move beyond natural language and learn a finer-grained representation of physical-world transitions for explicit reasoning?
Figure 1: PhiZero learns a compact discrete physical language from in-the-wild videos, reasons world evolution in this space, and renders the inferred transitions into videos.
Method Details
Core Paradigm: Reason-Then-Render
PhiZero adopts a "reason-then-render" paradigm: it first infers future world evolution as a physical-language sequence, then renders the inferred transitions into video. This paradigm separates dynamics inference from pixel-level synthesis, making world evolution an explicit reasoning target rather than a direct pixel-space prediction.
Physical Language Tokenizer
The Physical Language Tokenizer learns state-transition representations through self-supervised video reconstruction. Given a video, the tokenizer encodes its state transitions into a discrete physical-language sequence, which together with the first frame conditions the reconstruction. The key design disentangles world dynamics from visual appearance—physical language captures "how the world changes" rather than "how the world looks."
The tokenizer uses FSQ-induced discrete vocabulary, quantizing continuous visual features into $K$ atomic symbols. Each timestep generates $N$ physical-language tokens, forming a sequence of length $N$. Training has two stages: (1) pretraining—on ~5M four-second video clips, progressively increasing clip duration (1s→2s→4s) at 256×448; (2) SFT—at 512×896 on a curated corpus.
Figure 2: PhiZero pipeline. Physical Language Tokenizer learns discrete state-transition representations, Reasoner autoregressively predicts physical language, diffusion decoder renders to video.
Physical Language Reasoner
The reasoner is initialized from a pretrained VLM, with vocabulary extended to include FSQ-index atomic symbols. Given the first frame $I^0$ and a text prompt $c$, the reasoner autoregressively predicts a length-$N$ physical-language sequence. The conditional distribution is factorized in temporal order:
$$p_\theta(\mathbf{z} \mid I^0, c) = \prod_{j=1}^{N} p_\theta(z_j \mid I^0, c, z_{<j})$$
Training uses the autoregressive cross-entropy objective under teacher forcing:
$$\mathcal{L}_{\text{VLM}} = -\sum_{j=1}^{N} \log p_\theta(z_j \mid I^0, c, z_{<j})$$
Training has two stages: (1) continued pretraining on a 5M-clip general corpus to establish correspondence among textual intent, visual state, and state transitions; (2) SFT on ~1M motion-rich, physically informative clips to improve physical plausibility and precision.
Figure 3: Data pipeline. VLM generates high-level action summaries as conditions; frozen tokenizer provides physical-language targets.
graph LR
A["In-the-wild Videos"] --> B["Physical Language Tokenizer
Self-supervised Learning"]
B --> C["Discrete State-Transition Sequence
(Physical Language)"]
D["First Frame + Text Prompt"] --> E["Physical Language Reasoner
(VLM-initialized)"]
C --> E
E --> F["Predicted Physical-Language Sequence"]
F --> G["Diffusion Decoder"]
G --> H["Future Video"]
Figure 4: PhiZero inference flow. First reasons physical-language sequence, then renders to video.
Experimental Results
Physical Outcome Fidelity
Evaluated on the Physics-IQ Verified benchmark, including S-IoU (Structure), ST-IoU (Spatiotemporal), WS-IoU (World State), and IQ-Score (composite).
| Model | S-IoU | ST-IoU | WS-IoU | IQ-Score |
|---|---|---|---|---|
| Wan2.2-5B | 24.7 | 22.6 | 13.3 | 21.2 |
| Sora 2 | 37.3 | 27.0 | 26.9 | 26.5 |
| Cosmos3-Nano | 40.4 | 22.0 | 24.6 | 29.1 |
| Wan2.2-14B | 51.1 | 20.5 | 28.5 | 32.2 |
| Hunyuan-Video | 47.1 | 26.9 | 29.7 | 33.4 |
| Grok-Video | 52.7 | 21.4 | 35.7 | 34.8 |
| Cosmos3-Super | — | — | — | 39.5 |
| PhiZero | 58.2 | 36.8 | 27.6 | 41.2 |
Table 1: Physical outcome fidelity on Physics-IQ Verified. PhiZero achieves the highest IQ-Score.
Figure 5: Physical fidelity visual comparison. PhiZero generates videos with superior physical consistency.
Broader Applications
PhiZero also demonstrates potential in controllable interactive world modeling, fine-grained action-conditioned simulation, and zero-shot motion transfer.
Figure 6: Zero-shot motion transfer. PhiZero can transfer motion patterns from one video to new scenes.
Limitations
1. WS-IoU not optimal: While PhiZero achieves the highest IQ-Score, it does not achieve the best WS-IoU (World State IoU), indicating room for improvement in fine-grained state capture.
2. Training data scale: Requires ~5M video clips for tokenizer training and ~1M curated clips for reasoner SFT, incurring high training costs.
3. Resolution limitations: Currently trained at 512×896 resolution, which may be insufficient for high-resolution physical simulation needs.
Summary and Future Work
PhiZero's core contribution is the "physical language" intermediate representation—a compact discrete state-transition encoding learned through self-supervision from in-the-wild videos. This representation disentangles world dynamics from visual appearance, making world evolution an explicit reasoning target. The "reason-then-render" paradigm separates dynamics inference from pixel synthesis, avoiding the problem of implicit dynamics in direct pixel prediction.
Experimental results strongly validate this paradigm: on the Physics-IQ Verified benchmark, PhiZero achieves an IQ-Score of 41.2, surpassing strong baselines like Sora 2 (26.5) and Cosmos3-Super (39.5). The particularly large leads in S-IoU (58.2 vs. next best 52.7) and ST-IoU (36.8 vs. next best 27.0) demonstrate that the physical language reasoner can more accurately predict physical structural evolution.
"Reason-then-render"—this paradigm shift reveals a fundamental issue in world modeling: directly predicting the future in pixel space is like drawing a chess game without looking at the board; only by first reasoning about physical state transitions and then rendering does the model truly understand "how the world evolves."
