False Prophets: On the Security of World Models in Agentic Systems

False Prophets: On the Security of World Models in Agentic Systems

Institution: BIFOLD & TU Berlin  |  arXiv: 2607.23147v1

---

One-Sentence Summary

This paper systematically explores security and privacy threats of world models in agentic systems, discovering seven attack vector categories (computability, non-determinism, missing knowledge, external environment, token count, misleading patterns, prompt injections) with up to 95% attack success rate, and provides the AgentWorld-Robust benchmark with multi-layered mitigation recommendations.

---

Background and Motivation

LLMs now power autonomous agents capable of complex, multi-step tasks. Accurate execution requires predicting action results — world models serve as environment simulators. While world models improve performance, they can also mislead agents into executing harmful actions, creating security and privacy risks. This paper raises security concerns about world models in agentic systems and argues some risks are intrinsic to approximate world modeling.

Figure 1: A world model misprediction resulting in deletion of system files — the model assumes $GITHUB_WORKSPACE points to the correct directory.

Threat Model

Three security properties threatened: Confidentiality (private information extraction), Integrity (unintended command execution), Availability (denial of service). The attacker's goal is to exploit world model prediction failures to induce harmful agent actions.

Seven Attack Vectors

Fundamental Issues

Computability: Predicting program runtime is equivalent to solving the halting problem. The Ackermann function $ack(4,1)$ has time complexity $O(iA(i,n))$ — a world model may predict the correct result without recognizing the computational cost, leading to unbounded resource allocation and DoS.

$$\text{Predicting runtime} \equiv \text{Halting Problem} \quad \Rightarrow \quad \text{undecidable}$$

Non-determinism: Race conditions depend on runtime scheduler decisions. The world model outputs a single deterministic prediction rather than the space of possible interleavings. A malicious branch may execute only for a specific interleaving that the model doesn't predict.

$$P_{\text{malicious}} = P(\text{interleaving}_k) > 0, \quad \text{but } \hat{y}_{\text{WM}} \neq \text{interleaving}_k$$

External Environment: Programs involving network requests, system time, or external APIs depend on state outside the model's knowledge. Output accuracy drops to 0.05 for Qwen on external environment tests.

Technical Issues

Missing Knowledge: Post-training-cutoff changes in command-line tools. Token Count: Repetitive output induces infinite loops in LLMs, preventing exit code generation. Misleading Patterns: False information in comments. Prompt Injections: Malicious instructions embedded in input.

Figure 2: Taxonomy of seven attack vectors — fundamental vs technical issues.

graph TD
    A["World Model Vulnerabilities"] --> B["Fundamental Issues
Intrinsic to approximate modeling"]
    A --> C["Technical Issues
Potentially fixable"]
    B --> D["Computability
Halting problem undecidable"]
    B --> E["Non-determinism
Race conditions unpredictable"]
    B --> F["External Environment
Out-of-scope state"]
    C --> G["Missing Knowledge
Post-cutoff changes"]
    C --> H["Token Count
Infinite loops"]
    C --> I["Misleading Patterns
False comments"]
    C --> J["Prompt Injections
Malicious instructions"]
    D --> K["DoS / Resource exhaustion"]
    E --> L["Control flow hijacking"]
    F --> M["Incorrect predictions"]
    G --> M
    H --> K
    I --> M
    J --> M

AgentWorld-Robust Benchmark

A security benchmark dataset for text-based world models covering seven attack vector categories. Scripts designed to test each vulnerability with ground-truth exit codes and outputs from real sandbox execution.

Figure 3: AgentWorld-Robust benchmark construction pipeline.

Evaluation Results

Category	Qwen Exit Code	Qwen Output	CWM Exit Code	CWM Output
Misleading	1.00	0.91	0.90	0.31
Injections	1.00	0.96	0.99	0.55
Compute	0.72	0.48	0.56	0.31
Non-Determinism	0.94	0.31	0.92	0.09
Missing Knowledge	0.91	0.41	0.84	0.09
Ext. Environment	0.87	0.05	0.93	0.11
Token Count	0.87	0.25	0.26	0.16
Overall	0.86	0.44	0.77	0.23

Exit code accuracy consistently exceeds output accuracy. World models surprisingly robust against misleading/injection attacks (91% output accuracy) — because they simulate action results, not follow instructions. Computability and non-determinism show lowest accuracy — fundamental limitations. Attack success rate reaches up to 95%.

Figure 4: Per-category accuracy comparison between Qwen and CWM world models.

Mitigation Recommendations

Layer	Measure	Effect
World Model	Additional info (actual time)	Extra context reduces misjudgment
World Model	Adversarial training	Enhance robustness to adversarial inputs
Agent	Agent harness	Independent verification layer
Agent	Multiple rollouts	Consensus from multiple simulations
System	Resource limits	Hard limits prevent DoS and wallet drainage

Figure 5: Qualitative examples of world model mispredictions across attack categories.

Figure 6: Attack success rates by vector category — up to 95% for some attack types.

The attack success rate can be modeled as the probability that the world model's prediction $\hat{y}$ differs from the ground truth $y$ for a given attack vector $v$:

$$ ext{ASR}(v) = P(\hat{y}_{ ext{WM}} eq y_{ ext{real}} | v) \leq 0.95$$

The overall risk of a world-model-guided agent action combines prediction error probability with action consequence severity:

$$ ext{Risk}(a) = \sum_v P(v) \cdot ext{ASR}(v) \cdot ext{Severity}(a, v)$$

where $P(v)$ is the probability of attack vector $v$ being present and $ ext{Severity}(a, v)$ measures the harm of action $a$ under misprediction $v$. This formulation shows that risk reduction requires either lowering ASR (better models) or limiting Severity (resource constraints, verification layers).

The fundamental limitation of world models can be formalized through Rice's theorem — any non-trivial semantic property of a program's execution is undecidable. Since world models approximate program execution, they inherently cannot correctly predict all cases:

$$orall ext{WM}: \exists s \in ext{Scripts}: \hat{y}_{ ext{WM}}(s) eq y_{ ext{real}}(s)$$

This is not a limitation of current technology but a fundamental bound — no approximate world model can achieve 100% prediction accuracy on all possible programs.

Limitations

1. Only text-based world models analyzed; image/video world models may have different attack surfaces
2. Mitigation measures cannot solve fundamental issues (computability, non-determinism, external dependency)
3. Benchmark scale limited; more comprehensive attack surface exploration needed

---

Conclusion

This paper systematically explores world model security in agentic systems. Seven attack vector categories are discovered — from computability (halting problem undecidability) to non-determinism, missing knowledge, external environment, and LLM-inherited weaknesses. Attackers can induce mispredictions with up to 95% success rate, potentially causing unintended command execution, DoS, wallet drainage, and privacy extraction. The AgentWorld-Robust benchmark and multi-layered mitigation recommendations are provided. Core insight: some risks are intrinsic to approximate world modeling — world models as "false prophets" introduce new attack surfaces while enhancing agent capabilities, and must be treated as untrusted security boundaries in deployment.

Key Insight: The world model is the agent's crystal ball — but crystal balls can lie, and a lying prophet is more dangerous than no prophet at all, because you trust it.