Skip to content
RobotWorld
← Tags

#Code Generation (2)

WaterfallPagination
Dream-RSI: Recursive Self-Improvement through Evolving Worlds

Dream-RSI: Recursive Self-Improvement through Evolving Worlds

Recursive self-improvement is becoming increasingly vital for autonomous AI agents, where progress hinges on discovering high-value solutions across complex domains. The driver of this process is effective exploration, however, managing and improving exploration strategies remains a major bottleneck. Current systems face a fundamental dilemma: fixed strategies fail to adapt as search spaces scale, while online policy optimization requires navigating vast meta-search spaces under delayed and expensive feedback over long-horizon rollouts. We introduce {Dream-RSI}, a framework for scalable and recursively self-improving exploration. A lightweight orchestration layer makes exploration explicit and programmable while leaving the underlying coding agent unchanged. Our key insight is that accumulated discovery history can serve as a replay simulator over the realized search space. By performing dreaming in the replay simulator constructed from historical discovery trees, {Dream-RSI} secures immediate, low-cost off-policy feedback to evaluate and refine exploration policies without invoking repetitive, expensive online evaluations. The improved policy is subsequently redeployed online to drive further discovery, continuously expanding the simulator pool in a self-improving loop. Across algorithm engineering, mathematical optimization, and GPU kernel engineering, {Dream-RSI} achieves competitive or improved discovery quality while substantially reducing discovery cost in several settings.

Tong Zheng, Xidong Wu, Zheng ZhangSep 14, 2026
Recursive self-improvementWorld ModelsLLM AgentSep 14, 2026
ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search

ReST-RL: Reinforcing LLM Reasoning through Unified Self-Training and Value-Guided Search

GRPO is the representative RL method for improving LLM reasoning, yet it only ever sees one sparse reward at the end of a whole trajectory: when the rewards inside a sampled group land close together, the group-relative advantage collapses into noise and the policy learns almost nothing. ReST-RL reconnects policy optimization and value-guided search into a single self-training pipeline. Stage one, ReST-GRPO, first filters out low-information prompts by reward standard deviation, then draws prefixes from each prompt's highest-reward trajectory under a discrete exponential distribution and uses them as fresh online-GRPO starting contexts. A selected prefix is context only; its suffix is re-sampled and optimized rather than imitated. Stage two, VM-MCTS, runs MCTS under the now-static policy to self-collect value targets and trains a value model that predicts expected terminal reward. At inference the same model both allocates the tree search through UCT and ranks completed candidates in a Best-of-N fashion, so search and verification share one state-value scale. On coding benchmarks including APPS, BigCodeBench and HumanEval, Qwen3-8B moves from 0.503 to 0.689 average. In matched policy-value controls, ReST-GRPO + VM-MCTS reaches 0.642 on APPS-500 while GRPO + VM-MCTS reaches only 0.538, so the stage-one distributional shift survives value learning. End-to-end accounting puts ReST-GRPO at 1,752 GPU-hours against 2,080 for GRPO, hitting a 9% gain in 71 hours instead of 207. A value model trained only on code trajectories also transfers to MATH, Omni-MATH and GPQA-Diamond without target-domain tuning.

Sining Zhoubian, Dan Zhang, Jie TangAug 27, 2025
Reinforcement LearningGRPOMonte Carlo Tree SearchAug 27, 2025