ToolVerse: Unlocking Massive Environments and Long-Horizon Tasks for Agentic Reinforcement Learning

Paper: ToolVerse: Unlocking Massive Environments and Long-Horizon Tasks for Agentic Reinforcement Learning

Authors: Shuaiyu Zhou, Fengpeng Yue, Zengjie Hu, Yuanzhe Shen et al. (Meituan LongCat Interaction Team / Peking University / Fudan University / Wuhan University)

Source: arXiv:2607.15660

1. Background and Motivation

LLMs as autonomous agents show promise in interacting with tools and environments to accomplish complex tasks. Combined with Agentic RL, LLMs develop long-horizon reasoning and sequential decision-making. However, developing agentic systems faces three challenges:

• Insufficient environment diversity: existing Agent RL environments are typically limited to single or few tools (search engines, code interpreters), lacking complexity for long-horizon multi-turn tool integration.
• Difficulty of long-horizon task design: designing multi-turn tasks across multiple integrated tools is inherently hard, requiring each action to be informed by prior steps.
• Credit assignment difficulty: sparse terminal rewards cannot provide meaningful advantage estimates for individual actions in long trajectories, causing high policy gradient variance.

Figure 1: ToolVerse framework overview. Step 1: Scaling executable agent environments; Step 2: Graph-based long-horizon task synthesis; Step 3: Turn-Aware Relative Advantage estimation.

2. Scaling Executable Agent Environments

ToolVerse sources JSON tool definitions from open-source and proprietary repositories, developing an automated pipeline to convert them into executable MCP tools. Only toolsets forming closed loops (passing syntax validation and unit tests) are retained, yielding 422 executable tool environments with ~4438 tools. Toolsets are strictly filtered to contain 5–20 tools for appropriate reasoning complexity.

Figure 3: Tool distribution spans eight macro-domains and numerous specific entities, illustrating real-world scenario coverage.

3. Graph-Based Long-Horizon Task Synthesis

3.1 Tool Dependency Graph (TDG) Construction

A TDG $\mathcal{G}=(\mathcal{V},\mathcal{E})$ is defined per scenario, with nodes as tools and edges capturing dependencies. The LLM infers dependencies on two principles: (1) an edge $T_A\to T_B$ if $T_A$'s output is required as $T_B$'s input; (2) an edge if $T_B$ can only be invoked after $T_A$ in the logical sequence.

3.2 Dynamic Unlocking Sampling (DUS)

The algorithm maintains a ready queue $\mathcal{Q}\subseteq\mathcal{V}$ (zero in-degree tools), ensuring high-dependency tasks are "locked" until prerequisites complete. At each step $t$, a subset $S_t\subseteq\mathcal{Q}$ is sampled, and after execution, successor in-degrees are updated: $d_{\text{in}}(v)\leftarrow d_{\text{in}}(v)-1$. This topological progression induces a curriculum of increasing complexity:

$$T=[S_{1},S_{2},\dots,S_{m}], \quad k=\min(|\mathcal{Q}|,N)$$

3.3 Inverse Context Reconstruction

After DUS samples a dependency-compatible tool skeleton, arguments are instantiated in topological order against a mock database to obtain an executable Golden Trace. An LLM then converts each Golden Trace into a user-facing task, verified via LangGraph replay and Pass@8 filtering.

4. Turn-Aware Relative Advantage (TARA)

Figure 2: Turn-Aware Relative Advantage estimation. For each turn in a multi-turn trajectory, rule-based validation is performed, the normalized advantage is computed against the group distribution at that turn, and propagated to all tokens within the turn.

Standard GRPO normalizes rewards across the entire trajectory, assigning a single scalar advantage to all tokens. In long-horizon tool use, this coarse feedback cannot distinguish correct intermediate steps from fatal later errors. TARA decomposes turn-level advantage into Local (immediate correctness) and Future (downstream impact).

4.1 Binary Reward

$$r_{i,t}=\begin{cases}1.0,&\text{if }G_{t}\subseteq_{\mathrm{dict}}A_{i,t},\\0.0,&\text{otherwise}.\end{cases}$$

4.2 Local Advantage

$$A_{i,t}^{\text{local}}=\frac{r_{i,t}-\mu_{t}^{\text{local}}}{\sigma_{t}^{\text{local}}+\epsilon}$$

4.3 Gated Future Advantage

A consistency gate $\delta_{i,t}=r_{i,t}$ ensures future rewards are only credited if the current step is valid:

$$V_{i,t}=\delta_{i,t}\cdot\sum_{k=0}^{T-t}\gamma^{k}r_{i,t+k+1}, \quad A_{i,t}^{\text{future}}=\frac{V_{i,t}-\mu_{t}^{\text{future}}}{\sigma_{t}^{\text{future}}+\epsilon}$$

4.4 Total Advantage Fusion

$$A_{i,t}^{\text{total}}=A_{i,t}^{\text{local}}+\lambda\cdot A_{i,t}^{\text{future}}$$

with $\lambda=0.5$ by default.

5. Method Architecture Flow

flowchart TD
    subgraph S1["Step 1: Scale Environments"]
        R["422 MCP environments\n~4438 tools"] --> AUTO["Automated conversion\nJSON -> executable MCP"]
        AUTO --> ENV["Executable environments\n(5-20 tools each)"]
    end
    subgraph S2["Step 2: Task Synthesis"]
        ENV --> TDG["Tool Dependency Graph G=(V,E)"]
        TDG --> DUS["Dynamic Unlocking Sampling\nready queue Q (zero in-degree)"]
        DUS --> TRACE["Golden Trace\n(topological instantiation)"]
        TRACE --> FILTER["Pass@8 filtering\n(LangGraph verification)"]
        FILTER --> GUST["GUST dataset"]
    end
    subgraph S3["Step 3: TARA Training"]
        GUST --> ROLL["K rollouts"]
        ROLL --> REWARD["Per-turn binary reward\nr_{i,t}"]
        REWARD --> AL["Local advantage A^local"]
        REWARD --> AF["Gated future advantage A^future"]
        AL --> TOTAL["Total advantage A^total\n= A^local + lambda * A^future"]
        AF --> TOTAL
        TOTAL --> UPDATE["Policy gradient update"]
    end

6. GUST Dataset

Figure 4: GUST dataset statistical distribution: Pass@K score, graph complexity, tools per environment, tasks per data item.

Per toolset, three distinct dependency graphs are generated, with five tasks sampled per graph. Pass@8 filtering: a teacher agent (Qwen3-32B) attempts each task up to 8 times; at least one trajectory with trace_score=1 is required to retain the task. Maximum 10 distinct tasks per toolset. Data items contain 3–7 tasks; most environments have 7–13 tools.

7. Experimental Results

Model	BFCL Overall	τ²-Bench Overall	ACEBench Overall
Qwen3-4B baseline	25.38	21.06	41.28
+ ToolVerse (GRPO)	28.50 (+3.12)	25.77 (+4.71)	48.34 (+7.06)
+ ToolVerse (TARA)	28.25 (+2.87)	26.83 (+5.77)	55.00 (+13.72)
Qwen3-8B baseline	28.88	27.87	46.51
+ ToolVerse (GRPO)	35.25 (+6.37)	30.10 (+2.23)	56.66 (+10.15)
+ ToolVerse (TARA)	37.50 (+8.62)	32.37 (+4.50)	61.66 (+15.15)

ToolVerse consistently achieves significant gains across all model scales, with TARA achieving the best overall results. On ACEBench-Agent, TARA lifts Qwen3-8B from 46.51% to 61.66% (+15.15), with the most pronounced improvements on multi-turn and multi-step subtasks — where the credit assignment problem is most acute.

Figure 5: Training curves. The TARA-enhanced model shows continuous improvement in Trace Score and Val Score, with faster and more stable convergence.

Ablation Study

Method	BFCL-v3	τ²-Bench
Qwen3-8B baseline	28.88%	27.87%
+ GRPO	35.25%	30.10%
Turn-local only	33.75%	27.33%
Turn-local + future w/o gate	35.00%	28.17%
Full TARA	37.50%	32.37%

Full TARA is best. Local-only credit is insufficient for long-horizon optimization; ungated future credit also underperforms, confirming the gate's importance for filtering noisy future signals. Environment scaling from 100 to 422 environments raises BFCL-v3 from 35.00% to 37.50% and τ²-Bench from 27.33% to 32.37%.

8. Conclusion

ToolVerse advances agentic RL by scaling executable environments and synthesizing long-horizon tasks via graph-based dynamic unlocking sampling. The framework auto-builds environments from 422 real MCP environments (~4438 tools), generates the GUST dataset via TDG + DUS, and proposes the Turn-Aware Relative Advantage (TARA) algorithm to solve credit assignment in multi-turn tool interactions. Experiments demonstrate advanced performance on complex tool-use benchmarks, with TARA consistently outperforming naive GRPO. Future work will expand task generation methods for simulated and real-world environments and explore more scalable, fine-grained credit assignment strategies.