RSIAgent: Autonomous Exploration for Recursive Self-improvement in New Environments

Paper: Sibo Zhu, Shicheng Fan, Xinyue Wang, Wenyi Wu, Kun Zhou, and Biwei Huang, from Aether AI, UC San Diego, and the University of Illinois Chicago. arXiv:2609.15364. Project page. The official GitHub repository is Apache-2.0 licensed; this analysis checks commit 4d7ef43770dda341fccd9a5743db92407687c615. The repository appeared after the manuscript, so it supports inspection of the implementation but is not a complete run-by-run reproduction package.

One-sentence summary: RSIAgent keeps the underlying model frozen and improves an agent in an unfamiliar software environment by having curriculum, actor, and verifier roles autonomously create practice work, ground outcomes in the environment, and consolidate only audited experience into a reusable memory that is later frozen for a fresh evaluator actor.

1. Why a new environment cannot be solved by prompting alone

Long-horizon computer use depends on local knowledge. An agent may recognize a spreadsheet application, a browser, or a 3D editor in its visual input while still not knowing which import dialog preserves a required field, which save format changes the artifact, or which sequence avoids a destructive default. These facts live in the environment and in the conventions of a particular workflow, not reliably in a pretrained model's weights.

The usual adaptation answer is to collect demonstrations or outcome rewards and train the policy. RSIAgent instead treats the durable memory as the object that changes. A model with fixed parameters can retrieve procedures, scripts, constraints, and failure lessons from that memory. The design reduces the need to create a separate trained policy for every private or rapidly changing software environment.

Memory alone is not sufficient, because a transcript can preserve a mistaken success. A candidate might appear to work because a verifier inspected the wrong file, accepted an unsupported default, or missed one requirement. The paper therefore joins three functions that are often collapsed into one model call: proposing useful practice, executing it, and independently grounding the outcome. The actor is then responsible for learning from the resulting evidence rather than from its own optimistic narrative.

The word “recursive” describes two feedback paths. At the broad scale, knowledge consolidated after one wave changes what the curriculum proposes in the next wave. At the task scale, a target attempt produces a failure or uncertainty that determines the next focused exercise. The exploratory loop eventually stops, freezes memory, and evaluates a fresh actor without further learning, which separates adaptation during practice from reuse during the final attempt.

2. Formal view of interaction and reported performance

For a task instruction $q$ and persistent memory $M$, the actor observes an interaction history. The history contains the initial observation, prior actions, and observations returned after those actions:

$$h_t=(o_0,a_1,\ldots,a_{t-1},o_{t-1})$$

At step $t$, the fixed policy samples an action conditioned on the query, history, and memory:

$$a_t\sim\pi_\theta(\cdot\mid q,h_t,M)$$

The action is executable code, so the same interface can manipulate heterogeneous desktop applications. Execution changes an unobserved environment state and returns the next observation:

$$(s_t,o_t)\sim\mathcal{E}(\cdot\mid s_{t-1},a_t)$$

Here, the model parameters $\theta$ do not change. Interaction history is reset between tasks, while $M$ persists and is revised across attempts. This distinction matters: the paper does not claim parameter-level self-modification. Its self-improvement is an agent-system property produced by exploration, evidence, and memory maintenance.

Each benchmark task $i$ receives a partial score $s_i$ in the unit interval:

$$s_i\in[0,1]$$

The aggregate Partial metric is the mean score across $N$ tasks:

$$\mathrm{Partial}=100\sum_{i=1}^{N}\frac{s_i}{N}$$

Binary accuracy counts only full-credit outcomes:

$$\mathrm{Binary}=100\sum_{i=1}^{N}\frac{\mathbf{1}\{s_i=1\}}{N}$$

Reporting both quantities prevents an agent that repairs more subparts from being confused with one that finishes more whole tasks. The distinction becomes especially important because RSIAgent's reported Partial advantage on Agents' Last Exam is not paired with a Binary advantage over the strongest GPT-6 Astra entry.

3. A harness with deliberately unequal permissions

The actor is the working agent. It reads the task and relevant memory, writes and runs code, manipulates the environment, and produces a candidate artifact. After verification, the actor writes durable lessons into memory: new procedures, reusable scripts, constraints, and notes about failures. The actor owns canonical memory updates because it has the execution context needed to explain what changed and why.

The verifier is not a second actor that merely agrees with the first. It examines the environment and candidate without access to the actor's private reasoning or memory. Its job is to return a verdict and a report grounded in execution output, interface state, and artifact evidence. PASS, FAIL, and UNVERIFIED are not interchangeable: unresolved evidence should not become a capability lesson.

The curriculum role decides what experiment would be informative next. It can inspect the target direction, prior outcomes, and a disposable view of memory, but it cannot edit canonical memory or certify an answer. This separation prevents the agent that selects a practice task from also declaring that its chosen exercise was completed correctly.

The permission boundary is the central systems idea in the paper. A benchmark harness often rewards a clean end-to-end loop, yet a clean loop can hide ownership errors. RSIAgent assigns task generation, candidate execution, correctness judgment, and durable learning to different stages. The resulting protocol is more expensive than a single self-critique call, but it creates observable points at which a wrong result can be rejected before it changes future behavior.

4. Broad exploration creates coverage; deep exploration tests causality

Broad Recursive Self-exploration, or BRS, maps the environment before concentrating on one failing path. The curriculum proposes a wave of related but distinct projects: prerequisites, variants, contrasts, and stress cases. Each branch runs in an isolated virtual machine with a fresh actor context. A verifier inspects that branch's evidence, and the branch actor later distills the result.

The implementation deliberately separates concurrent execution from concurrent memory writes. Every branch in a wave starts from the same pre-wave memory snapshot. Branches may discover conflicting or complementary facts, but they cannot observe their siblings during execution. Their updates are resumed in curriculum-authored order and committed into a single canonical bank one at a time.

The code makes this ownership visible. In explore/phase1_wave.py:507-548, _commit_branch_memory restores the branch actor's history, asks it to distill the terminal outcome and verifier report, then asks it to reconcile the result before the memory can be installed:

# explore/phase1_wave.py:507-548
def _commit_branch_memory(...):
    """Resume the same branch Actor and serialize its update into one bank."""
    before = _read_memory_tree(str(canonical_memory_dir))
    _push_canonical_memory(hooks, branch.vm, str(canonical_memory_dir))
    history, _ = _run_free_phase(
        ...,
        prompt=self_evolving_actor_memory_distillation_msg(
            branch.terminal_outcome, branch.verifier_report),
        history=branch.actor_history)
    history, _ = _run_free_phase(
        ...,
        prompt=self_evolving_actor_memory_reconciliation_msg(),
        history=history)
    candidate = _pull_terminal_memory(hooks, branch.vm)
    accepted = hooks.audit_memory(candidate, corpus_path, ...)
    hooks.install_memory(str(canonical_memory_dir), accepted)

A second excerpt shows how the same immutable wave snapshot is provided to isolated workers. The thread pool controls host concurrency, but the curriculum decision controls the number and purpose of projects:

# explore/phase1_wave.py:684-719
wave_memory_dir = wave_dir / "memory_snapshot"
wave_memory = _read_memory_tree(str(memory_dir))
hooks.install_memory(str(wave_memory_dir), wave_memory)
assignments = tuple((
    project_index + offset,
    project,
    episodes_dir / f"ep{project_index + offset:03d}"
) for offset, project in enumerate(decision.projects, 1))

with ThreadPoolExecutor(
        max_workers=min(max_parallel, len(assignments)),
        thread_name_prefix=f"phase1-wave-{wave_index}") as pool:
    futures = {
        pool.submit(
            _execute_branch_with_infra_retry,
            vm_factory=vm_factory,
            project_index=index,
            project=project,
            episode_dir=episode_dir,
            wave_memory_dir=wave_memory_dir,
            actor_cfg=copy.deepcopy(actor_cfg),
            verifier_control_cfg=copy.deepcopy(verifier_control_cfg),
            target_direction=target_direction): index
        for index, project, episode_dir in assignments
    }

The docstring in evolve_parallel_phase1 explicitly says that project_budget is checked only between complete curriculum-authored waves, while max_parallel is scheduling capacity rather than wave width. Treating those controls as equivalent would let available hardware silently change the exploration distribution and the meaning of a budget.

Deep Recursive Self-exploration, or DRS, is sequential because each decision depends on the previous grounded outcome. The system first attempts the actual target with memory inherited from BRS. A FAIL triggers targeted practice around the observed gap, followed by a fresh target attempt and another verification. If a target passes under the CURRICULUM_REVIEW policy, the curriculum may still request a contrastive exercise when it expects high information value. Under VERIFIER_PASS, a grounded PASS can terminate immediately.

The official lifecycle in core/self_evolving_loop.py:241-277 lets FAIL and optionally PASS produce same-actor learning. Lines 307-331 route FAIL into targeted evolution, and lines 333-383 handle curriculum review, fresh-target requirements, and the final test after a stall. UNVERIFIED terminates as a harness condition instead of being rewritten as a semantic failure.

flowchart TD
    Q[Task direction q] --> C[Curriculum Agent]
    M0[Memory M0] --> C
    C --> S[Snapshot memory before a BRS wave]
    S --> B1[Branch A: isolated VM]
    S --> B2[Branch B: isolated VM]
    S --> B3[Branch C: isolated VM]
    B1 --> E1[Verifier report A]
    B2 --> E2[Verifier report B]
    B3 --> E3[Verifier report C]
    E1 --> D1[Same Actor distills A]
    E2 --> D2[Same Actor distills B]
    E3 --> D3[Same Actor distills C]
    D1 --> K[Serial audit and canonical memory commit]
    D2 --> K
    D3 --> K
    K --> C
    C -->|saturated or budget exhausted| T[Attempt target with inherited memory]
    T --> V[Grounded verdict]
    V -->|FAIL| P[Targeted practice]
    P --> U[Same Actor updates memory]
    U --> T
    V -->|PASS with optional curriculum review| T
    V -->|PASS or stalled final test| Z[Freeze memory]
    V -->|UNVERIFIED| Z
    Z --> F[Fresh Actor evaluation with learning disabled]
  

5. What the frozen-memory test does and does not establish

After DRS terminates, the runner freezes the memory tree, records its hash and contents, and transfers it to a reset environment. The final actor starts without the exploration actor's live interaction history. The curriculum and host writeback are disabled. This is a stronger test than letting the same actor finish the same open session, because it asks whether the retained artifact helps a fresh attempt.

It is still a known-target test. The target query guided both broad and deep exploration, and the final evaluation is on that same target. The paper's appendix explicitly calls this known-target adaptation. The result should not be generalized to an unseen task that shares only a vague domain, nor to an agent that must decide which target to pursue without one being supplied.

Memory size is not used as a proxy for quality. In the case studies, T044 grows from 109,647 bytes after 10 BRS projects to 316,541 bytes at freeze; T049 ends at 141,312 bytes; T065 ends at 281,085 bytes. The history also contains revision and compression. T065's third BRS consolidation shrinks the memory from 13,243 to 11,510 bytes, showing that useful learning may remove redundancy rather than append another document.

The distinction between a transport failure and a task failure is part of the protocol. An environment crash can stop a run, but it should not be distilled into a claim about software behavior. The same principle applies to an unresolved verifier report: the lifecycle returns UNVERIFIED for external measurement instead of pretending that the ambiguous evidence produced a factual lesson.

6. Main benchmark evidence

The primary experiments cover 82 OSWorld 2.0 offline tasks and all 67 Near-term Agents' Last Exam tasks. The default actor is GLM-5.3; Kimi-K3 is used in separate verifier and curriculum contexts. BRS has a nominal eight-project budget with at most four branches running concurrently, while DRS proceeds sequentially. Official scores and hidden checks remain unavailable to all learning roles.

Against the paper's own no-RSI harness, the reported OSWorld Partial score rises from 71.97 to 78.98 and Binary rises from 37.80 to 42.68. ALE rises from 83.75 to 84.82 on Partial and from 49.25 to 50.75 on Binary. The direction is consistent at the aggregate level, and the OSWorld absolute gain is large enough to motivate the memory-construction design.

System or configuration	OSWorld Partial	OSWorld Binary	ALE Partial	ALE Binary
GPT-6 Astra	72.60	Not reported	82.26	52.24
Claude Opus 5	70.19	34.72	79.54	46.27
RSIAgent without RSI	71.97	37.80	83.75	49.25
RSIAgent	78.98	42.68	84.82	50.75

RSIAgent reports the highest Partial scores among the compared systems: 6.38 points above GPT-6 Astra on OSWorld and 2.56 points above it on ALE. The comparison is not a clean sweep. GPT-6 Astra's ALE Binary score is 52.24, 1.49 points above RSIAgent. A precise summary should say that RSIAgent leads the compared Partial metrics while the strongest ALE full-success result still belongs to GPT-6 Astra.

The aggregate construction also limits what can be inferred from a single number. The OSWorld RSI row replaces 41 baseline task scores and retains 41. The ALE row replaces only 19 and retains 48. T082 is counted as zero because its setup failed. The ALE aggregate includes locally corrected grades, an ECG result with public-label-transfer qualifications, and a Tax Form variant without BRS. These choices are disclosed, but they mean the result is not a clean all-task run of the full method under one budget.

7. Progress curves, stage ablation, and game development

Three OSWorld trajectories make memory growth visible. T044, T049, and T065 reach Partial scores of 100%, 80%, and 100% by exploration step 8. The useful detail is not only the endpoint. T049 falls during an intermediate attempt, receives additional practice, and later recovers. This is consistent with the paper's claim that failures can produce useful memory, while also showing that the improvement path is not monotonic.

The four-task ablation isolates BRS and DRS. Full RSI averages 74.54%, compared with 65.52% for BRS-only and 56.50% for DRS-only. BRS improves every task over the recorded baseline, while DRS-only falls below baseline on T085 and T089. The result supports a sequence: broad experience supplies reusable procedures and failure patterns, then deep practice exploits that base to test difficult boundary conditions.

Task	No RSI	No BRS, DRS only	No DRS, BRS only	Full RSI
T080	5.00	60.30	55.46	68.69
T085	68.00	60.06	91.07	94.15
T089	67.50	64.00	77.00	81.25
T106	36.01	41.66	38.55	54.06
Mean	—	56.50	65.52	74.54

This ablation is exploratory rather than a conventional randomized study. The four tasks were selected from cases with recorded improvements, Full RSI averages two historical frozen-memory evaluations, and the stage conditions use archived runs. T085's no-BRS result uses the completed infrastructure-replacement run. The pattern is evidence for the architecture, but it is not a seed-averaged estimate of each component under an identical compute budget.

The GameCraft-Bench transfer experiment samples 40 tasks and freezes the base game for each generator comparison. Full RSIAgent improves overall quality for all four base generators: 52.77 to 61.28 for GPT-5.5, 31.28 to 46.37 for Kimi-K2.6, 30.55 to 48.72 for GLM-5.3-Flash, and 41.30 to 57.46 for Qwen3.8-27B. Play2Code improves the weaker three generators but lowers GPT-5.5 from 52.77 to 51.05.

Base generator	Baseline	Play2Code	RSIAgent without RSI	RSIAgent
Codex + GPT-5.5	52.77	51.05	57.84	61.28
Kimi-K2.6	31.28	36.02	42.61	46.37
GLM-5.3-Flash	30.55	38.25	44.73	48.72
Qwen3.8-27B	41.30	47.67	53.82	57.46

Each development run allows at most 15 iterative improvement rounds and 26 actor tool calls per round. GLM-5.3-Flash serves as both development actor and verifier, while Qwen3.8-27B plays the games and performs final evaluation. The shared development/verification model is a practical cost choice, but it weakens the independence argument that the benchmark harness otherwise emphasizes.

8. Failure analysis: how bad knowledge survives a loop

The first failure mechanism is insufficiently targeted exploration. A curriculum may add document-processing, conflicting-information, and persistence tasks while never testing the unavailable-user-information condition that actually determines the target. Practice volume grows, but the curriculum avoids the assumption most likely to invalidate memory. The audit reports exploration mismatch in 3/4 inspected cases and unchallenged assumptions in 1/4.

The second mechanism is incomplete verification. A verifier can see that an artifact exists and still miss that a field lacks evidence, an interpretation conflicts with the rubric, or one requirement was never checked. The audit reports verification fidelity problems in 3/6 cases, grounding problems in 2/6, and coverage problems in 1/6. These categories overlap because a weak evidence chain can support both a false PASS and an overconfident lesson.

The third mechanism is unreliable memory consolidation. An actor may turn a missing-data marker into a valid answer, interpret unavailable information as a negative response, or drop the scope condition under which a local rule was observed. The audit finds memory-scope loss in 2/3 cases and failure to enforce uncertainty in 1/3. Once written into a reusable procedure, such a rule can influence several later attempts without being challenged by the target itself.

These failures compound. Poor exploration fails to produce disconfirming evidence; incomplete verification labels a local success as correct; memory consolidation then removes the uncertainty that might have limited reuse. The next curriculum sees an apparently grounded lesson and is less likely to design an experiment that challenges it. This is why adding more self-critique without strengthening evidence boundaries can make a self-improving loop more efficient at preserving the wrong thing.

9. Author-stated and independent limitations

The authors name four limitations. Recursive self-improvement requires additional test-time exploration and computation. Its outcome depends on finite budgets, stopping policies, and the quality of the learned memory. A model-based verifier can make incorrect judgments that propagate into later exploration and memory updates. Different environments need different tools, verification signals, and exploration strategies, and the current experiments do not fully isolate every component's contribution.

An independent limitation is scope. The exploration query and final evaluation use the same target, so the experiments study known-target adaptation. They do not show that the learned memory transfers to an unseen task with a different interface, or that the curriculum can infer an entirely new objective without a target direction. The paper is explicit about this boundary, and it should remain in any practical claim.

A second independent limitation is comparison quality. Published systems use their own harnesses and execution budgets; the cross-system table was not produced by a matched protocol. Within RSIAgent's own rows, the RSI aggregate includes selected retries, checkpoints, locally corrected scores, and baseline-retained tasks. Reporting a strong aggregate is valid only if the mixed construction is read alongside the per-task details.

A third limitation is reproducibility. The repository confirms the control flow and role boundaries, but it postdates the paper and does not expose every result receipt, model configuration, or environment budget needed to repeat the reported aggregates exactly. The code inspection is strong evidence that the snapshot/commit protocol exists; it is not proof that every benchmark number follows from one clean run.

The result also leaves the verifier bottleneck unresolved. The same paper that demonstrates gains from autonomous memory construction shows that verification fidelity and memory scope are major failure sources. A production system would need calibrated uncertainty, tamper-resistant evidence, and a mechanism for retiring rules whose supporting conditions disappear. Those are not optional cleanup steps; they are part of what makes recursive improvement safe enough to reuse.

10. Conclusion: governance is part of the learning algorithm

RSIAgent contributes a useful decomposition of training-free adaptation. Broad exploration builds a map; deep exploration attacks a grounded weakness; the verifier constrains what counts as evidence; the actor retains the right to revise durable knowledge; and the frozen-memory test separates learning from final execution. The design turns self-improvement into an auditable system process rather than a single model asked to “reflect” on its answer.

For agent and embodied-AI developers, the transferable lesson is to treat memory as governed state. Preserve preconditions, evidence, uncertainty, failure scope, and provenance. Parallelize experience acquisition only when branches start from a stable snapshot and their conclusions can be reconciled deterministically. Evaluate whether the final actor succeeds with frozen memory, not whether the exploring actor can finish while the whole learning loop remains attached.

Golden quote: Recursive self-improvement is not the accumulation of more experience; it is the disciplined process that decides which evidence is allowed to rewrite future behavior.

Primary sources: arXiv:2609.15364, the project page, and the official implementation snapshot.