ModularRSI: Modular and Generalizable Recursive Harness Self-Improvement

Background and Motivation

An agent does not operate through its foundation model alone. The surrounding harness determines how model responses become tool calls, how terminal feedback is returned to the model, how conversation history is compressed, and when a task is considered complete. This layer converts raw model capability into long-horizon behavior, and its design can matter as much as the model itself.

Recursive self-improvement attempts to let an agent modify that harness from execution experience. Existing terminal and coding agents show that such changes can improve performance, but the paper identifies a generalization problem. Many systems evolve directly on evaluation benchmarks or benchmark subsets. Their final scores may therefore measure useful reimplementation of executable mechanisms, or they may simply measure adaptation to the exact tasks that supplied the feedback.

A second problem appears at the trajectory level. A single success may contain a task-specific shortcut, while a single failure may mix a harness defect with a model reasoning error or a peculiar repository issue. Updating the whole harness from one trajectory can turn a local workaround into a permanent general policy. This makes transfer to unseen tasks unreliable.

The third problem is credit assignment across mechanisms. A failure such as premature termination could be addressed in the agent loop, the task-completion detector, the context manager, or the tool layer. Rewriting the full harness creates a large modification space and changes many interacting behaviors at once. ModularRSI responds with three constraints: benchmark-disjoint evolution data, same-task contrastive diagnosis, and module-restricted code changes.

Benchmark-Disjoint Evolution Data

ModularRSI does not evolve on instances from the downstream evaluation sets. Instead, human annotators first extract high-level category labels from the Terminal-Bench and SWE-Bench families. Those labels guide retrieval from independent public sources such as GitHub repositories, Hugging Face, Kaggle, and Linux kernel documentation. The retrieved material is then reconstructed into new executable Harbor tasks.

The resulting pool contains 2,000 tasks: 1,000 terminal tasks and 1,000 software-engineering tasks. Each task goes through three quality filters. Environment completeness checks that required data, dependencies, files, and tools are present. Practicality and non-triviality reject artificial or overly simple cases. Evaluator validity requires a functional test, rather than a superficial completion signal.

The construction pipeline also runs the reference solution to ensure that it passes every task-specific test, and it runs a no-op submission to ensure that no reward is given for doing nothing. Human review then checks task clarity, environment completeness, and evaluator correctness. Finally, semantic similarity screening removes instances that overlap too strongly with downstream benchmarks.

Category	SWE tasks	Terminal tasks	Total	Share
Data processing	219	162	381	19.1%
ML and science	104	168	272	13.6%
Build and dependencies	133	111	244	12.2%
Systems and OS	102	134	236	11.8%
Language and API	186	41	227	11.3%
Web and network	136	53	189	9.4%
Algorithms	162	14	176	8.8%
Security and crypto	49	101	150	7.5%
Database	57	68	125	6.2%

Table 1 reproduces the category counts from Figure 2 of the paper. The pool balances terminal interaction with repository-level software engineering. It is not intended to be a conventional supervised training set. It is an executable environment in which the harness can gather experience, compare outcomes, and test whether a proposed mechanism change survives validation.

Five Functional Modules

The paper starts from Terminus-2 in Harbor but limits evolution to mechanisms that directly mediate agent-environment interaction. Sandbox initialization, parallel execution, and model communication are treated as infrastructure and remain outside the editable scope. The behavioral harness is divided into five modules.

The Agent Loop controls the iterative reasoning-action-observation process. It maintains execution state, schedules module calls, handles retries, and decides whether to continue. Observation Management converts terminal and environment output into model-readable feedback while filtering noise and preserving relevant evidence. Tool Use parses model responses into commands, completion signals, and tool calls, then executes and validates them.

Context Management maintains and organizes interaction history, including retention, compression, and retrieval. Task Completion Detection reads the current state and recent evidence to recommend whether execution should stop. The Agent Loop coordinates the other modules, but the architecture does not prescribe a single fixed call order.

Module	Input	Output	Failure class
Agent Loop	Task, conversation, four peer modules	Status, final text, failure tag	Loops, weak recovery, premature stop
Observation Management	Environment and terminal state	Readable observation text	Lost logs and incremental-output stubs
Tool Use	Model response and tool call	Commands, completion signal, result	Parse errors and multiline dispatch failures
Context Management	Conversation and original task	Updated history and handoff prompt	Dropped constraints and context overflow
Completion Detection	Loop state and recent signals	Stop recommendation and reason	Accepting unsupported completion claims

Table 2 describes the behavioral contract behind the module boundaries. The decomposition follows responsibility rather than file layout. Completion Detection only recommends a stop; the Agent Loop keeps control of the actual state transition. This separation lets a detector evolve while the loop continues to enforce global execution rules.

The function library adds two management mechanisms. Function Merge compares descriptions and behavior and removes redundant implementations. Task-Aware Function Composition chooses a task-relevant subset of functions for each problem. The library can therefore grow across generations while the active prompt and tool set for any one task remains compact.

Contrastive Trajectory Diagnosis

For every evolution instance, the system runs the agent K times. Let $r_{ik}$ be the binary reward of rollout k on task i. Tasks are grouped by whether all rollouts succeed, all rollouts fail, or both success and failure occur. The paper's group definition can be written as:

$$ G_i= \begin{cases} \mathrm{Positive}, & \sum_{k=1}^{K}r_{ik}=K,\\ \mathrm{Contrastive}, & 0<\sum_{k=1}^{K}r_{ik}<K,\\ \mathrm{Negative}, & \sum_{k=1}^{K}r_{ik}=0. \end{cases} $$

The equation is not a smoothed reward estimate. It identifies whether a same-task contrast exists. In the Contrastive group, a successful and failed rollout share the same task and initial harness but diverge during execution. That divergence supplies a stronger causal clue than comparing unrelated tasks. In the Negative group, the system searches Trajectory Memory for a historical success. If none exists, it performs a single-sided diagnosis for explicit failures such as repeated loops, wrong tool use, ineffective recovery, or premature termination.

The Positive group is not discarded. A task can succeed while wasting many steps, repeating exploration, or invoking tools without progress. The diagnosis therefore asks whether a module change could preserve success while making execution leaner. Each finding records the task, target module, observed divergence, counterfactual outcome, and a proposed change. A proposed change is allowed only if a concrete module modification could plausibly move a failure toward success.

Trajectory Memory retains historical trajectories and rewards across evolution epochs. The paper reports the following evidence distribution across five runs and later replay: 732 groups, or 40.67%, contain a current success-failure pair; 150 groups, or 8.33%, are all-fail groups that can be paired with a historical success; 262 groups, or 14.56%, fail without a historical success; and 648 groups, or 36.00%, succeed in every current rollout.

Evidence type	Count	Share	Analysis path
Current success-failure pair	732	40.67%	Direct same-task contrast
All-fail group with historical success	150	8.33%	Cross-epoch contrast
All-fail group without historical success	262	14.56%	Single-sided diagnosis without forced attribution
All-success group	648	36.00%	Efficiency and redundant-work analysis

Table 3 comes from Table 9 of the paper. Across evolution, the fraction of immediately usable success-failure pairs falls from 38% in epoch 1 to 34.17% in epoch 3, a decrease of 7.50 percentage points. The authors interpret this decline as evidence that reusable improvements exposed by repeated contrasts are being absorbed into the harness, reducing the frequency of mixed outcomes on the same task.

Module Modification and Validation

Structured findings do not directly become code changes. The system first merges semantically similar diagnoses that target the same function. Each candidate is then ranked by the number of distinct tasks that support it. If C is a candidate and S(C) is its supporting task set, the evidence score can be written as:

$$ \operatorname{Vote}(C)=|\{x_i\mid x_i\text{ supports }C\}|. $$

The vote reduces the influence of a single vivid trajectory. A repair that only works for one benchmark case is unlikely to receive support from independent tasks. An Evolution History stores previous revisions and the functionality they introduced, helping later edits preserve prior improvements and avoid oscillating between conflicting mechanisms.

Every modification passes through three gates. The program check runs AST validation, import checks, protocol compliance, discovery-contract verification, and static self-attribute audits. The diff review is reward-blind and rejects task names, task-specific files, output strings, or constants that fit only one instance. Execution validation samples two tasks from the current batch and runs the modified harness; runtime errors or protocol violations trigger a rollback.

$$ A(m)=P(m)\cdot D(m)\cdot E(m),\qquad A(m)\in\{0,1\}. $$

Here $P(m)$, $D(m)$, and $E(m)$ indicate whether the program check, diff review, and execution validation pass. This equation formalizes the paper's textual gate description. A modification is retained only when all three conditions hold. The gates therefore test not only executability, but also whether the change appears general enough to leave the evolution instances.

After all five modules evolve independently, a Cross-Module Integration epoch combines them. Independent variants may duplicate functions, conflict in call order, or interact in inconsistent ways. The integration agent first tries to retire redundancy, then merge overlapping variants, then perform a small in-place repair. Once integration finishes, the function library is frozen for downstream evaluation.

Function Merge and Task-Aware Function Composition control growth. For a candidate function library $\mathcal F$, the composer selects a task-dependent subset $\mathcal S(x)\subseteq\mathcal F$ using natural-language descriptions. The lower-level library can expand while the active harness remains small. This is an important engineering constraint because unrestricted prompt and tool growth can erase the benefits of new variants.

flowchart LR
  A[Evolution pool
2000 disjoint tasks] --> B[K rollouts per task]
  B --> C{Reward group}
  C -->|all pass| D[Efficiency diagnosis]
  C -->|mixed| E[Paired success-failure contrast]
  C -->|all fail| F[Historical success or single-sided diagnosis]
  D --> G[Structured findings]
  E --> G
  F --> G
  G --> H[Five independently evolved modules]
  H --> I[Program check
Diff review
Execution validation]
  I --> J[Cross-module integration]
  J --> K[Frozen harness]
  K --> L[Terminal-Bench 2.0]
  K --> M[SWE-Bench Verified]

Experimental Protocol

The main evaluation uses Terminal-Bench 2.0 and SWE-Bench Verified. Terminal-Bench 2.0 contains 89 long-horizon terminal tasks, while SWE-Bench Verified contains 500 human-validated repository tasks. The original released benchmark versions are evaluated under Harbor. The main evolution models are DeepSeek-V4-Flash-Preview and DeepSeek-V4-Flash-0731, with a 2 million token-per-minute limit and an evolution batch size of 10.

The paper selects 120 terminal and 120 software-engineering instances from the 2,000-task pool because the full evolution run is computationally expensive. Each module or joint configuration is evolved for three epochs. Evaluation reports average accuracy, Pass@3, Pass3, and average step count. These metrics separate average capability, breadth of search, repeated-run stability, and interaction efficiency.

Average accuracy is the mean reward across all tasks and rollouts:

$$ \mathrm{Acc}=\frac{1}{NK}\sum_{i=1}^{N}\sum_{k=1}^{K}r_{ik}. $$

Pass@3 measures whether at least one of three independent attempts succeeds, while Pass3 measures whether all three attempts succeed:

$$ \mathrm{Pass@3}=\frac{1}{N}\sum_{i=1}^{N}\mathbf 1\!\left(\bigvee_{k=1}^{3}r_{ik}=1\right), $$ $$ \mathrm{Pass}_{3}=\frac{1}{N}\sum_{i=1}^{N}\prod_{k=1}^{3}r_{ik}. $$

Average step count is the mean number of agent-environment interactions per rollout:

$$ \mathrm{StepNum}=\frac{1}{NK}\sum_{i=1}^{N}\sum_{k=1}^{K}T_{ik}. $$

These metrics prevent a single average from hiding reliability failures. A method can raise average accuracy by making one lucky rollout work while lowering Pass3. A method can also raise Pass@3 without improving the consistency of all three attempts. ModularRSI improves average accuracy, Pass@3, and Pass3 on the main Terminal-Bench comparison, while its average step count rises only slightly from 34.70 to 35.57.

Main Results

The first experiment tests transfer beyond the evolution domain. With DeepSeek-V4-Flash-Preview, the unmodified harness reaches 47.57% on Terminal-Bench 2.0 and 73.40% on SWE-Bench Verified. A harness evolved only on terminal tasks reaches 52.43% on unseen terminal tasks and 75.80% on software engineering. A harness evolved only on software-engineering tasks reaches 76.45% in-domain and 49.40% on terminal tasks.

Evolution set	Evaluation setting	SWE accuracy	SWE Pass@3	SWE Pass3	TB2.0 accuracy	TB2.0 Pass@3	TB2.0 Pass3
No evolution	Baseline	73.40	83.20	62.80	47.57	58.43	30.34
Terminal tasks	Cross-domain and in-domain	75.80	84.67	66.20	52.43	65.17	35.96
SWE tasks	In-domain and cross-domain	76.45	85.30	66.80	49.40	60.67	30.34

Table 4 reproduces Table 2 of the paper. The key result is bidirectional transfer. Terminal evolution raises the unseen SWE score by 2.40 percentage points, and SWE evolution raises the terminal score by 1.83 points. The frozen harnesses never saw the evaluation instances during evolution. Terminal-Bench Pass3 also rises from 30.34% to 35.96%, which indicates that the gain is not limited to one favorable rollout.

Cross-model transfer freezes the terminal-evolved harness and changes only the inference model. GLM-5.2 improves from 59.55% to 61.80%, MiniMax-2.5 from 41.57% to 44.94%, and DeepSeek-V4-Flash from 47.57% to 52.43%. Pass@3 and Pass3 improve for all three models. Because models differ in planning, tool use, and recovery, this consistency is evidence that the modified mechanisms are not solely exploiting one model's behavior.

The modularity study is equally important. Non-modular evolution falls to 46.44% accuracy, and jointly evolving all modules falls to 44.19%. The independently evolved and integrated version reaches 52.43%. Individual modules improve the baseline but vary in their effects: Agent Loop gives the highest single-module accuracy at 50.56%, while Observation Management reduces average step count to 22.50.

Method	Accuracy	Pass@3	Pass3	Mean steps
Baseline	47.57	58.43	30.34	34.70
Non-modular evolution	46.44	64.04	24.72	29.03
Joint all-module evolution	44.19	61.80	24.72	44.34
Context Management only	49.44	61.80	31.40	35.10
Tool Use only	50.19	62.92	30.34	41.28
Agent Loop only	50.56	64.04	34.83	40.40
Observation Management only	49.81	65.17	33.70	22.50
Completion Detection only	49.44	65.17	31.40	31.06
ModularRSI integrated	52.43	65.17	35.96	35.57

Table 5 combines results from Tables 4 and 5 of the paper. Every single module improves on the baseline, but no module alone reaches the integrated result. That pattern supports complementary effects rather than a single dominant patch. Joint evolution uses fewer steps than the integrated version but loses substantial accuracy, showing that interaction efficiency cannot substitute for satisfying the task requirements.

The controlled comparison with prior RSI methods uses 120 evolution instances, the same Terminus-2 starting harness, and DeepSeek-V4-Flash-0731. The baseline reaches 61.79%, Meta-Harness 62.92%, AHE 62.54%, and ModularRSI 67.42%. These absolute values should not be compared with the earlier 47.57% result because the model version and protocol differ. Within the controlled table, ModularRSI's improvement over both baselines is more than four accuracy points.

Figure 6 shows accuracy rising across generations and reaching 52.2 at generation 22, while the trajectory judge score reaches 74.0. The two curves are related but not identical, which confirms that task success and observed execution quality are different signals. A separate difficulty study finds 76.45% on SWE-Bench Verified with a medium-centered distribution and 74.25% with a hard-plus-easy distribution. Moderate tasks are more useful because they produce both successful and failed evidence, whereas easy tasks lack failures and extremely hard tasks lack successes.

Code and Implementation Evidence

The official repository includes a sanitized runnable generation with 20 registered implementations: three agent loops, two observation modules, two context managers, five tool modules, two verification modules, and six solver helpers. The paper's function library is therefore visible in code rather than being only an abstract design.

The completion-integrity guard appears in `generations/merged_active/gen_0/modules/agent_loop/planning_with_guard.py`. The loop extracts requirements from the original instruction and maintains completed and pending items. If the model declares completion while requirements remain, the loop resets the consecutive completion signal and sends the pending list back to the model instead of entering the baseline two-phase confirmation.

# generations/merged_active/gen_0/modules/agent_loop/planning_with_guard.py
if pending_text or (self._acceptance_checklist and zero_cmd):
    self._pending_completion_rejections += 1
    state.consecutive_complete_signals = 0
    prompt_parts = [
        "You declared the task complete, but the following requirements are "
        "still pending:\n\n",
    ]
    return prompt, pending_completion

That behavior matches the FFmpeg case study. The task requires `ldd` output to contain `libavcodec`, `libavformat`, and `libx264`. The original run finds only `libx264` and declares completion twice without a command. After the guard is evolved, the loop repeats the acceptance checklist and makes the agent run a verification script. All three libraries appear, and all ten external tests pass.

Tool robustness is implemented in `combined_robust.py`. The module routes `write_file` and `edit_block` by their first token, bypasses the shell metacharacter check for their content, and passes the remaining text as a multiline file body. This prevents source code from being rejected or sent to the shell as an executable command.

# generations/merged_active/gen_0/modules/tools/combined_robust.py
parts = ks.split(maxsplit=1)
cmd_name = parts[0] if parts else ""
if cmd_name in ("write_file", "edit_block"):
    remainder = parts[1]
    path, content = remainder.split(maxsplit=1)
    # helper receives the raw multiline remainder as content
    return helper.run([path, content], ctx)

This explains the Zip Slip improvement. The original tool layer sends multiline Go source to the shell and receives `write_file: command not found`. The evolved helper writes 2,204 bytes directly. All three rollouts still pass 60 tests, but episode counts fall from 41, 24, and 43 to 26, 18, and 31, reducing the mean from 36 to 25.

Observation Management has a similarly direct implementation. `terminal_scrollback.py` reads the full terminal scrollback instead of only the increment since the previous poll. It merges unflushed bytes, keeps an accumulated fallback, prefixes the observation with a full-history marker, and suppresses stale repeats. That implementation corresponds to the large StepNum reduction observed for Observation Management in Table 5.

Case Studies and Evidence Boundaries

The first case study ports a Scala sales pipeline to PySpark. The written task excludes NULL categories, while the old Scala code does not. A successful rollout adds `isNotNull()` and produces 47 rows without null categories. A failed rollout follows the legacy implementation, produces 52 rows, and retains five null categories. The proposed change is not a filter for that pipeline, but a persistent task checklist that keeps explicit requirements in view.

The second case builds FFmpeg 0.10.16 with three required shared libraries. All three rollouts fail because the agent sees only one required library and declares completion without a fresh check. The Negative diagnosis produces an evidence-gated completion guard. Later rollouts pass after the loop repeats the acceptance checklist and the agent writes an explicit verification script.

The Zip Slip case studies a successful but wasteful group. All rollouts pass, yet the tool dispatch error makes the agent try several alternative write strategies. The Positive diagnosis proposes routing content-bearing helpers before generic shell dispatch. Later rollouts retain success while reducing episode counts. The paper notes that these later runs contain other updates, so the cases show the intended behavior rather than isolating one change's causal effect.

Limitations and Risks

The authors explicitly state that they do not conduct a dedicated ablation that isolates contrastive trajectory analysis. The changing contrastive-pair ratio and the case studies support its usefulness, but they do not replace a controlled removal experiment. The authors also note that main evolution experiments use only 120 terminal and 120 SWE instances from the 2,000-task pool because of computational cost.

A second limitation is the breadth of validation evidence. Two downstream benchmarks and a limited set of foundation models show transfer, but they do not establish long-term safety or reliability in production repositories. The improvement in average accuracy is meaningful, yet it appears alongside a small increase in average steps, so the method does not dominate every efficiency metric.

The completion guard also relies on keyword overlap. A short requirement can be marked complete after one matching keyword, while longer requirements generally need two. This heuristic is designed to push the model toward additional verification, not to act as a formal proof system. The anti-hang safety valve eventually falls back to the baseline two-stage gate, which reduces the risk of infinite rejection but weakens the strictness of the evidence check.

Finally, Task-Aware Function Composition is itself an LLM decision based on natural-language function descriptions. A missing or misleading description can activate the wrong variant. The paper does not report separate measurements for composer error rate, prompt-token overhead, function-library search latency, or the cost of running the three validation gates. These are important engineering costs for anyone attempting to deploy the method continuously.

There is also a reviewer-level concern: the code-modifying agent performs diagnosis, implementation, and diff review. The diff review is reward-blind, which helps, but the same model family may share blind spots across those stages. Independent verification models, adversarial tests, and longer-horizon regression suites would strengthen the gate design.

Conclusion

ModularRSI converts harness self-improvement from a large, opaque rewrite into a constrained search over five behavioral modules. Its benchmark-disjoint pool reduces the risk of memorizing evaluation instances. Same-task success-failure contrasts turn noisy task outcomes into localized evidence. Program checks, reward-blind diff review, and execution validation filter code that is broken or task-specific. Cross-module integration then combines complementary changes, while function merging and task-aware composition manage complexity.

The results show that the frozen harness improves unseen in-domain and cross-domain tasks and transfers to different foundation models. The broader lesson is methodological: recursive self-improvement needs a generalization protocol before it can support strong claims about autonomous improvement. The next useful experiments should isolate contrastive analysis, scale beyond 120 evolution instances, measure cost and safety, and test whether the same mechanisms survive in production environments over long maintenance horizons.