RRSI: Regularized Recursive Self-Improvement of Agent Harnesses

How RRSI Regularizes the Search

Figure 1: RRSI separates proposal-side control of search capacity from selection-side control over which improvements become permanent harness state.

Proposal side: annealed update cardinality

An unconstrained proposer can bundle unrelated prompt, tool, control-flow, and memory edits into one candidate. The bundle may fit the current feedback unusually well, but an observed gain cannot be assigned to any one mechanism. RRSI therefore caps the number of independently attributable edits in a candidate.

At round $t$ of a $T$-round run, the budget follows a cosine schedule from $b_{\max}$ to $b_{\min}$:

Early rounds may combine several coordinated changes to discover a mechanism. Later rounds become sparse. If the drafting pool is $E_t$ and $z_t\in\lbrace0,1\rbrace^{\lvert E_t\rvert}$ marks included atomic edits, the implementation enforces:

This is a bound on update sparsity, not on the components a harness may eventually contain. The edit vocabulary can change between rounds, and there is no continuous $L_0$ penalty.

Proposal side: falsified hypotheses remain evidence

Every measured edit is tagged with its component, hypothesis, source diff, score change, cost change, and outcome. The accumulated history can be summarized as:

The flag $a_i=1$ means that the edit belonged to the candidate selected as the winner of its round. Rejected mechanisms remain negative evidence, so later proposers can avoid spending capacity on hypotheses that earlier evaluations already falsified.

The history also supports pruning. For each component $\ell$, RRSI computes the best measured gain inside a recent window:

If $g_t(\ell)\le 0$, the component enters a pruning target set. The proposer receives that target together with existing accepted machinery and is asked to remove structure that has stopped paying for itself.

Proposal side: structured exploration after a stall

The history also reveals when the proposer has collapsed onto one family of changes. RRSI marks a stall when progress over $w$ rounds remains inside the empirical noise band:

If $T_t$ is the set of components with measured edits and $\mathcal{K}$ is the editable vocabulary, then $U_t=\mathcal{K}\setminus T_t$ contains components never exercised. During a stall, a reserved proposal slot is directed toward $U_t$.

The vocabulary includes prompts, control flow, configuration, output plumbing, context management, client tools, skills, memory, and subagents. This resembles diversity or entropy regularization, but it changes where proposal capacity is spent rather than banning any component from the final harness.

Selection side: screen leakage before evaluation

A critic reads the candidate diff before full evaluation. It rejects changes that explicitly encode task names, entity names, task-specific values, answers, or other logic specific to the evolve suite. It also rejects inert machinery added only to look sophisticated.

The timing is essential. If a leaking candidate were evaluated first, its inflated evolve score could enter the history and steer later proposals. Screening the diff before scoring prevents that contamination. The critic targets benchmark-specific logic, not a component category, so generic prompt and tool improvements remain eligible.

Selection side: a floor above the noise band

Before evolution, the unchanged base harness is repeatedly evaluated to estimate an empirical noise tolerance $\delta$. If $S^\star$ is the best score observed so far, every candidate must satisfy:

This prevents a sequence of individually small regressions from becoming a larger downhill walk. A candidate can fluctuate within the calibrated noise band, but it cannot indefinitely lower the running frontier.

Selection side: performance gains must pay for cost

Relative score and cost changes are:

When the gain exceeds the noise band, the candidate must satisfy:

The intercept $\beta_0$ permits a small cost increase for a negligible gain, while the slope $\beta_1$ determines how much additional token use a larger measured gain can justify. This rule directly blocks candidates that raise accuracy slightly by making the context and trajectory much longer.

Selection side: within-noise candidates need another reason

If a candidate passes the stability floor but its score change remains inside the noise band, a small score increase is not treated as sufficient evidence. RRSI instead applies:

Here $\nu_t(H')$ counts structural component types touched by the candidate that have never appeared in a winning edit. The novelty term covers client tools, skills, memory, and subagents. Lower cost contributes positively through $-w_c\Delta C$, and an untested structural mechanism contributes through the novelty term.

In the coding instance, $w_s=0$. A within-noise score increase therefore cannot admit a candidate by itself; the candidate must save cost or explore a new structural mechanism.

The unified transition

A regularized round can be written as:

If no candidate is admissible, the incumbent remains. The proposal distribution controls how search capacity is used; the admissible set controls which evidence is strong enough to become permanent.

flowchart TD
    A[Incumbent harness H_t] --> B[Run evolve tasks and analyze failures]
    B --> C[Propose edits under annealed budget]
    D[Full edit history] --> C
    C --> E[Redirect stalled search to untried components]
    E --> F[Leakage critic reads source diff]
    F -->|reject| R[Skip full evaluation]
    F -->|pass| G[Measure score and policy tokens]
    G --> H{Pass best-score minus noise floor}
    H -->|no| R
    H -->|yes| I{Gain exceeds noise band}
    I -->|yes| J[Gain must pay for token growth]
    I -->|no| K[Need cost saving or structural novelty]
    J --> L[Admissible candidate]
    K --> L
    L --> M[Select highest-scoring admissible candidate]
    M --> N[Record hypothesis diff score cost and verdict]
    N --> A
    

Implementation correspondence

The public implementation keeps the search core in small, inspectable modules. The cosine budget appears directly in rrsi/schedule.py, and the proposer enforces it rather than relying on prompt instructions alone.

def edit_budget(t: int, T: int, b_min: int, b_max: int) -> int:
    v = b_min + (b_max - b_min) * 0.5 * (1.0 + math.cos(math.pi * t / T))
    return int(math.ceil(round(v, 9)))

The two acceptance branches appear in rrsi/selection.py. The following excerpt preserves the paper's distinction between a gain above the noise band and a within-noise candidate:

if delta_S > delta:
    budget = cfg.beta0 + cfg.beta1 * delta_S
    ok = delta_C <= budget
else:
    shaped = cfg.w_s * delta_S - cfg.w_c * delta_C + cfg.w_n * nov
    ok = shaped > 0

Candidates run in separate git worktrees. Accepting a candidate fast-forwards the evolution branch, so the incumbent is always a commit and every proposal, critic verdict, measurement, and reason for rejection can be audited.

Experimental Design

Coding uses Terminal-Bench 2.1, an 89-task containerized terminal suite with hidden unit tests, and tests transfer on SWE-bench Verified. The workspace instance uses Harvey LAB with 120 evolve tasks and 40 pristine in-distribution held-out tasks, then evaluates on JobBench, GDPval, and APEX-Agents.

Engineering design uses 61 EngDesign tasks with deterministic simulator grading and tests transfer on Frontier-Eng. Every main run freezes Claude Opus 4.8; the proposer, failure analyst, and leakage critic also use that model. All baselines start from the same $H_0$ with the same candidate budget.

The engineering instance is especially important because it removes judge-mediated scoring. Harvey LAB, JobBench, and GDPval all involve language-model judges, leaving room for a harness to improve by learning how to satisfy the judge. EngDesign and Frontier-Eng instead use simulators or testbenches, so a design either meets the stated constraints or it does not.

Each evolved harness and its baseline are measured in the same window with the same tools and trial count. This pairing prevents infrastructure drift from being reported as an evolution gain.

Main Results: Less Evolve-Set Winning, More Transfer

Figure 2: Main results for coding, agentic workspace, and engineering design. Evolve, in-distribution held-out, and out-of-distribution splits are shown separately.

Table 1: Main RRSI results using the paper's native metric for each benchmark. Frontier-Eng is reported in Medal points.

The most informative row is not the largest gain. In the workspace domain, RRSI improves Harvey LAB by only 1.1 points, yet raises the in-distribution held-out split by 2.3 and lifts the three-benchmark OOD average from 39.7 to 43.6.

By contrast, Meta-Harness is the strongest baseline on the evolve split at 93.0, but improves the OOD average by only 0.9. RRSI gives up some visible development-set fit in exchange for mechanisms that remain useful when the benchmark changes.

Every held-out split improves. No held-out benchmark regresses, which is the pattern expected if the regularizers suppress memorized task logic rather than merely moving noise between evaluations.

Ablation: noise and context growth are expensive

Table 2: Workspace ablation. OOD average is the mean of JobBench, GDPval, and APEX-Agents.

Unregularized evolution reaches 92.8 on the evolve split, 2.3 points above RRSI, but its OOD average falls to 40.3 and cost rises to 3.80 million policy tokens per trial. RRSI uses 2.42 million, about 36.3% fewer tokens than the unregularized arm; the abstract reports roughly 30% fewer across the broader experimental setting.

Removing proposal regularizers costs only 0.2 points on the evolve split but 1.7 OOD points. This separates two effects: directing where the search looks matters even before candidates are filtered.

Removing acceptance regularizers produces the opposite trap. The evolve score rises to 91.5, but OOD falls to 41.0 and token use rises to 3.59 million. Greedy selection is not merely choosing better mechanisms; it is accumulating noise and context overhead.

Cost and Policy Transfer

Figure 3: Final-harness policy-token cost and trajectory length. RRSI occupies the lower-cost region among evolved harnesses.

The final RRSI harness runs 26.3 steps per trial, compared with 27.3 to 34.6 for prior evolved methods. AHE is the extreme cost case at 3.82 million policy tokens per trial, about 58% more than RRSI, while scoring 4.4 OOD points lower.

Evolution still has a price. Every evolved harness costs more than the 1.56-million-token base harness, and RRSI uses 2.42 million. The regularizers decide how much test-time compute can be purchased; they do not make adaptation free.

RRSI also transfers across policy families. With Gemini 3.5 Flash as the frozen policy, Terminal-Bench 2.1 rises from 64.6 to 78.7 and SWE-bench Verified from 76.8 to 79.0. The resulting coding harness is then run unchanged with Gemini 3.1 Flash Lite, a model absent from the search; Terminal-Bench still rises from 11.2 to 14.6, a 30.4% relative improvement.

Figure 4: Evolve-set gains versus OOD transfer. RRSI favors transferable mechanisms over the largest immediate development-set score.

A second independent run gives an OOD average of 43.7 versus 43.6 in the first run. Individual tasks and benchmarks vary, but the aggregate transfer result is stable enough to rule out a single lucky search trajectory.

Limitations and Critical Assessment

Author-stated limitation: frozen weights. The study focuses on harness-level recursive self-improvement and does not address settings in which model weights change during evolution. RRSI therefore says little about joint weight-harness credit assignment, weight drift, or catastrophic forgetting.

Author-stated limitation: finite feedback and hyperparameters. RRSI still depends on a finite evolve set and several regularization hyperparameters. Its effectiveness may depend on feedback quality and the chosen search budget.

Author-stated limitation: architecture coverage. Although the experiments span domains, benchmarks, and policy models, broader validation is needed for substantially different agent architectures, tool ecosystems, and longer-running improvement processes.

From an engineering perspective, the reported cost metric covers policy tokens along evaluated trajectories. It does not fully price the proposer, analyst, leakage critic, repeated candidate evaluation, and orchestration needed to run evolution itself. RRSI can make the deployed harness leaner without proving that the complete search has amortized its own cost.

Evidence for physical robotics is also indirect. The benchmarks exercise terminal work, document workflows, and engineering design, but not continuous control, perception error, hardware safety, or sim-to-real shift. RRSI's results support software-agent harness evolution; they should not be generalized automatically to every embodied control stack.

The stability floor is deliberately conservative. It blocks accumulated regressions, but may also keep a run on a plateau longer. Structured exploration creates an escape route, yet calibrating how conservative the search should remain over long horizons is still an open design problem.

Finally, the main OOD benchmarks remain finite proxies. Their breadth is stronger than a single test suite, but they cannot certify that a harness contains no benchmark-specific mechanism outside the categories the leakage critic recognizes.

Conclusion

RRSI's contribution is not another fixed harness. It identifies recursive harness evolution itself as an optimization process that needs regularization. The method keeps edits open while limiting update capacity, retaining failure evidence, redirecting stalled search, screening leakage before evaluation, rejecting noise-level gains, and charging complexity against measured benefit.

The strongest result is the reversal at the top of the leaderboard. The harness with the largest evolve-set gain is not the harness that transfers best. A workspace harness with only a 1.1-point evolve gain produces a 3.9-point average OOD gain, while less constrained evolution fits the development split more tightly and then loses most of that advantage.

Three engineering lessons follow. Edit history should be treated as first-class evidence rather than discarded after each round. Leakage checks should happen before expensive evaluation so contaminated scores never enter the loop. Runtime cost should be part of candidate acceptance rather than a post-deployment concern.

If future systems also evolve weights, the same principles may remain useful, but atomic edits, noise estimation, and resource budgets will need to be redefined. Robotic agents would additionally need safety, hardware cost, simulation error, and long-horizon recovery in the admissible set.