PAPER DEEP DIVE
GUARD: Stabilizing Recursive Self-Improvement with Regret Guidance
Autonomous AI agents pair a frozen base model with an editable harness of prompts, skills, tool rules, and memory. Recursive self-improvement (RSI) adapts this harness from experience without weight updates, yet self-editing without external graders often backfires: reflection and scalar scores conflate what an edit newly broke with what remains unsolved, causing local regressions, skill forgetting, and memory bloat. We introduce GUARD (Grader-free Unsupervised Environment Design [UED] with Adaptive Rule Distillation), a closed-loop framework for reference-free continual harness evolution. Our key insight is to decompose each task into pre-execution self-checks and guide harness updates via directional regret. Each round grades trajectories against these checks, pinning mastered rules while prioritizing newly broken behaviors, and selects among parallel candidate edits by critic-free group-relative fitness. A Soft-HURT gate then promotes an edit only when gains outweigh regressions, scoping specialized fixes to prevent over-triggering and guaranteeing non-regression on its gated buffer. At test time, one frozen harness runs in a single pass. Across benchmarks on memory, skill evolution, continual task streams, and domain reasoning, GUARD consistently improves performance without catastrophic forgetting or memory pollution while substantially cutting deployment token usage.
Paper metadata
Title: GUARD: Stabilizing Recursive Self-Improvement with Regret Guidance. Authors: Minghe Zhang, Xuetao Kang, Yifei Ming, Yanfei Chen, Han Yu, Yufan Zhuang, and Chen-Yu Lee. arXiv identifier: 2611.23476. Paper: https://arxiv.org/abs/2611.23476. Code status: the paper does not expose an official public repository, and a repository search found no matching implementation. The authors state that reference harness code, checklist-based self-reviewers, and prompt templates will be released subject to approval.
One-sentence summary
GUARD keeps model weights frozen while continuously rewriting an agent harness, using paired directional regret to distinguish newly broken rules from old gaps and a Soft-HURT gate to accept only edits whose measured gains exceed their induced regressions.
Why this problem matters
An agent is more than its base language model. Tool declarations, task decomposition, retry policy, failure analysis, response formatting, and safety rules all live in an external harness. Recursive self-improvement promises to turn experience from one task stream into better harness behavior on the next, without retraining the model. That is attractive because harness edits are cheap, inspectable, and can be deployed immediately.
The difficulty is that an agent can improve in one direction while losing competence in another. Some systems keep appending successful habits until the harness is crowded with stale clauses that survive a task update. Other systems rewrite the prompt or skill library aggressively, fix the current failure, and erase a rule that was already working. This is the familiar plasticity-stability conflict, moved from neural network training into the agent's external control policy.
Most existing self-improvement methods assume an oracle, a stronger teacher, reference solutions, an official evaluator, or reliable environment replay. These assumptions are often false in production. Hidden task boundaries arrive without notice, writes mutate state, and prior sessions cannot always be rerun. GUARD therefore asks whether stable improvement is possible when the optimizer has no reference answer and only a limited ability to revisit experience.
The paper also makes a careful distinction between being reference-free and being unsupervised. GUARD cannot see the official answer, but it can inspect whether a trajectory respects explicit task constraints: valid tools, legal fields, correct recipients, required formats, idempotent writes, and the absence of superseded rules. Those checks form a self-review objective. The authors do not assume that this objective is identical to the hidden benchmark grader.
The engineering appeal of GUARD is that it separates diagnosis from proposal and promotion. The system first asks what the latest edit broke, then asks what remains missing, proposes a rule with explicit preconditions, and finally estimates whether the rule causes more harm than gain. Improvement becomes a controlled process with rejection memory rather than an opaque prompt rewrite.
Formal setup
For group g in round t, the harness evolves according to
$$W_{t,g}=\mathcal E(W_{t-1,g},\mathcal H_{t,g})$$
The important design question is what H_t,g contains. If a proposer receives only mixed success and failure traces, it tends to write unconditional advice such as “try another tool” or “keep working until the task is complete.” GUARD instead bundles regressions, remaining gaps, mastered invariants, positive-negative trajectory contrasts, and deduplicated rejected edits. The proposer can therefore attach a local precondition to each rule.
A completed trajectory τ receives a self-review score equal to the fraction of applicable atomic checks that pass:
$$S(\tau)=100\frac{1}{|\mathcal C^{gr}(\tau)|}\sum_{c\in\mathcal C^{gr}(\tau)}r(\tau,c)$$
For a criterion c, the harness-level obedience rate is averaged over tasks that exercise that criterion:
$$\mathrm{obey}(c;W)=\frac1{|\mathcal T_c(W)|}\sum_{\tau\in\mathcal T_c(W)}r(\tau,c)$$
These equations look simple, but they change the optimization target. A single aggregate score can hide a critical rule that just regressed. GUARD edits from criterion-level changes, allowing the analyzer to tell a newly introduced failure apart from a pre-existing capability gap.
Inside the method
At Turn 0, before the solver acts, the reviewer reads only the prompt, visible tool declarations, and schemas. It derives four to eight atomic binary predicates and freezes them for the task. Execution-level predicates cover clean sandbox execution, valid entity identifiers and columns, and idempotent mutations. Semantic predicates cover recipients, formats, prompt constraints, safety rules, and compliance with a replacement rule instead of its predecessor.
Layer 1 evaluates the deterministic conditions. Layer 2 reads the complete trajectory and grades semantic conditions. The separation is deliberate. Program checks should not be left to a language model when direct verification exists, while semantic checks need evidence from the full execution trace. Together they produce a traceable criterion vector for the Analyze stage.
Analyze computes directional regret rather than producing new text. If criterion c was satisfied under W_t-2 and fell under W_t-1, the edit hurt it. If c remains unsatisfied, it is a gap. The paper defines
$$\Delta_{\mathrm{hurt},t}(c)=\max(0,\mathrm{obey}(c;W_{t-2})-\mathrm{obey}(c;W_{t-1}))$$
$$\Delta_{\mathrm{gap},t}(c)=1-\mathrm{obey}(c;W_{t-1})$$
$$\mathrm{Priority}_t(c)=\Delta_{\mathrm{hurt},t}(c)+\Delta_{\mathrm{gap},t}(c)\in[0,2]$$
The distinction changes the proposed repair. A newly broken behavior should be guarded with a narrower precondition before adding capability. A genuine gap can be addressed with a new method derived from positive and negative traces. Rules with zero priority are pinned as mastered invariants. GUARD permits one in-round iff repair to narrow an over-broad rule, so a precondition error does not require a fresh round of candidate sampling.
Propose samples at least two candidates from the same incumbent harness, task bundle, and context. It evaluates strict and ordinary task scores, then combines them:
$$\mathrm{Perf}_m=\bar S^{strict}(W_t^{(m)})+\frac14\bar S(W_t^{(m)})$$
Fitness is standardized inside the candidate group:
$$\hat A_m=\frac{R_m-\mu_R}{\sqrt{\frac1M\sum_m(R_m-\mu_R)^2+10^{-6}}}$$
The pool has two jobs. It gives the first round a contrast signal even before paired history exists, and it reduces the chance that one noisy rollout rejects a useful edit. ContrastEvolve then explains differences among candidates that share the same starting point. This is not test-time best-of-M sampling; the pool is removed before deployment and exists only to improve credit assignment during adaptation.
flowchart TB A["Prompt
visible tool schemas"] --> B["Turn 0
freeze atomic checklist"] B --> C["Layer 1
execution and structure"] C --> D["Layer 2
semantic trace grading"] D --> E["Analyze
paired directional regret"] E --> F["Separate new hurt
old gaps and mastered rules"] F --> G["Propose
M critic-free candidates"] G --> H["Group standardization
ContrastEvolve"] H --> I["Soft-HURT
gain versus induced regressions"] I -->|"accept"| J["Install harness
pin protected rules"] I -->|"reject"| K["Deduplicated rejection ledger"] K --> F G -->|"iff precondition only"| L["In-round repair
narrow over-broad rule"] L --> I J --> M["Frozen test-time harness
no pool reviewer repair retry or gate"]
Soft-HURT compares a candidate with the incumbent on same-signature replay tasks. For tasks whose score drops, the expected harm is
$$\mathrm{Hurt}(\tilde W,W)=\frac1{|\mathcal Q_g|}\sum_{q\in\mathcal Q^-(\tilde W,W)}(S_q(W)-S_q(\tilde W))$$
Promotion is allowed only when the overall gain is greater than that harm:
$$\mathrm{Promote}(\tilde W,W)\iff\bar S(\tilde W)-\bar S(W)>\mathrm{Hurt}(\tilde W,W)$$
A candidate that raises the average but loses more on some replayed tasks than it gains overall is rejected. The implementation adds a noise allowance for finite replay. Rejected edits are deduplicated in a ledger, so later proposers see a concrete counterexample instead of repeatedly inventing the same harmful rule.
At test time, GUARD removes the candidate pool, self-reviewer, repair, retry, and gate. Every task runs once under the final frozen harness. The paper treats this separation as a validity condition: adaptation may use diagnostics and replay, but the reported benchmark result must not receive extra test-time sampling or reviewer calls. Deployment cost is bounded by a compact harness prefix, not by the number of adaptation rounds.
The theoretical guarantees must be read with the same discipline. GUARD stabilizes a self-review objective over a gated task buffer. Its theorem does not say that hidden official accuracy must improve for any distribution. The sampling guarantee is asymptotic and does not supply a non-vacuous finite-sample certificate. Those limits are central to interpreting the empirical results.
What the experiments show
On PAST-Bench, GUARD raises the overall task score from 0.372 to 0.854. The procedural task score reaches 83.0%, procedural mechanism reaches 90.0%, and procedural pollution remains 0.000. Overall pollution is 0.167, the benchmark floor. The gain is not purchased by suppressing old knowledge: update correctness reaches 0.901, and all twelve families beat the initial harness.
EvoHarnessBench uses the official hosted SDK, a unified Gemini 4 setup, and 43 test tasks. GUARD reaches 80.80% accuracy and a 58.14% strict solve rate, equivalent to 25/43. The initial harness reaches 71.91% and 48.84%. Cohort results are 78.47%, 84.17%, and 89.29%. GUARD passes 182/221 official verifier checks and eliminates eight-step budget exhaustion, reaching 0/43 where baselines exhaust 6 to 10 tasks.
The efficiency result is at least as important as accuracy. GUARD uses 59,651 tokens per task, compared with 131,032 for the initial harness, a 54.5% reduction. MCP calls fall by 22.5%. It solves 9.75 tasks per million tokens, 1.50 times the best baseline. The final playbook is only 2,315 characters, compared with 4,538 to 10,013 for competing methods. Compact conditional rules shorten trajectories instead of adding a permanent reasoning tax.
| Setting | Metric | Initial or key baseline | GUARD | Interpretation |
|---|---|---|---|---|
| PAST-Bench | Task score | 0.372 | 0.854 | Large cross-session improvement |
| PAST-Bench | Procedural mechanism | At most 78.0% among strong baselines | 90.0% | Old rules survive while new procedures are learned |
| PAST-Bench | Procedural pollution | 1.000 on aggressive arms | 0.000 | No stale clause propagation |
| EvoHarnessBench | Accuracy / strict solve | 71.91% / 48.84% | 80.80% / 58.14% | 8.89 percentage-point accuracy gain |
| EvoHarnessBench | Tokens per task | 131,032 | 59,651 | 54.5% reduction |
| Drift Gauntlet | Final mean | GEPA trails by 0.82 points | 73.22% | Best aggregate result across hidden drift |
| Drift Gauntlet | Stream-mean drops | 3 to 8 drops for evolving baselines | 0/12 | No average decrease across observed block transitions |
Drift Gauntlet stress-tests continual adaptation over four hidden drift types: workflow, topic, environment, and compound shifts. Across twelve blocks, GUARD finishes at 73.22%, gains 3.30 percentage points from the starting point, records +4.55 points of plasticity, and has 0.00 points of post-block forgetting. It has 0/12 stream-mean drops. Evolving baselines experience three to eight drops, and some high-plasticity arms lose earlier capabilities when opposing conventions arrive.
A concrete opposing-workflow case explains the mechanism. BFCL-Miss rises from 53.33% to 70.00% after one update. The next candidate attempts to cover the opposing BFCL-Base convention with a broader rule. Replay estimates equal hurt and lift at 6.25 points, so Soft-HURT rejects the edit. On ScienceWorld environment drift, the iff repair narrows an over-broad rule and raises accuracy from 74.80% to 82.73%; the following potentially conflicting update is again rejected.
Leave-one-out ablations isolate the stages. Removing Analyze drops the five-domain macro average from 66.46% to 59.13%. Removing Gate drops it to 56.11%. The result supports two distinct roles: paired regret supplies direction, while the gate limits collateral regressions. At matched candidate-evaluation budget, parallel M=2 pooling reaches 81.08%, compared with 78.85% for the matched sequential arm.
Professional suites show both promise and uncertainty. PRBench-Legal reaches 66.67%, a 7.19-point gain with four helped tasks, zero hurt tasks, and a confidence interval of [1.85, 12.85]. Fin-Rate and Ledger reaches 50.00%, a 12.50-point gain, but its interval [0.00, 31.25] touches zero with only sixteen tasks. The paper retains that uncertainty rather than presenting it as a resolved gain.
| Experiment | Comparison | Result | Meaning |
|---|---|---|---|
| Five-domain EvoHarnessBench | Full / no Analyze | 66.46% / 59.13% | Directional regret is a high-value diagnostic |
| Five-domain EvoHarnessBench | Full / no Gate | 66.46% / 56.11% | Unguarded improvements overwrite prior competence |
| ITSM dual-axis study | Parallel M=2 / matched sequential | 81.08% / 78.85% | Candidate contrast improves credit assignment |
| BFCL-Miss | Before / after adaptation | 53.33% / 70.00% | A local tool convention can be absorbed safely |
| ScienceWorld | Before / after iff repair | 74.80% / 82.73% | Precondition narrowing fixes an over-broad rule |
| PRBench-Legal | Initial / GUARD | About 59.48% / 66.67% | Gain is 7.19 points and the interval excludes zero |
| Fin-Rate and Ledger | Initial / GUARD | 37.50% / 50.00% | Gain is 12.50 points but the interval touches zero |
The promotion audit is favorable but not universal. All nineteen execution-gated promotions have non-negative held-out deltas; fifteen are strictly positive and four are neutral. This supports the claim that the gate does not systematically promote harmful edits in the tested streams. It does not prove the same behavior on arbitrary distributions. PAST-Bench accepted 24/24 round candidates under critic scoring, so that benchmark does not test whether the gate can separate difficult candidates.
There are also two meanings of stability in the paper. The observed 0.00-point forgetting is an empirical result across the tested continual blocks. The theorem concerns the self-review process and gated buffer. A hidden grader could disagree with the checklist target, and the gate could then stabilize improvement against the wrong objective. The paper explicitly leaves hidden official accuracy as an empirical question.
Limitations
First, the reviewer and gate were calibrated post hoc against hidden official graders. No oracle labels were exposed during adaptation, but calibration still means that the relationship between the checklist target and the official target is estimated rather than proven. GUARD guarantees behavior relative to its self-review objective and gated buffer, not accuracy under every unseen grader.
Second, the sampling result is asymptotic. It is not a non-vacuous finite-sample certificate that a practitioner can deploy without assumptions. Real systems must choose a noise allowance, replay size, and ledger window. Those values interact with task dependence and score variance, so independent held-out monitoring remains necessary.
Third, the evidence is mixed in places by design. PAST-Bench accepted every candidate, while some confidence intervals touch zero and the Meta-Harness comparison has a p-value of 0.11. The Fin-Rate and Ledger interval includes zero. These are directional findings, not universal superiority claims.
Fourth, coverage is limited to selected Gemini and Claude configurations, benchmark families, domains, and task lengths. The cross-model experiments show that GUARD can be inserted into a teacher-student harness pipeline, but they do not cover open-weight models, long-horizon physical control, multimodal robot policies, or adversarial deployment. A tool error in robotics can have irreversible physical consequences, which a text checklist cannot by itself prevent.
Fifth, no public code is available. The exact gate calibration, rejection-ledger deduplication, iff repair prompt, and grouping signatures cannot yet be reproduced independently. The authors promise a release subject to approval, but until the implementation is available, external teams must reconstruct important mechanisms from the paper.
Conclusion
GUARD turns recursive self-improvement into a staged governance loop. It freezes the criteria before execution, uses per-criterion regret to identify newly broken behavior, compares candidates under matched conditions, and requires a proposed edit to beat an explicit estimate of the harm it induces. Its main contribution is not simply a higher benchmark score, but a model for turning experience into bounded, conditional, testable harness changes.
For robot agents, the most promising extension is to move Layer 1 beyond text sandboxes and into state machines, simulation replay, and hardware safety monitors. Tool responses, sensor states, collision constraints, action idempotence, and recovery paths can become executable predicates, while Layer 2 handles task intent and exception explanations. Pairing regret guidance with physical state replay would turn harness evolution from a text agent trick into a durable skill layer.
The caution must scale with the capability. The better a system becomes at optimizing its own review target, the more important independent auditing of that target becomes. Future work needs coverage tests, cross-environment counterfactuals, gate-bias monitoring, and reversible skill versions. Without those safeguards, recursive self-improvement can become a longer prompt with better rhetoric; with them, it can become a credible mechanism for long-term capability accumulation.
Golden line
The hard part of recursive self-improvement is not writing more rules. It is noticing which working capability the newest rule just broke, then refusing to deploy it before the regression spreads.
SOURCE LINKS



