SoL-Pi: Recursively Scaling Auto-Research Loops for Efficient Agent Harness

Paper: Haozhe Liu, Tian Ye, Sensen Gao, Qihang Cao, Yitong Li, Mingchen Zhuge, Duomin Wang, Ruihua Zhang, Ping Luo, Jiawang Bian, Lei Zhu, Ligeng Zhu, Enze Xie, and Song Han. Published September 17, 2026. arXiv:2609.20519. Code. Project page.

SoL-Pi turns harness optimization into a scaled autonomous research loop. It searches across hundreds of executable environments, applies fixed capability and efficiency gates, and retains four mechanisms that reduce repeated model turns, context replay, oversized observations, and unnecessary log reading without retraining the underlying model.

Why Long-Running Agents Waste Tokens

Coding agents are moving from short code-completion sessions to unattended runs that can last for hours. A single task may require dozens of model calls while the agent edits files, runs commands, reads failures, revises code, validates the result, and decides whether more work is needed. The model is no longer making one isolated prediction. It is operating a stateful control loop through an execution harness.

This changes the economics of inference. Token cost is not determined only by model size or per-token pricing. It is also determined by how often the same context, tool output, and evidence are replayed. A long log can remain in the provider context after its useful information has been consumed. A completed subtask can continue to occupy the active prefix. An edit can be followed by a predictable build or test command that costs another full model decision. Individually these effects look small; over a long trajectory they become a first-order systems cost.

Prior efficiency work mostly changes the cost per token through faster kernels, serving infrastructure, quantization, or cheaper models. SoL-Pi targets an orthogonal layer: the harness that mediates interaction between the model and its environment. The base model can remain fixed while the harness changes which actions are exposed, how observations enter context, when compaction occurs, and how delegated reading is verified.

Harness optimization is difficult because its components are coupled. Tool schemas influence action selection, context compaction changes prompt-cache reuse, observation compression can hide evidence, and delegation introduces another model whose output may be unreliable. A change that reduces one local cost can move expense elsewhere or damage task quality. Human engineers often need to inspect long trajectories, identify recurring failures, and translate those observations into code. That process is expensive and hard to scale.

The paper therefore treats the harness as an artifact that can be improved through recursive self-improvement. A research agent inspects traces from a separate agent running the base harness, proposes mechanism changes, and tests them in prepared environments. Its search scope is broad, but every retained mechanism still needs to pass a capability gate and an efficiency gate.

Broad-to-Deep Search with a Hard Evaluation Boundary

The search starts with 152 proposed directions across six proposal families: context, progress, tools, delegation, prompt and policy, and improvement and evaluation. These families describe where an idea came from, not where the final implementation must live. ObservationPack, for example, began as a context hypothesis but ultimately changed the boundary at which tool observations are projected into the provider context.

Each promising direction runs as an isolated lineage. The lineage follows a conventional autonomous research cycle: propose a change, implement it, run a fixed experiment, inspect the result, and retain, revise, or discard it. The authors extend this cycle with an iterative implementation loop based on the Ralph Loop. An implementer works until an explicit completion criterion is met; an independent reviewer checks the candidate before evaluation. Failed reviews trigger another revision.

Isolation is a deliberate orchestration choice. A lineage can be discarded without changing any other lineage. Search breadth comes from running more independent directions; depth comes from repeated implementation and review inside a smaller set of promising directions. This keeps failures local and makes the search easier to scale than one monolithic optimization procedure.

The search corpus contains 535 executable environments. Of these, 495 are repository tasks built from GitHub issue and pull-request pairs. Each task contains a pre-fix repository, offline dependencies, and a hidden regression test. An environment is retained only when the test fails before the accepted patch and passes afterward. The other 40 tasks are verifier-driven environments. A verifier is created first, then a task is constructed around it, allowing multiple valid solution paths rather than a single reference patch.

EdgeBench is reserved for final evaluation. Its 51 public tasks never participate in candidate development. Eleven tasks are used for one-way acceptance of a frozen candidate, and the remaining 40 measure generalization. An evaluation failure rejects the candidate; it does not return to the search process and trigger task-specific repairs.

The acceptance rule is frozen before experimentation. Let $S_0$ and $C_0$ denote the base harness score and token cost, and let $S_c$ and $C_c$ denote the candidate values. The candidate must remain inside a predeclared ability tolerance and improve at least one declared efficiency metric:

$$ \operatorname{CapabilityPass}(c)=\left(S_c \ge S_0-\tau_c\right) $$

$$ \operatorname{EfficiencyPass}(c)=\left(C_c < C_0\right) $$

Only candidates passing both gates enter nondominated selection. The optimizing model cannot edit the metrics or tolerances. This structure matters because a system that can redefine success can always report improvement without becoming more useful.

flowchart LR
    A[Base Harness Traces] --> B[Oracle Analysis]
    B --> C[Broad Hypothesis Pool]
    C --> D[Isolated Lineages]
    D --> E[Implement and Review]
    E --> F[Capability Gate]
    F --> G[Efficiency Gate]
    G --> H[Nondominated Mechanisms]
    H --> I[Integrate and Freeze]
    I --> J[Unseen EdgeBench Evaluation]
    J -. no feedback into search .-> I

Figure 1: The scaled auto-research loop and the final frozen evaluation boundary. Source: paper Figure 1.

Figure 2: Development feedback is separate from held-out evaluation. Source: paper Figure 2.

The Four Mechanisms That Survived Selection

Action Fusion: removing a predictable model round trip

Pi frequently performs an edit or write followed by a command that tests, builds, or starts the changed artifact. The command often does not require a new model decision; it is the predictable next action. Action Fusion adds an optional then_run field to file-mutation tools. When supplied, the mutation succeeds and the command executes before a single combined observation returns to the model.

The mechanism does not skip verification. If the mutation fails, then_run is not executed. Before the command starts, the implementation hashes the modified file again and aborts if the content changed. This protects the fused operation from another writer changing the file between mutation and validation. Commands whose choice depends on inspecting the mutation remain separate.

Action Fusion was strongest on the held-out Opus 5 backend. Adding it to Pi raised average EdgeBench score from 44.756 to 50.482 and reduced total tokens from 2.3697 B to 2.1016 B. Under GPT-5.6 Sol it reduced tokens from 2.1538 B to 1.8968 B while raising score from 44.833 to 46.664. The search case study estimated an 11.5% token reduction under full triggering, but production triggering is naturally selective.

Online Context Compact: deciding when prompt rewrites pay for themselves

Earlier compaction is not always cheaper. Moving the prompt prefix invalidates some cache reuse, so a smaller context can create a larger cache-write cost. Online Context Compact uses plan-step completion as a signal that the agent has reached a stable subtask boundary. At that point it estimates how many model requests remain.

The estimate combines two constraints. One is the observed request count per completed plan boundary multiplied by the remaining boundaries. The other is the number of requests that can fit in the current context window before growth reaches the limit. The smaller value becomes the effective horizon:

$$ \widehat H=\min\left(1+\left\lfloor \bar r\,N_{\mathrm{rem}}\,s \right\rfloor,\left\lfloor\frac{W-x}{\Delta x}\right\rfloor\right) $$

Here $\bar r$ is the mean requests per boundary, $N_{\mathrm{rem}}$ is the number of unfinished plan steps, $s$ scales the estimate, $W$ is the context window, $x$ is current context usage, and $\Delta x$ is the observed average context increment.

Compaction saves $A-M$ tokens, where $A$ is the archived context size and $M$ is the compacted memo size. Rewriting the prompt cache costs $W_c(R-1)$, with $W_c$ equal to write tokens and $R$ equal to the cache write/read price ratio. The break-even request count is therefore:

$$ B=\frac{W_c(R-1)}{A-M} $$

Compaction is economic when $B\le\widehat H$. Later compactions require a larger margin, $1.5B\le\widehat H$, and account for cache debt that earlier compactions have not yet recovered. Window protection can still force compaction when usage approaches the context limit, provided the operation actually shortens the context.

ObservationPack: preserve recall without replaying full observations

Large tool results often remain in later provider requests long after most of their lines have stopped influencing decisions. ObservationPack archives pure-text results larger than 10 KiB. The full result remains visible for the first two requests. From the third request onward, the harness projects a stable handle, original byte and line counts, an estimated token count, and complete head and tail excerpts instead of the full payload.

$$ x_i'=\begin{cases}x_i,&q_i<2\\ \operatorname{Pack}(x_i),&q_i\ge2\end{cases} $$

In this projection, $q_i$ is the number of prior provider requests containing result $x_i$. Pack modifies only the provider-context copy, not the stored session history. The original bytes remain available through obs_recall, which reads bounded pages by byte offset. This distinction is important: the mechanism reduces repeated context cost without making the evidence disappear.

ObservationPack produced the highest single-mechanism average score under GPT-5.6 Sol, 47.208, while reducing total tokens from 2.1538 B to 2.0224 B. Under Opus 5 it reduced total tokens to 1.4418 B and raised score from 44.756 to 47.047.

Evidence-Preserving Reducer: delegation with a verifier

Build and test logs are observation-heavy, but only a small number of lines usually determine the next action. Evidence-Preserving Reducer applies to diagnostic command results of at least 4 KiB. It archives the original log, asks a lower-cost model to extract a structured receipt, and accepts the result only after deterministic checks.

The verifier checks the receipt schema, source SHA-256, process exit status, evidence kinds, quote lengths, total receipt size, and whether every quote appears byte-for-byte in the archived source:

$$ \forall e\in E_{\mathrm{receipt}},\quad e_{\mathrm{quote}}\in \operatorname{bytes}(o_{\mathrm{archive}})\land h(e_{\mathrm{quote}})=h_{\mathrm{claimed}} $$

If any check fails, if credentials are suspected, if the reducer model is unavailable, or if the receipt is not smaller than the source, the full original log is passed through. A failing log that contains a failure signal must keep at least one fatal or failure quote. The reducer is not allowed to diagnose the fix or decide whether the task passed. The primary agent retains diagnosis, repair, rerun, and adjudication.

Ordering also matters. Evidence-Preserving Reducer runs before ObservationPack's context projection. ObservationPack recognizes the reducer receipt and skips it, so a verified receipt is not replaced by a generic excerpt.

Figure 3: The four mechanisms operate at different points in the agent-environment loop and define explicit fallback paths. Source: paper Figure 4.

Code Evidence for the Design Claims

The public repository ships SoL-Pi as an extension for Pi rather than a patched Pi fork. Each mechanism is opt-in and disabled by default. The implementation follows the paper's separation between model ownership and harness logic: provider credentials, model choice, authentication, and shell behavior remain under Pi's control.

Action Fusion maps directly to the paper's tool-schema change. The extension inherits the original edit and write tools and adds then_run. The execution wrapper performs mutation, hashes the target file, runs the requested command, and returns a combined result:

// src/sol-pi/extensions/action-fusion/index.ts
const editParameters = Type.Object({
  ...editTemplate.parameters.properties,
  then_run: createThenRunSchema(EDIT_THEN_RUN_DESCRIPTION),
});

return executeMutationThenRun({
  absolutePath: resolveToolPath(ctx.cwd, input.path),
  thenRun: then_run,
  mutate: () => baseEdit(ctx.cwd).execute(
    toolCallId, editInput, signal, onUpdate, ctx
  ),
});

The same directory contains a per-file queue and assertUnchangedBeforeCommand. This is the code path that supports the paper's claim that the fused command is skipped when the mutation fails or the file changes before validation.

The compaction economics are also explicit. The implementation computes saving, incremental cache cost, break-even horizon, first-compaction scaling, later margins, and carried cache debt:

// src/sol-pi/extensions/online-context-compact/economics.ts
const savingTokens = input.archiveTokens - input.memoTokens;
const incrementalCacheCostRatio = Math.max(
  0, input.cacheWriteReadRatio - 1
);
const breakevenRequests =
  (input.writeTokens * incrementalCacheCostRatio) / savingTokens;
const economic =
  breakevenRequests <= horizon.expectedRemainingRequests;

Evidence-Preserving Reducer has the clearest code-to-paper correspondence. It archives the source, calls the configured reducer model, validates every quoted line against the archive, and returns undefined when the receipt is not smaller:

// src/sol-pi/extensions/evidence-preserving-reducer/receipt.ts
if (
  parsed.schema !== REDUCER_RECEIPT_SCHEMA ||
  parsed.source_sha256 !== archive.hash ||
  parsed.status !== expectedStatus ||
  !body.includes(quote)
) return { ok: false, reason: "schema-mismatch" };

if (receiptBytes >= archive.bytes) {
  return undefined; // retain the original log
}

ObservationPack uses the same thresholds described in the paper: 10 KiB activation, two full provider sends, and a 1 KiB complete-line excerpt. The repository therefore provides enough implementation detail to inspect how each mechanism fails open and where it preserves the original evidence.

EdgeBench Results: Higher Efficiency or Higher Performance

SoL-Pi reports two EdgeBench operating points. The Efficiency configuration always uses all four mechanisms. The Performance configuration selects the highest-scoring single mechanism separately for each backend. This avoids pretending that one configuration optimizes every metric simultaneously.

Harness	Total tokens (B)	Cost	Average score	Cost / score
Codex / GPT-5.6 Sol	3.0537	$1,787	34.738	$1.0086
Pi / GPT-5.6 Sol	2.1538	$1,339	44.833	$0.5855
SoL-Pi Efficiency / GPT-5.6 Sol	1.0990	$894	42.003	$0.4174
SoL-Pi Performance / GPT-5.6 Sol	2.0224	$1,271	47.208	$0.5280
Pi / Opus 5	2.3697	$1,741	44.756	$0.7625
SoL-Pi Efficiency / Opus 5	1.3101	$1,158	42.224	$0.5376
SoL-Pi Performance / Opus 5	2.1016	$1,605	50.482	$0.6235

On GPT-5.6 Sol, the Efficiency point retains 93.7% of Pi's average score while reducing total tokens by 49.0% and cost by 33.2%. The Performance point raises average score from 44.833 to 47.208, reduces held-out token traffic by 6.1%, and improves cost per score by 9.8%. The two operating points show why a harness evaluation needs both score and cost columns.

The model-transfer experiment is more consequential. SoL-Pi is developed only on GPT-5.6 Sol trajectories, then applied unchanged to Opus 5. The Efficiency configuration retains 94.3% of Pi's average score, cuts token traffic by 44.7%, and cuts API cost by 33.5%. Activation becomes less frequent and less intense on Opus 5, so the result supports transfer of the mechanisms but not identical behavior across backends.

Terminal-Bench 4 gives a more mixed picture. On 63 CPU-only tasks, Codex and Pi each solve 18, while SoL-Pi solves 15. SoL-Pi's total cost is $211.12 versus Pi's $286.45, a 26.3% reduction. Cost per solved task falls from $15.91 to $14.07. This is an efficiency gain under a lower completion count, not a universal capability win.

Harness	Terminal-Bench solved / 63	Total cost	Cost / solved	IMO passed / 6	IMO cost
Codex	18	$272.35	$15.13	5	$114.47
Pi	18	$286.45	$15.91	3	$75.95
SoL-Pi	15	$211.12	$14.07	3	$62.69

On IMO 2026, all three harnesses use GPT-5.6 Sol with Lean 4 verification. Codex passes five problems, while Pi and SoL-Pi each pass three. SoL-Pi has the lowest cost per passed problem, $20.90, versus $22.89 for Codex and $25.32 for Pi. The result is economically favorable but not a capability improvement over Codex.

Mechanism Ablations and Interactions

The add-one experiment adds each mechanism to Pi in isolation. Every mechanism reduces total token traffic under both backends. Under GPT-5.6 Sol, ObservationPack obtains the highest single-mechanism score, Online Context Compact obtains the lowest single-mechanism cost, and the complete stack obtains the lowest total tokens and cost. Under Opus 5, Action Fusion has the highest average score, while ObservationPack provides the largest observation-oriented token reduction.

The complete stack combines all mechanisms for the efficiency objective rather than selecting each mechanism by its score-only result. The code allows this composition because each mechanism acts at a different boundary. Action Fusion changes tool execution, Online Context Compact changes plan-boundary behavior, ObservationPack changes provider-context projection, and Evidence-Preserving Reducer changes delegated observation processing.

The paper reports that ObservationPack triggers more selectively in the full stack, likely because Evidence-Preserving Reducer already handles observation-heavy logs. Every mechanism shows a larger efficiency gain in the full stack on its own triggered subset, but the authors explicitly describe this as consistent with complementarity rather than isolation of causal interaction effects.

The Action Fusion case study explains how one mechanism became stable. The lineage contains 27 recorded iterations across oracle analysis, baseline construction, prompt and tool-schema optimization, and final validation. Prompt-only triggering was unreliable; exposing then_run in the tool schema created a stable interface. The search then used trigger rate as an intermediate metric alongside task score before freezing the final configuration.

Figure 4: Action Fusion moved from an observed repeated-action pattern to a frozen and held-out validated mechanism. Source: paper Figure 8.

How to Read the Token and Cost Numbers

The paper decomposes recorded token traffic into four components:

$$ T=T_{\mathrm{input}}+T_{\mathrm{cacheRead}}+T_{\mathrm{cacheWrite}}+T_{\mathrm{output}} $$

This decomposition prevents a misleading conclusion that reducing context always reduces cost. On GPT-5.6 Sol, the complete stack cuts cache-read traffic from 2.1326 B to 1.0605 B, but cache-write traffic rises from 0.0141 B to 0.0316 B. Even with the larger write volume, total model cost falls from $1,339 to $894. The relevant objective is the whole-task bill, not cache reuse in isolation.

The paper also reports API cost per unit of aggregate score:

$$ E=\frac{C}{S} $$

The Efficiency configuration reaches $0.4174 per score point on GPT-5.6 Sol and $0.5376 on Opus 5. Every single-mechanism configuration also improves cost per score over Pi in both backend blocks. This is a more informative systems metric than token traffic alone because it preserves the relationship between cost and useful output.

The agent-swarm experiment extends the same logic beyond a single process. A Codex coordinator with 20 SoL-Pi workers reaches 1,127 kernel cycles at $60.11, while the Pi baseline swarm reaches 1,366 cycles at $82.12. The SoL-Pi swarm reduces cost by 26.8% relative to the Pi baseline and passes all eight speed thresholds; the single Codex agent remains the cheapest configuration at $39.20. The result suggests that cheaper harness execution can increase the exploration capacity of a fixed multi-agent budget, but it does not prove that a swarm is always more efficient than one agent.

Limitations and Open Questions

The first limitation is search cost. The authors state that running complete auto-research loops is computationally expensive, making controlled comparisons of search breadth and depth under a fixed budget difficult. The reported 152 directions, 535 environments, 3,000 runs, and 60,000 interactions describe the study's scope; they do not establish a scaling law.

The second limitation is single-backend development. The harness is updated from GPT-5.6 Sol trajectories. Opus 5 still benefits, but mechanisms trigger less often and less intensely. Multi-backend training may improve robustness, but it remains a future direction rather than demonstrated evidence.

The third limitation is the capability trade-off. The complete efficiency stack retains most of Pi's score on EdgeBench, but Terminal-Bench 4 solved-task count falls from 18 to 15. The paper separates the efficiency and performance operating points, which is honest, but production users must choose how much score loss is acceptable for a given cost reduction.

The fourth limitation is evaluation coverage. The primary evidence comes from coding, terminal, mathematical, and kernel-optimization tasks. The mechanisms are generic harness components, but the paper does not establish that they preserve their gains in real-time robotics, multi-modal perception, delayed physical feedback, or safety-constrained embodied control.

The fifth limitation is privacy and deployment complexity. Evidence-Preserving Reducer sends eligible diagnostic logs to a configured lower-cost model. The archive and receipt design preserve evidence integrity, but they do not eliminate the need for credential filtering, local policy review, or provider trust. The repository disables all mechanisms by default for this reason.

Conclusion

SoL-Pi's strongest contribution is not a single compression trick. It is a method for discovering harness-level mechanisms while controlling search overfitting. The system separates broad hypothesis generation from deep implementation, isolates search lineages, freezes capability and efficiency gates, and prevents held-out results from returning to the search process.

The four discovered mechanisms form a practical systems template. Action Fusion removes a deterministic model round trip, Online Context Compact prices prompt rewrites against future savings, ObservationPack preserves exact recall without replaying full observations, and Evidence-Preserving Reducer delegates log reading without delegating final judgment. Each mechanism has an explicit fallback and a code implementation that can be inspected.

The paper does not claim that cheaper execution automatically makes an agent more capable. It shows that a scaled research process can discover reusable efficiency mechanisms, transfer them to a held-out model, and reduce token traffic by roughly half on EdgeBench while retaining most of the base score. The remaining question is whether the same search and verification structure can scale across models, domains, and embodied environments without losing the evidence needed for correct action.

A good harness does not make the agent do less useful work. It stops paying repeatedly for work and evidence that have already been produced.