ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AI

Figure 1: ScientistTwo pushes the frontier of human knowledge across diverse research domains — LLMs, robotics, neuroscience, speech, robustness, RL, game theory, privacy, optimization, and time series — with resulting methodologies consistently outperforming human SOTA baselines.

Background: The Gap from "Assistive Tool" to "Autonomous Researcher"

Scientific discovery has long been the hallmark of human ingenuity, defined by the ability to identify the boundaries of current knowledge and venture into the unknown. With the rapid advancement of foundation models, AI is transitioning from a passive conversational assistant to an active participant in the scientific process. The ultimate ambition of this paradigm is purely problem-driven autonomous discovery: a human researcher specifies a scientific challenge, and the AI independently navigates the landscape of human knowledge, diagnoses theoretical and empirical bottlenecks, formulates novel hypotheses, and executes the end-to-end research lifecycle.

Despite recent progress in autonomous research agents, a substantial gap remains between automated assistant systems and rigorous empirical scientific standards. Existing systems primarily focus on optimizing single scalar metrics on isolated benchmarks, lacking the multi-dimensional reasoning needed for complex scientific problems. More importantly, they lack the closed-loop empirical rigor of human scientists who continuously iterate based on evidence: they cannot systematically conduct ablation studies to isolate causal mechanisms, nor engage in the dynamic peer-review process essential for validating ideas and addressing methodological critiques through targeted supplementary experiments.

To overcome these challenges, ScientistTwo introduces an expert-level autonomous multi-agent framework. Its validation is unusually demanding: given competitive peer-reviewed human research from ICLR, ICML, and NeurIPS, it must independently discover meaningful advancements beyond the established SOTA, while showing that the discovery process is measurable, reproducible, and transparent.

Method: A Six-Stage Agent Ecosystem

Figure 3: Overview. ScientistTwo leverages previous experiments, ablation studies, and reviewer feedback to generate and validate novel ideas, verifying each across multiple datasets and metrics, then simulates peer review to iteratively refine drafts into publication-ready papers.

Formally, given a scientific problem $\mathcal{G}$, the framework produces a paper $\mathcal{P}^{+}$ paired with a reproducible codebase $\mathcal{C}^{+}$:

$$(\mathcal{P}^{+},\ \mathcal{C}^{+}) = \mathcal{A}(\mathcal{G}), \quad \text{where} \quad \mathcal{A} = \{\mathcal{A}_1, \mathcal{A}_2, \dots, \mathcal{A}_{N_a}\}$$

$\mathcal{A}$ comprises $N_a$ specialized agents assigned to distinct phases of the research lifecycle. Every stage follows the same abstraction — generate candidate → critique → accept or refine (Listing 1):

Figure 4: Python-style pseudocode for each pipeline stage. A critic agent evaluates the candidate artifact; if accepted it is returned, if rejected discarded, otherwise iteratively refined from feedback up to a maximum round budget.

Stage 1: Generating Novel Seed Ideas (Targeting Limitations)

Figure 5: Generating novel seed ideas. ScientistTwo begins by identifying limitations of the human state-of-the-art.

Finding limitations. The Limitation Extractor extracts a set of limitations from $\mathcal{G}$; the Limitation Verifier checks whether the set suffices to guide novel improvements. If deemed insufficient, the extractor identifies missing weaknesses and expands the collection — the loop repeats until the Verifier confirms all actionable limitations are thoroughly extracted or the iteration cap is reached.

Generating seeds. An initial idea $h_0$ targeting the identified limitations is generated and scored by the Novelty Checker. Starting from $\mathcal{H}_0 = \{h_0\}$, the Idea Generator iteratively expands the pool with distinct, higher-novelty ideas until $N_{\text{seed}}$ candidates are gathered. Seeds are then sorted by novelty score $\{s_i\}_{i=0}^{N_{\text{seed}}-1}$ in descending order, with $s_i \ge s_j$ whenever $i < j$ — so the most original ideas get implemented first.

Stage 2: Evaluating Ideas (Subset-to-Full-Set Funnel)

To balance computational efficiency, ScientistTwo adopts a subset-first strategy: a Baseline Coding Agent reproduces the primary experiments of $\mathcal{G}$ on a benchmark subset, producing $E_{\text{base}}$ and a reproducible codebase $C_{\text{base}}$. A Subset Coding Agent then implements candidate $h$ by modifying $C_{\text{base}}$, yielding logs $E^{\text{sub}}_h$. A Subset Critic Agent compares them and emits a three-way decision $d_h$ with feedback $r_h$:

1. Bad: substantially inferior to baseline — $h$ is discarded;
2. Good: consistently outperforms the baseline — approved for scale-up;
3. Engineer: shows potential but needs tuning or code fixes — a Subset Engineering Agent refines $h$ guided by $r_h$, looping until $d_h \in \{\text{Good}, \text{Bad}\}$ or the $N_{\text{eng}}$ budget is exhausted. If the budget runs out without Good, $h$ is designated Bad and pruned — mirroring real research practice where unpromising avenues are abandoned after bounded optimization.

Ideas validated as Good are adapted to the full benchmark by a Full-Set Coding Agent, with a Full-Set Critic and Engineer performing final validation. The whole idea-experiment pipeline is abstracted into a single high-level interface $\mathcal{A}_{\text{Coder}}$:

$$h,\ E^h,\ C^h,\ d_h,\ r_h = \mathcal{A}_{\text{Coder}}(\mathcal{G},\ h)$$

Stage 3: Refining Ideas (Evolution from Execution Traces)

At round $k=0$, the top-$N_0$ seed ideas run through $\mathcal{A}_{\text{Coder}}$, yielding traces $\mathcal{R}_0$. For refinement round $k \ge 1$, all historical traces $\mathcal{R}_{<k}$ are aggregated and an Idea Evolver generates $N_k$ evolved ideas — analyzing both successful results ($d_h=\text{Good}$) and diagnostic failure logs ($d_h=\text{Bad}$) to propose refined hypotheses.

Exploration–exploitation balance is a key design: relying solely on the Evolver risks trapping optimization in local optima around early seeds, so the evolved set is complemented with $N_e$ previously unevaluated seeds drawn from $\mathcal{H}_0$ in descending novelty order, giving candidate pool $\mathcal{I}_k \cup \mathcal{H}^{(k)}_0$. Formally, round $k$ executes:

$$\mathcal{R}_k = \Big\{ (h, E^h, C^h, d_h, r_h) \ \big|\ (h, E^h, C^h, d_h, r_h) = \mathcal{A}_{\text{Coder}}(\mathcal{G}, h),\ h \in \mathcal{H}_k \Big\}$$

The loop iterates until $S$ successful ideas are collected or the maximum refinement limit $K$ is reached, with termination condition $\sum_{i=0}^{k}\sum_{h \in \mathcal{H}_i} \mathbb{I}(d_h = \text{Good}) \ge S$. If round $K$ ends with zero successes, the entire process terminates. A Selector Agent then picks the best candidate among all fully validated ideas:

$$h_{\text{best}},\ E_{\text{best}},\ C_{\text{best}} = \mathcal{A}_{\text{Selector}}\Big(\mathcal{G},\ \big\{(h, E^h, C^h)\ \big|\ d_h = \text{Good}\big\}\Big)$$

Stage 4: Ablation Studies (Isolating Gain and Feeding Back)

An Ablation Planner formulates $N_p$ executable ablation plans $\{p_1, \dots, p_{N_p}\}$ tailored to $h_{\text{best}}$; an Ablation Coding Agent modifies $C_{\text{best}}$ to execute each, yielding $\mathcal{E}_{\text{abl}} = \{c_1, \dots, c_{N_p}\}$.

There is a distinctly human design here: ablation is not merely reporting — it feeds back into idea refinement. An Ablation Critic inspects the component breakdown and outputs $d_{\text{abl}} \in \{\text{Good}, \text{Refine}\}$; on Refine, the Full-Set Engineering Agent produces a refined hypothesis guided by $r_{\text{abl}}$. Because structural refinement does not guarantee improved performance, a Result Comparison Agent strictly verifies whether $\mathcal{E}_{\text{new}}$ outperforms $\mathcal{E}_{\text{best}}$ — the core state is updated only when strictly superior, and any update re-triggers ablation planning. The loop repeats for at most $N_{\text{abl}}$ iterations.

Stage 5: Manuscript Drafting (Simulated Peer Review and Rebuttal)

An Initial Drafter (incorporating PaperOrchestra) synthesizes $h_{\text{best}}$, main results $E_{\text{best}}$, and ablations $\mathcal{E}_{\text{abl}}$ into a conference-formatted manuscript $\mathcal{P}_{\text{new}}$. Simulated review then begins: a Peer-Reviewer Agent (ScholarPeer) produces strengths, weaknesses, targeted questions, and an ICLR-scale score $s_{\text{review}} \in [1,10]$.

If the score falls below the acceptance threshold (e.g., 8), an automated rebuttal stage launches: a Rebuttal Planner formulates $N_t$ supplementary experimental tasks $\{t_1,\dots,t_{N_t}\}$; a Rebuttal Coding Agent implements and executes them on $C_{\text{best}}$, yielding $\mathcal{E}_{\text{reb}}$; a Paper Enhancer integrates the review and supplementary findings back into the manuscript, revising claims and updating empirical tables and figures. The cycle repeats until $s_{\text{new}} \ge 8$ or the $N_{\text{peer}}$ budget is exhausted.

Stage 6: Meta-Reviewing (Final Assessment and Review-Driven Refinement)

A Meta-Review Agent evaluates the revised manuscript alongside the peer review, emitting $d_{\text{meta}} \in \{\text{Accept}, \text{Refine}\}$. On Accept, the final paper and codebase are exported as $\mathcal{P}^{+} \leftarrow \mathcal{P}_{\text{new}}$ and $\mathcal{C}^{+} \leftarrow C_{\text{best}}$.

On Refine — meaning the meta-reviewer identified a critical algorithmic or empirical weakness — a deep idea refinement phase starts: the Full-Set Engineering Agent updates $h_{\text{best}}$ using the meta-critique $r_{\text{meta}}$. Again, the core state updates only if the Result Comparison Agent verifies $\mathcal{E}_{\text{new}}$ is strictly superior; because the fundamental idea has changed, downstream ablation planning, execution, manuscript re-drafting, and simulated peer review all re-run. Otherwise the refinement is discarded and the previous best outputs are used. This meta-refinement loop runs for at most $N_{\text{meta}}$ iterations.

Experiments

Comparison with Autonomous Research Agents

The Stanford Agentic Reviewer column deserves attention — it is a held-out evaluator unseen during development by both baselines and this method. Every prior baseline scores 0% acceptance on it; ScientistTwo is the only system where 72.1% of generated papers meet high acceptance standards. Qualitatively, ScientistOne produces just 4 figures and 2 tables (single metric, isolated benchmark, limited baselines), while ScientistTwo produces 9 figures and 12 tables (including the appendix) with multi-metric evaluation across diverse datasets and extensive baseline comparisons.

Comparison with Human and AI-Generated Papers

†denotes AI-generated papers. Using human-accepted papers from each venue as input, ScientistTwo's outputs score 7.5 with 91.9% acceptance under ScholarPeer — above the level of human-accepted ICLR 2026 (6.8 / 60.0%) and NeurIPS 2025 (6.2 / 65.8%) papers — and reach 5.7 on the held-out Stanford reviewer, on par with human papers (ICLR 5.2, NeurIPS 5.5).

The AutoSOTA comparison is also telling: AutoSOTA achieves a median gain of 2.7% and average 7.5% over 105 papers, while ScientistTwo reaches a median of 7.7% and average of 25.2% over 86. The authors attribute this to a paradigm difference — ScientistTwo proposes novel methodological innovations that overcome baseline limitations, whereas AutoSOTA primarily searches within the existing hyperparameter and execution space. Appendix examination of five ICLR 2026 papers supports this: none of AutoSOTA's changes introduces a new algorithmic component; each is a configuration-level edit of at most a few lines.

Ablation Studies

Three readings. First, even with no rebuttal and zero review rounds, ScientistTwo's initial draft far exceeds ScientistOne (46.9% vs 14.3% acceptance), showing the preceding autonomous research cycle — holistic benchmark reasoning, idea refinement, ablation-driven hypothesis refinement — is highly effective on its own. Second, the review-rebuttal loop lifts the ScholarPeer score from 5.2 to 7.6 and acceptance from 46.9% to 93.9%. Third, an important generalization finding: although ScientistTwo uses ScholarPeer reviews to polish drafts, incorporating review feedback also raises acceptance under the held-out Stanford Agentic Reviewer (49.0% → 73.5%), indicating the review simulation framework is not overfitting to a single reviewer.

Another ablation examines idea evolution: relative gain rises steadily with iterative refinement, but improvements are most pronounced in early rounds; most top-performing ideas selected by the Selector emerge early, indicating the initial seed ideas are already well-formed and competitive. When seeds yield marginal gains or fail, the Idea Evolver leverages execution traces to evolve them into stronger hypotheses — evidenced by the growing share of evolved candidates among selected best ideas in later rounds.

A review-driven refinement ablation using Kwon et al. (2026) as input reports unlearning performance on TOFU (forget10 split, Llama-3.2-1B-Instruct): the human method scores Overall 0.705, the ScientistTwo variant without meta-review refinement 0.897, and the full version with meta-review refinement 0.916, with memory, utility, and privacy metrics all improving. An Integrity Audit shows only the full refinement-agent configuration achieves 49/49 score verifiability, 0 specification violations, 0 hallucinated references, and 49/49 method–code alignment — dropping any component introduces failures on these axes.

Significance and Limitations

ScientistTwo raises the acceptance bar for "autonomous research" from "the pipeline runs" to "the output clears top-venue acceptance standards under a held-out reviewer and ships with a reproducible codebase." Several engineering choices transfer well beyond this paper: subset-first compute allocation (rapidly filter ideas on representative slices before spending full-scale compute); treating failure traces as evolution signal (the Evolver reads both success and failure logs); update-only-if-strictly-better gating (the Result Comparison Agent prevents performance regressions from being mistaken for improvements inside the feedback loop); and the dual use of review feedback (raising manuscript scores and driving algorithmic re-refinement).

On limitations, the paper is primarily experimental; the design surfaces a few boundaries. The system relies on ScholarPeer as one review signal — held-out reviewer experiments demonstrate generalization, but the reviewer's own preferences still propagate into the optimization target. All 107 scientific problems come from machine-learning venues; transfer to natural sciences (wet labs, real-world interaction) is unverified. Error handling and budget caps ($N_{\text{eng}}$, $N_{\text{abl}}$, $N_{\text{peer}}$, $N_{\text{meta}}$) guarantee termination, but also mean hard problems may be abandoned before a longer exploration would have paid off.