PAPER DEEP DIVE
Do AI Scientists Know What's New?
A major challenge facing AI scientific discovery pipelines is how to determine when a generated idea is meaningfully novel. Ideally, novelty checks in these pipelines must be able to reject non-novel ideas while not rejecting too many novel ideas, and they should also be able to distinguish which parts of a proposal are non-novel, as opposed to opaquely discarding entire proposals. We design an analytical framework for evaluation of novelty checks with temporal ground truth at both the proposal- and claim-level, using four datasets of 1,060 proposals built from published papers, reviewer disputes, and generated ideas matching ones the field went on to publish. Auditing existing novelty checks with this framework, we find that checks with high rejection rates on non-novel work also tend to have high false alarms on ideas that are later realized by human researchers, and observe that checks built to evaluate proposals as a whole can point to claim-level prior art only by rejecting the whole proposal. We therefore introduce ClaimNov, a fine-grained, interpretable claim-level novelty module. ClaimNov catches 97% of disguised papers at a 1.5% false-alarm rate, and its claim-level novelty profile cuts the LLM judge's false alarms from 11.8% to 1.5%. Because its verdict rests on claim-level evidence, ClaimNov also keeps proposals that build on established contributions, rejecting only 3% of ICLR-accepted proposals with a reviewer-disputed contribution and keeping 91 of 100 novel proposals with one spliced prior contribution, against 76 for the strongest existing check. Code and data are available at https://github.com/emmatliu/claimnov.
One-Sentence Summary
The paper audits existing novelty checks with temporally grounded proposal and claim datasets, then introduces ClaimNov, a module that extracts typed contribution claims, retrieves and grades prior claims, and gives a verdict judge a complete evidence profile rather than a single similarity score.
1. The Problem: Semantic Similarity Is Not Novelty
AI scientist pipelines can now produce research questions, proposed methods, experiment plans, and paper-like narratives. Once generation becomes cheap, the bottleneck moves to judgment. A system must determine whether a generated proposal is already established in the literature, whether it merely borrows the vocabulary of an existing paper, or whether it contains a meaningful new contribution. Accepting duplicates wastes scarce experimental resources; rejecting genuinely new work removes the very output the pipeline was built to discover.
Many existing novelty filters collapse this decision into a comparison between a proposal and a ranked list of abstracts. A dense embedding can find topical neighbors efficiently, but closeness in embedding space does not reveal what the prior paper actually established. A proposal may share a problem with an old paper while proposing a different method, or use an established method on a new problem, or contribute a new dataset rather than a new algorithm. A single similarity threshold cannot distinguish these cases.
LLM-based judges add semantic understanding, but their output is often an opaque binary verdict supported by a short explanation. If the retrieved context contains the wrong paper, or if one local overlap dominates the prompt, the model can reject an entire proposal without identifying which part is old. The paper argues that this evaluation unit is wrong for science. Research proposals are bundles of contributions, and novelty should be assessed contribution by contribution before deciding what the bundle as a whole deserves.
This distinction matters for incremental science. Most published work builds on established problems, datasets, and methods. A paper can legitimately contain one borrowed component and several new ones. A filter that rejects any proposal containing prior art will discard most normal research. A filter that rejects only when every component is old may instead pass lightly disguised copies. The design challenge is therefore not choosing a more conservative threshold; it is representing the structure of the proposal.
The paper also argues that evaluation has to be counterfactual. A novelty check should reject a disguised paper when its source is present in the reference pool and stop rejecting the same proposal when the source is removed. Merely counting rejections on known duplicates rewards systems that reject everything. Pairing that count with later-realized ideas, source removal, and claim attribution exposes the trade-off between false confidence and false alarm.
2. A Shared Evaluation Contract
To compare systems with different interfaces, the authors define a common contract. Every check receives a proposal consisting of a title and a 150–400 word description, plus a reference pool whose papers are represented by identifier, title, and abstract. Systems that normally search externally are sandboxed and can only search that pool. This removes differences in internet access, index freshness, and search budget from the headline comparison.
The output contract has three levels. L1 is a pass or fail verdict with reasons. L2 is a novelty score between zero and one. L3 is a claim-level evidence profile containing each proposal claim, its candidate prior claims, source papers, and graded coverage. Most audited systems emit only L1. ClaimNov produces L3, derives L2 from it, and gives L3 to the final judge that makes the L1 decision. The distinction matters because ClaimNov is not just another prompt wrapper: the score, verdict, and evidence are designed as one inspectable structure.
The authors select March 2026 as a temporal cutoff and evaluate judges whose training cutoff predates it. Disguised proposals come from ICML 2026 spotlight papers whose first arXiv versions appeared after the cutoff. Later-realized ideas are linked to papers that cited 2025 NeurIPS anchors after the cutoff. No temporal benchmark can remove every form of contamination, but this construction makes recognition of the exact future paper less likely than in a random historical sample.
3. Four Datasets with Different Failure Modes
The disguised-paper set starts with 100 ICML 2026 spotlight papers. Each paper is rewritten at four levels of increasing difficulty. The first changes voice, the second paraphrases, the third changes motivation and reorders or hides contributions, and the fourth reconstructs the material from a different theoretical starting point with generalized values. A verifier accepts a disguise only if the core contributions remain the same, and the rewrite must have low six-gram overlap with the source. Five sources lack usable pools, leaving 380 proposals.
Each pool contains the original source plus up to 19 pre-cutoff neighbors. This enables a direct intervention. If the source is present, a correct check should reject the proposal and name it. If the source is removed, the same proposal is novel relative to the remaining pool and should pass. Reporting rejection after source removal prevents a system from looking good merely because it treats polished research prose as suspicious.
The later-realized set contains 400 proposals. The authors begin with 30 NeurIPS 2025 anchors and construct a pre-cutoff related-work pool and a post-cutoff follow-up pool of citing papers. A model generates 3000 ideas from the anchors. Another model labels each contribution against both pools. A proposal enters the later-realized set when at least one contribution is established by the follow-up pool but none is established by the original pool. These ideas represent work that the scientific community subsequently realized, so a useful check should generally pass them.
The claim-dispute set comes from about 8,300 ICLR 2022–23 reviews. The authors identify reviews where a reviewer says that a specific contribution of paper A was already established by paper B. After rewriting and filtering, 160 pairs have usable pools. The set contains 62 eventually accepted papers, 96 rejected papers, and two with unknown outcomes. A correct system should locate B’s contribution without automatically treating the whole proposal as non-novel.
The hybrid set begins with 100 later-realized ideas. For each idea, the authors select a paper B of moderate similarity and ask a model to splice one core contribution from B into the proposal. The correct behavior is doubly specific: pass the proposal because most of it remains new, but identify B’s established contribution. This separates the quality of the final verdict from the quality of attribution, a distinction that whole-proposal baselines often conflate.
| Dataset | Size | Construction | Correct behavior |
|---|---|---|---|
| Disguised papers | 95 sources, 380 proposals | Four rewrite levels, source retained in pool | Reject and name source; pass after source removal |
| Later-realized ideas | 400 | Anchor ideas whose contributions appear in later papers | Pass instead of treating future work as prior art |
| Reviewer disputes | 160 | ICLR reviews alleging that paper B established A’s claim | Find B while avoiding whole-proposal rejection |
| Hybrid proposals | 100 | New idea plus one spliced contribution from B | Pass the proposal and identify B’s local contribution |
4. ClaimNov: A Claim-Level Novelty Pipeline
The first stage extracts typed contribution claims from the title and description. A small model at temperature zero returns between one and four atomic claims. Each claim has a dimension from problem, methodology, resource, or result, plus a comparison attribute and value. For example, a result claim might use classification accuracy as its attribute and specify an improvement from 30.1% to 76.9% as its value. Proposals average about 2.4 to 2.7 extracted claims. Pool papers are processed once from their abstracts.
The proposal claim set is:
$$C_P=\{c_1,c_2,\ldots,c_m\}$$
The dimension $d_c$ is not cosmetic metadata. It prevents a paper’s release of a dataset from being treated as direct coverage of a new algorithm, and it prevents a shared application domain from being treated as a shared result. The extraction prompt contains boundary rules for ambiguous cases, including the distinction between a new task formulation and a new method for that task.
For each proposal claim, the system embeds the claim text together with the paper title and retrieves prior claims from the same dimension by cosine similarity. It keeps at most one claim from each pool paper, so repeated sentences from one source cannot dominate the evidence. Retrieval depth depends on the number of papers with claims in that dimension:
$$k_d=\min\left(40,\max\left(10,\operatorname{round}(0.2n_d)\right)\right)$$
This gives at least ten candidates for small pools and caps the search at forty for large pools. The rule controls cost, but it also creates a recall boundary: a valid prior claim below rank forty never reaches entailment. The paper tests retrieval separately and discusses fixed-depth ablations, but the cap remains relevant when ClaimNov is moved to a much larger corpus.
Every retained pair goes to a graded entailment judge. The judge sees both titles and abstracts and assigns a level from one to five. Level one means unrelated, level two means shared topic only, level three means partial coverage with important differences, level four means strong coverage with minor omissions, and level five means full coverage of the substantive claim. The level is mapped to a continuous score:
$$e(c,c')=\frac{\ell(c,c')-1}{4}\in[0,1]$$
This scale preserves partial evidence. A level-three match is neither discarded nor treated as proof of duplication. It can contribute to the anticipation score, but its weight is lower than a level-five claim. The authors argue that this granularity is necessary because scientific contributions often overlap partially rather than being exactly identical.
For each proposal claim, the entailment scores are sorted and combined with rank discounting:
$$a(c)=\frac{\sum_{i=1}^{k_c}\alpha^{i-1}e_i}{\sum_{i=1}^{k_c}\alpha^{i-1}},\qquad \nu(c)=1-a(c)$$
The implementation uses $\alpha=0.5$, so the strongest match receives the largest weight while several moderately strong matches still matter. Here $k_c$ is the minimum of the retrieval depth and the actual number of retrieved candidates. $\nu(c)$ is the novelty remaining in one claim after its best surrounding evidence has been considered.
Claim-level scores are then combined with dimension weights and claim-importance weights. The paper assigns 0.4 to problem, 0.3 to result, 0.2 to methodology, and 0.1 to resource. For proposals, claim importance is usually uniform. For full papers, importance can be proportional to the number of supporting passages associated with each claim.
$$A(P)=\sum_{c\in C_P}w_{d_c}\operatorname{imp}(c)a(c)$$
$$A_{\max}(P)=\sum_{c\in C_P}w_{d_c}\operatorname{imp}(c),\qquad N(P)=1-\frac{A(P)}{A_{\max}(P)}$$
$A(P)$ is the weighted amount of proposal contribution already anticipated, and $A_{\max}(P)$ is the value if every claim were fully anticipated. Their ratio therefore ranges from zero to one, and $N(P)$ is its complement. The score alone is not the final decision. The complete profile $\Phi(P)$ is passed to a stronger verdict model, which sees the extracted claims, candidate prior claims, source abstracts, and entailment levels.
The verdict model returns pass or fail. On failure it also returns a source paper $p^*$, the subset of proposal claims already established, and a short rationale. This makes the output suitable for feedback into an ideation loop: an agent can preserve a new problem or method while revising the contribution that overlaps with prior work.
flowchart TD
P[Research proposal: title and description] --> C[Extract one to four typed contribution claims]
L[Reference pool: titles and abstracts] --> PC[Extract and cache pool claims]
C --> R[Retrieve nearest claims in the same dimension]
PC --> R
R --> O[Keep at most one claim per source paper]
O --> G[Grade coverage from one to five]
G --> D[Map to zero-one and apply rank discount]
D --> A[Aggregate by dimension and claim importance]
D --> F[Build the full claim-level profile]
A --> S[Compute novelty score N of P]
F --> V[Verdict judge reads the evidence profile]
V --> Q{Pass or fail the proposal}
Q -->|Fail| X[Name prior paper and established claims]
Q -->|Pass| Y[Keep the proposal and expose remaining new claims]
The architecture deliberately separates aggregation from judgment. A numerical score is useful for ranking and threshold experiments, but the final verdict can reason about combinations that a scalar obscures. A proposal with one established claim and three new contributions should usually remain in the pipeline. A proposal whose central problem and method are both covered should fail. The profile lets the judge distinguish those cases.
5. Correspondence with the Public Repository
The public repository is a working implementation rather than a documentation-only release. In src/novelty/pipeline.py, score_paper coordinates claim extraction, support passage counting, retrieval, entailment, signal aggregation, and proposal scoring. Its returned report contains the global score, search status, per-claim anticipation, top matches, graded judgments, and candidate counts. This matches the evidence profile shown in Figure 2.
src/novelty/retrieve.py implements top_k_works. It filters candidates by dimension, embeds the target attribute-value text, computes cosine similarity, and keeps only the highest-scoring claim from each paper. src/novelty/config.py contains the same top-k range, alpha value, dimension weights, entailment range, strong level, and embedding model described in the paper.
src/novelty/aggregate.py implements claim_anticipation, claim_signals, and paper_novelty. The first performs the rank-discounted average, the second exposes maximum and breadth signals, and the third applies dimension and claim importance weights. The verdict stage is in gate.py::profile_veto; it takes the complete claim profile rather than asking the judge to rediscover prior art from a handful of abstracts.
The extraction prompt in the repository is unusually specific about contribution boundaries. A new task, setting, or formalization is typed as problem; a new algorithm or training procedure is typed as methodology; a released dataset, benchmark, or tool is typed as resource; an empirical or theoretical finding is typed as result. This is the operational basis of the paper’s interpretability. The explanation is not appended after an opaque score; the unit of evidence itself is constrained.
6. Main Results
On the 380 disguised proposals, ClaimNov rejects 367 and names the original paper in 366 of those rejections. When the source is removed from the pool, it rejects only 18. It incorrectly rejects 1.5% of the 400 later-realized ideas. These produce $J=0.95$ and $\Delta=0.92$. The first metric, catch minus false alarm, captures balanced discrimination. The second, catch minus rejection after source removal, tests whether the system depends on the source rather than on stylistic cues alone.
CoI-Agent is the strongest close baseline on this set. It catches 368, still rejects 31 after source removal, and false-alarms on 3.8%, for $J=0.93$ and $\Delta=0.89$. The paper reports that the confidence intervals do not establish a clear advantage over CoI-Agent on whole-proposal decisions alone. The cost comparison is more favorable: the measured final ClaimNov verdict call costs about $0.010, compared with $0.039 for CoI-Agent. This is not a full-pipeline cost comparison because claim extraction and entailment also consume tokens.
Other checks illustrate the two common failure modes. Si et al. rejects all 380 disguises but still rejects 359 after source removal and false-alarms on 96.0% of later-realized ideas. AI Scientist v1 catches 292 but false-alarms on 51.8%. AI-Scientist-v2 catches 362 with a 39.2% false alarm rate. The OpenNovelty-style per-contribution baseline catches 379, rejects 38 after source removal, and false-alarms on 11.5%. High recall on duplicates is therefore not enough evidence of a useful novelty module.
| Check | Caught / 380 | Named / 380 | Source out | False alarms | J | Δ |
|---|---|---|---|---|---|---|
| AI Scientist v1 | 292 | 292 | 91 | 51.8% | +0.48 | +0.76 |
| CoI-Agent | 368 | 362 | 31 | 3.8% | +0.93 | +0.89 |
| Si et al. | 380 | 379 | 359 | 96.0% | +0.04 | +0.14 |
| AI-Scientist-v2 | 362 | — | 117 | 39.2% | +0.56 | +0.64 |
| OpenNovelty-style | 379 | 379 | 38 | 11.5% | +0.88 | +0.90 |
| ClaimNov | 367 | 366 | 18 | 1.5% | +0.95 | +0.92 |
The claim-dispute set tests whether a system can recognize one contested contribution without rejecting the entire proposal. ClaimNov rejects 3% of accepted ICLR proposals and 3% of rejected proposals. Its native top-ranked profile paper is the reviewer’s paper B in 21% of cases. A dense ranking of the whole proposal identifies B in 37%, showing that the reviewer-selected source is often closest to the proposal globally even when it covers only one extracted claim.
When the authors give ClaimNov and CoI-Agent the same dense top-one attribution, ClaimNov preserves its 3% rejection rate and identifies B in 37%, with 36% among passed proposals. CoI-Agent rejects 10% of accepted and 11% of rejected proposals and identifies B in 37%, with 31% among passed proposals. The comparison therefore isolates the verdict layer from the ranking layer: when attribution is held fixed, ClaimNov’s evidence profile keeps more proposals that build on an established contribution.
| Claim-dispute check | Reject accepted | Reject rejected | Name B | Name B on pass |
|---|---|---|---|---|
| ClaimNov | 3% | 3% | 21% | 20% |
| ClaimNov + dense top-1 | 3% | 3% | 37% | 36% |
| CoI-Agent + dense top-1 | 10% | 11% | 37% | 31% |
| CoI-Agent native | 10% | 11% | 6% | 0% |
| Si et al. | 81% | 78% | 47% | 0% |
The hybrid set makes the intended behavior explicit. Every proposal is built from a later-realized novel idea and should pass, while each also contains one known contribution from B. ClaimNov passes 91, compared with 76 for CoI-Agent and 83 for the OpenNovelty-style baseline. ClaimNov’s native profile names B in 25 cases and names B while passing in 18. With a shared BM25 whole-proposal ranking, it passes 91, names B in 74, and names B while passing in 67.
The entailment evidence supports the mechanism. When the spliced claim is extracted as a separate claim, the judge often assigns B a level of four or five. Among the 15 papers rated at level four or above, 13 are the true spliced source. Only nine of the 400 later-realized ideas reach that level. The profile therefore tends to isolate a genuine local overlap rather than diffusing suspicion across the whole proposal.
| Hybrid check | Pass / 100 | Name B | Pass and name B |
|---|---|---|---|
| CoI-Agent | 76 | 11 | 0 |
| OpenNovelty-style | 83 | 68 | 64 |
| ClaimNov | 91 | 25 | 18 |
| ClaimNov + BM25 top-1 | 91 | 74 | 67 |
| CoI-Agent + BM25 top-1 | 76 | 74 | 53 |
7. Ablations and Robustness
The most diagnostic ablation removes the claim profile but keeps the same final judge. The judge instead reads the eight pool abstracts nearest to the whole proposal. Disguise catch remains high at 373, but source-out rejection rises from 18 to 57 and false alarms rise from 1.5% to 11.8%. Since the final model is unchanged, the improvement must come from the structured evidence supplied to it rather than from a stronger judge alone.
A per-claim veto is equally revealing. If any central claim is entailed, the whole proposal fails. This catches all 380 disguises, but source-out rejection rises to 197 and false alarms reach 56.8%. The result shows why claim-level evidence cannot be converted into a simple OR rule. Proposals contain multiple contributions, and the verdict must consider whether the overlap is central, whether other contributions remain, and whether the proposal is still worth executing.
Removing the verdict judge and using only the aggregated score plus a strong-match rule catches 349 disguises with 1.8% false alarms. It loses 18 catches relative to the full method but adds only one false alarm. This variant may be attractive when inference cost matters. A fixed retrieval depth of ten leaves whole-proposal catch effectively unchanged but raises false alarms to 2.0%; the dynamic depth matters more as the pool grows.
Alternative labelers from another model family keep ClaimNov’s false-alarm rate between 1.3% and 1.5% on the later-realized set. Alternative final judges reduce catch, but the profile still lowers false alarms and source-out rejection relative to a nearest-abstract judge. When pools are enlarged to about 200 papers, the hardest disguise rung becomes more difficult, while source-out rejection remains close to zero. The trend is consistent, though the absolute operating point changes with corpus size.
8. Limitations
The authors state that their labels are heuristics, not a perfect ground-truth dataset. Papers are disguised by a model, later-realized ideas are labeled by a model, and reviewer disputes are not a uniform expert annotation of novelty. They cross-check later-realized labels with another model family and conduct a small human audit, but ambiguity about what counts as the same contribution cannot be fully removed.
The evaluation is restricted to AI research and English-language papers. The dimension weights encode an assumption that problem contributions matter most, followed by results, methods, and resources. Other fields may value clinical evidence, materials synthesis, proof novelty, or data collection differently. The authors explicitly note that the weights would need domain-specific tuning, so ClaimNov is not a universal cross-disciplinary metric in its current form.
The reference pools are fixed lists rather than open literature search. Disguise and dispute pools contain about 20 papers, while generated-idea pools can reach 400. This supports clean counterfactual evaluation but leaves open retrieval failures caused by missing papers, date filtering, synonymy, and citation coverage. If the relevant prior work is absent from the pool, even a correct claim-level comparison cannot find it.
Proposal generation, disguise generation, and final judging use related model families. The paper runs alternative-judge and alternative-labeler experiments, but self-preference bias is not completely eliminated. Error propagation is another concern: if extraction misses a spliced claim or merges two distinct contributions, retrieval and judgment operate on the wrong object. ClaimNov’s native profile names B in only 25 of 100 hybrids, showing that extraction and attribution remain imperfect even when the verdict logic is sound.
The temporal datasets also decay. A judge trained after the cutoff may recognize the supposed future papers, invalidating the time-out-of-distribution premise. Maintaining the benchmark requires choosing a new cutoff and rebuilding anchors, follow-up pools, and disguises. Finally, the static experiments do not fully measure what happens when ClaimNov sits inside a generation loop, repeatedly rejecting and causing rewrites. Authors leave the final idea distribution, compute cost, and potential reward hacking of that loop to future work.
9. Implications for AI Scientists
The practical contribution is a form of actionable feedback. If a problem claim is anticipated, an agent can preserve the task and search for a different method. If a method claim is anticipated, it can change the technical route. If only a result is already established, it can redesign the experiment. This is more useful than a binary “not novel” label because it identifies what should change in the next generation round.
ClaimNov also suggests a better reward signal for scientific agents. Most science is cumulative: new work reuses established tasks, datasets, and techniques while adding one or two genuinely new pieces. A reward based only on distance from the literature encourages unusual vocabulary and cosmetic differentiation. A claim-level reward can preserve legitimate inheritance while rewarding the part that is actually new.
The system should not be treated as an automatic arbiter of scientific priority. Its verdict depends on the pool, model cutoff, extracted claims, and graded entailment. A defensible deployment uses the evidence profile as decision support: it surfaces possible prior work and identifies repeated claims, while researchers inspect the underlying papers and judge whether the conditions are truly equivalent. The contribution is a more auditable representation of the problem, not a claim that machine judgment has solved novelty in science.
10. Conclusion
The paper contributes both an evaluation framework and a method. The framework separates duplicate detection, false alarms, source dependence, and claim attribution using disguised papers, later-realized ideas, reviewer disputes, and hybrid proposals. The method, ClaimNov, extracts typed contributions, retrieves same-dimension prior claims, grades partial coverage, aggregates a novelty score, and gives a final judge a complete claim-level profile.
ClaimNov catches 367 of 380 disguises, names the source in 366, rejects only 18 after the source is removed, and false-alarms on 1.5% of 400 later-realized ideas. On reviewer disputes and hybrids, it preserves more proposals that contain one established contribution while still exposing that contribution. The public code exposes the same extraction, retrieval, aggregation, and verdict stages described in the paper.
The broader conclusion is not that AI scientists already understand novelty. It is that many current checks trade sensitivity for excessive conservatism or achieve high catch rates by rejecting almost everything. Moving novelty judgment to typed claims and supplying the final judge with traceable evidence is a more faithful abstraction of how scientific contributions are structured. The approach remains bounded by retrieval and model judgment, but it makes those failures easier to locate.
Golden quote: A research proposal is not one claim. Novelty should be judged by asking what each contribution adds, which prior work already establishes it, and what still remains unseen.
