paper-search: the deliberately thin first leg of a three-leg paper relay
local/paper-search
Search, ingest, then deep-read one paper at a time — this skill owns only the first leg and deliberately makes no judgement; it lays the candidates out so a human can. Three routes: search_papers.py for the daily path, the arXiv API directly, and defuddle parse for a known ID. Eleven preset topics and two layers of relevance filtering: an arXiv category allowlist plus hard exclusion keywords, then one more pass by the agent against the site's editorial scope.
Our takeThe deliverable is four lines you can decide on in thirty seconds: arXiv ID and title, a one-line summary the agent wrote itself, a relevance call with reasons, and a link — the reasons column matters more than the title. The transferable idea is asymmetric throttling: search may list twenty candidates at once, ingestion is confirmed one at a time, deep reading happens one paper at a time, and auto-batch ingest followed by batch-generated detail pages is forbidden.
Where it sits in the chain
The paper line is a three-leg relay: paper-search (find) → robotworld-ingest (ingest) → paper-detail-generator (deep-read one paper at a time into a detail page). This skill owns only the first leg and is deliberately thin: it does not make the call, it lays the candidates out so a human can. Each leg has a boundary it cannot cross. Searching may be batched (a script sweeps arXiv by keyword and returns a dozen or two candidates at once); ingestion is confirmed one item at a time; detail pages are deep-read one paper at a time. Forbidden: auto-batch ingest followed by batch-generated detail pages. That prohibition traces to the deleted gen_paper_details.py incident in this repo.
Three ways to search
| Route | How | When |
|---|---|---|
1 search_papers.py (preferred) | Runs inside the production container; takes --query / --topic / --days N / --max N | The daily path. The two-layer filter is built in, so what comes back is already readable |
| 2 Raw arXiv API | export.arxiv.org/api/query, hand-assembling cat:cs.RO+AND+all:... with sortBy=submittedDate | When you need a combination the script does not expose, or want to control the sort field |
3 defuddle parse | Parses the abs page when the arXiv ID is already known | Not searching at all; reading one confirmed paper in |
SERVER=root@43.162.123.152
ssh -F /dev/null -o BatchMode=yes $SERVER \
"docker exec -w /app robotinfo-api python /app/scripts/search_papers.py \
--query 'humanoid stair climbing' --days 30 --max 20"There are 11 preset topics: embodied-ai, world-model, vla, vm, 3dgs, simulation, spatial-intelligence, humanoid, manipulation, robotics, reinforcement-learning. When a direction is known but keywords are not, --topic beats free text, because behind each topic sits a query string that has already been tuned.
Two layers of relevance filtering
The script does not hand back whatever it finds. It narrows by arXiv category first, then removes noise with hard exclusion keywords:
- Category whitelist:
cs.RO,cs.AI,cs.LG,cs.CV,cs.MA,eess.SY - Hard exclusions: astronomy, quantum, pure art, pure medicine, radar and remote sensing, and other hits unrelated to robotics
Clearing the script filter is only the pass mark; the agent then applies the site's editorial scope. The focus is robotics and embodied AI, so these get dropped even on a keyword hit: pure benchmark papers (numbers without a method contribution), GUI and mobile agents (not physical robots), non-robotics applications (agriculture, photovoltaics, medical ultrasound), pure cognitive or architecture reviews (no robotic method), niche 3DGS and graphics work unrelated to robots, pure BCI and neuroscience (no robot control), and software or security testing papers. The full criteria live in robotworld-ingest/SKILL.md, and the two must agree, otherwise search waves something through that ingestion rejects and the round trip is wasted.
Output format: a decision in thirty seconds
Results are not the arXiv abstract pasted back. Four lines per paper: arXiv ID plus title, a one-sentence summary the agent writes from the abstract rather than copying it, a relevance verdict (high / medium / low, with the reason), and the link. The summary is the actual product here: the reader is deciding "does this belong in the library", not "what does this say", which makes the reason column worth more than the title. Shape:
1. arxiv:2607.25895 — Paper title
Summary: method XXX achieves ZZZ in setting YYY
Relevance: high (directly about robot stair climbing)
Link: https://arxiv.org/abs/2607.25895After the user nods
ssh -F /dev/null -o BatchMode=yes $SERVER \
"docker exec -w /app robotinfo-api python /app/scripts/ingest.py --source paper <arxiv_url>"Ingestion only means "this paper has a metadata row". Two steps cannot be skipped: backfill bilingual titles and summaries with SQL (then read the row back, since a zero exit code is not proof of a write), and hand each paper individually to paper-detail-generator. Search can list 20 papers in one pass; deep reading does one at a time.
There is also an audit entry point: scripts/cleanup_papers.py audit rescans ingested papers by category and abstract and flags the ones that no longer fit the filter above. Running it after a scope change clears out-of-bounds papers from the existing corpus far faster than paging through the frontend.
Honest limits
This is an internal skill of this site, not a package you can install elsewhere, and there is no one-line install command to give: it depends on search_papers.py in the production container, egress to arXiv, Postgres, and the admin confirmation queue. What transfers is the method: a two-layer filter of category whitelist plus hard exclusions, an asymmetric throttle where searching batches but ingesting and interpreting never do, and a single-sentence summary with a relevance reason as the only deliverable. Port those three ideas to any vertical content site and the cost of "we ingested a pile of irrelevant things" drops sharply.