daily-scout: five-source daily discovery, four-dimension triage, human gate

The problem it solves

robotworld-daily-cron v3.2.0 is the site's content engine: every day it sweeps five sources for new material, scores it into a confirmation queue, and once a human approves, an agent deep-collects item by item until it is published. In the digital twin the worker is daily-scout, task key shaped like daily:2026-08-28, with idempotent start/finish pairing.

The core of this skill is not "can crawl". It is an asymmetric throttle: discovery batches (a script hits public APIs in seconds), ingestion runs serially one item at a time, enrichment metadata is written per item by an agent, and paper detail pages are deep-read one paper at a time. The triage stage is explicitly forbidden from downloading media or deep-reading papers; that work waits until a human confirms. Saving tokens is the secondary benefit. The primary one is refusing to let a machine decide what is worth collecting.

Five sources

Source	How it sweeps	Candidate granularity
twitter	Roughly 50 tracked robotics and embodied-AI accounts	Account level; a browser pass over the profile is still needed to find the specific video tweet
github	Keyword search, created within N days, at least 1 star, sorted by stars	Repository level
hf	Sorted by last update, filtered to updates within 7 days	Model and Space level (novelty rule: older than 7 days counts as stale)
blog	Official blog RSS, feed list in BLOG_FEEDS	Article level, verified-working feeds only
paper	arXiv last N days, embodied / world-model / VLA / humanoid topics	Paper level, abstract included

SERVER=root@43.162.123.152
ssh -p 2222 -F /dev/null -o BatchMode=yes $SERVER \
  "docker exec -w /app robotinfo-api python /app/scripts/daily_cron/discover_sources.py --days 7 > /tmp/candidates.json"

Triage: four dimensions, three priorities

triage_candidates.py scores every candidate 0-10 as a weighted total, and each dimension has concrete signals:

• novelty: frontier lab, fresh release, number of keyword hits
• freshness: days since last update or publication, more than 7 days judged stale
• relevance: hits against the site's scope keywords
• impact: stars, likes, downloads, whether code is open, whether real hardware is demonstrated

P0 is heavy news (collect today), P1 routine, P2 skipped. P0 and P1 are written straight into content_queue_items with status=pending, deduplicated idempotently by URL, so sweeping the same source daily cannot flood the queue.

State machine and the human gate

Queue states run pending to confirmed to done or rejected, managed at /manage/queue. Each row shows the multi-dimension scores, a one-line summary, the raw signals and a preview entry: ingested items preview in-site (admins can see unpublished rows), uningested ones get the original URL. Publishing authority was settled on 2026-08-31 with two rules:

• A URL the user specifies by hand goes straight to ingest.py --publish, no queue
• Content the agent discovered itself only gets lightweight triage into the pending queue; the user clicks "confirm collection" in the admin panel, and deep collection runs only after the user summons the agent and says to process the confirmed queue

The gate looks slow. In practice it pushes the cost of casting a wide net to nearly zero and leaves the cost of deep reading for a human to authorise. Execution is serial to avoid concurrent SSH disconnects, and each finished item writes back status='done', done_at and content_id (where the content landed), which makes the queue itself an auditable collection ledger.

The definition of "published" was rewritten

Since 2026-09-04, published = collected + social kit complete. Content that is ingested but not distributed costs the growth flywheel a full turn, and a late OG card means social crawlers never get an og:image at all: they remember the 404 and do not retry. Stage 2.5 has a fixed order: render the bilingual 1200x630 OG cards first (kind taken from the content type: demo, paper, article, opensource), then hand-write a social rewrite per item, which also renders the 1080x1350 knowledge card, then pull the six-platform English bundle, then eyeball.

Two traps here. Knowledge cards cannot be backfilled in batch, because their content comes from the rewrite, and no rewrite means a 404. And rewrites must be written one at a time: hooks, fact lines and numbers have to come from the material actually collected, never invented from title plus summary. That is the same iron law as per-paper deep reading. Also, the api container has no curl or wget, so social endpoints are called with local curl plus X-API-Key against the public domain.

Stage 2.6: the back catalogue is the biggest inventory

Once the day's collection is distributed, a few items are pulled from the back catalogue. Measured on 2026-09-04, production held 291 published deep-read papers (published=true with detail_html_en over 3000 words) while social_rewrites had exactly 1 row, and Umami showed 12 utm referrals, all from chatgpt.com. Almost the entire inventory had never been distributed. The draft queue is the single entry point for this flow, so nobody has to dig through an SOP or guess what went out today:

curl -s "https://robotworld.top/api/social/rewrite?status=draft&limit=20&lang=en"

The cadence is 2-3 rewrites a day from the catalogue, prioritising recent items with a cover image and hard experimental numbers; drafts land in draft, ship the same or next day, then flip to published. Angles are staggered: insight for the first pass over an item, then number, howto, contrarian, timeline, so the same sentence never repeats across a feed. Angles and platforms that return zero referrals for several days get written less, and the quota moves to what works.

The second selection signal is on-site share clicks: telemetry_events rows with kind='share' carry a payload of {platform, kind, id}, which means a real person wanted to forward that item. When picking catalogue items to rewrite, share clicks outrank raw pageviews, because a pageview can be a search engine sweeping the page while a share click is someone willing to put their name next to it.

Honest limits

The cron only discovers; collection and enrichment need an agent present. This is not an installable open-source package either: it is bound to this site's content_queue_items table, the /manage/queue panel, the five source scripts and the social material chain. What transfers are three design decisions: cost-tiering discovery against collection (batch the sweep, serialise the collect), turning human approval into a state machine instead of a verbal agreement, and widening the definition of "published" to include distribution material. The third is worth the most. Most content sites are not short of inventory; they collect things nobody ever ships.