video-crawler: the six gates between a dropped URL and a published item

How many gates a URL passes before it ships

robotworld-ingest v2.3.0 is the site's front door for external content: the user drops a Twitter/X, YouTube or arXiv URL and says "collect it", and this skill turns it into a publishable item. The digital twin has two workers for it: video-crawler for video, paper-analyst for papers.

The pipeline has six stages. Three are run by scripts, three must be run by an agent:

Stage	Who	Output and acceptance
1 Collect media	ingest.py (script)	Video and cover on disk plus an ingestion_records row; source auto-detected from the URL, existing succeeded rows skipped idempotently
2 Metadata	Agent, one item at a time (enrich.py set)	Bilingual title and summary, English canonical tags, category; written from raw_text, never from a template
3 Paper detail page	Agent, one paper at a time (separate skill)	Six-round deep read, quality_score at 8.0 or above
3.5 Two webp tiers	Script (in-container, idempotent, incremental)	.card.webp 800px/q78 for list cards, .body.webp 1400px/q80 for article figures and above-the-fold imagery
4 Live verification	curl	API returns video and thumb, title and summary non-empty, frontend responds 200
5 Quality gate	Machine score plus human	Paper detail pages need a human confirm at /manage/confirm before published=true
6 Social kit	Agent, one item at a time	OG card, rewrite, knowledge card, six-platform English bundle

Video spec: download at 1080p, publish at 720p

The rule looks self-contradictory. It is really an accounting split between picture quality and bandwidth. The yt-dlp format selector prefers the highest quality with a long edge under 1920px; after download, downscale_video_if_needed() probes with ffprobe and transcodes back to a long edge under 1280px only when the source exceeds the site spec (libx264 -preset veryfast -crf 20, aac 128k, faststart). The download rung does not drop to 1280: a platform's own 720p rung is already a second-generation encode, and transcoding the site's 720p from its 1080p rung is visibly cleaner.

The selector has one detail that is easy to get wrong: width and height must both be constrained. For portrait video the height is the long edge, so writing only height<=N filters out 720x1280 and falls back to 480x852, which means deliberately choosing the worse stream. That was a real incident.

scale=min(1280\,iw):min(1280\,ih):force_original_aspect_ratio=decrease,pad=ceil(iw/2)*2:ceil(ih/2)*2

scale has to be written in min() form: a bare scale=1280:1280 is unconditional interpolation and will upscale a 712x294 clip to 1280x520, softer and heavier. The comma inside min() must be escaped or the filtergraph fails to parse (rc=234). Temp files must end in .mp4 because ffmpeg picks the container from the extension. ffmpeg has to run inside the container, since the host build is a stripped one without libx264; the server has 2 cores, so manual transcodes take nice -n 19 and run one at a time.

Covers: take a bright frame from the middle of the video

A native cover is usually the first frame, which is usually black. After download the adapter calls image_is_dark(), then generate_video_thumbnail() samples frames at 10%, 33%, 60% and 80% and takes the first non-black one, falling back to a dense 5%-95% sweep if all are dark and finally to the highest-scoring frame if nothing has usable luminance. The result is compressed back to a long edge under 1280px. The blackness test is deliberately conservative: peak luminance under 50/255, or mean luminance under 20/255 with fewer than 0.5% of pixels at 120 or above. An image with a black background but clear bright subject matter does not count as black, otherwise plenty of legitimate covers would be needlessly regenerated.

The hard rule in stage 2: read back what you wrote

Metadata is written per item by an agent reading raw_text. Tag storage order is fixed: English canonical tags first, Chinese retrieval aliases last, with the frontend choosing per site language. One hard rule here came from a real incident: after any SQL or script touches metadata, read the row back immediately; a zero exit code is not evidence of a write. With the ZEST paper, the SQL backfilling the Chinese title failed silently on shell quoting, the agent published without checking, and the Chinese site showed an English title, an English abstract and a tag list containing only the placeholder {Twitter}. Nested-quoting UPDATEs now always go into /tmp/fix.sql, run with docker cp plus psql -f, and any raw SQL edit to titles, summaries or tags is followed by reindex_search.py --fix, otherwise search and display disagree.

Long batches and background bookkeeping

yt-dlp really downloads: 6 URLs take 5-10 minutes, 18 take about 30, while SSH is typically dropped by the server after 5-10 minutes. Processes started by docker exec outlive the SSH session, so batches of 4 or more launch with setsid plus < /dev/null, fully detached, then poll the process and the DB row count every 60-180s. nohup ... & is unreliable: the channel often dies before nohup finishes writing the PID, so the command never runs. Running several ingests concurrently against the same server gets connections killed, so everything is serial.

The thing background batches most often lose is the closing status report. The process has left the session, attention moves to the next stage once polling says FINISHED, and a status='running' row stays in the DB forever, leaving that employee permanently "working" in the digital twin office. Three such rows accumulated in production. Fixed structurally on 2026-09-08: ingest.py reports itself, wrapping the whole batch in try/finally so that normal completion, partial failure and mid-run crash all emit a finish. The agent adds nothing, and must not finish manually either, which would insert a duplicate row.

Stage 6: publishing is not the end

Every item that publishes successfully gets its social kit immediately, never batched for later: when og:image points at a 404, social crawlers give up on that card permanently and do not retry. The order is OG card (write), rewrite plus 1080x1350 knowledge card (authorship), English post bundle (read), eyeball. Rewrites are written one at a time with hooks, fact lines and numbers drawn from the material actually collected; no script may fill that table from title plus summary. Before eyeballing, compress cards with scripts/rw_tools.py small to a long edge under 1280px and under 200KB, because the budget is encoded bytes, not pixel count.

Honest limits

This is an internal skill, welded to yt-dlp, ffmpeg, Postgres and the bind-mounted data/static/img/ inside the production container. It cannot be installed as an open-source package and there is no one-line install command. What transfers is a set of engineering constraints earned from production incidents: separating the download rung from the publish rung, taking covers from a bright mid-video frame instead of trusting the platform's first frame, reading back every database write, detaching long jobs with setsid and having the script close its own status report, and changing the URI whenever published static content changes, because the CDN cache key ignores the query string. All of these hold for any site that hosts user-generated media.