blog-collect: turn an external blog URL into a long-term hosted bilingual article
local/blog-collect
Our blog ingestion line, v2.2.0, executed in the digital-twin office by blog-crawler. Every image and video in the original is downloaded and served locally, the body is translated and adapted in full by the agent per article (never a summary), and publishing runs publish_blog.py into POST /api/blog/media and /api/blog/publish, landing in the Postgres articles table that the frontend reads at runtime. Editorial scope is hard-coded into the skill.
Our takeEditorial criteria live in the skill rather than in memory, because selection scope is the first thing to drift in a pipeline that runs for months. Two rules earned the hard way are worth copying: a cover must never be an .mp4 (cover_image renders as an img src, and two cards went fully black on 2026-08-25), and a bare angle bracket inside KaTeX must be written as an HTML entity, or the formula truncates there and the rest of the article silently disappears without an error.
What this line owns
blog-collect is our blog ingestion line, currently v2.2.0, executed in the digital-twin office by the agent blog-crawler. It does exactly one thing: turn an external blog URL into a bilingual article we host on our own storage for the long term. "Long term" is the point — every image and video in the original is downloaded and served locally, the body is translated/adapted in full by the agent per article (never a summary), and publishing writes to Postgres, which the frontend reads at runtime through /api/articles.
It is deliberately not a general-purpose scraper. The editorial scope is hard-coded into the skill: embodied AI, robot learning and manipulation/navigation, world models, VLA/VLM, egocentric data, sim-to-real, robot datasets and hardware/sensors, physical-AI infrastructure. Pure SaaS marketing, non-robotics AI applications and promotional copy are out of scope. Writing the selection criteria into the skill instead of relying on memory matters because editorial drift is the first thing that rots in a pipeline that runs for months.
Five non-negotiable rules
| Rule | What it says | Why |
|---|---|---|
| Body | The Chinese version is a full translation/adaptation of the English original, not a summary | A summary-style repost has no search value and no retention value; readers leave once they notice three paragraphs are missing |
| Promotion | Strip every "Work With Us" / "contact sales" / "custom services" block from the original | We do not send traffic to vendors, and readers should not meet unrelated sales copy on our pages |
| Attribution | Every article ends with the original URL | Clear attribution is the precondition for running this line indefinitely |
| Media | Images and videos land in data/static/img/blog/<id>/ and are referenced locally, never hot-linked | Hot links die when the source site redeploys; local paths ride Caddy's /static/img/* → max-age=86400, including video Range requests |
| Cover | Must be .jpg/.png, never .mp4 | cover_image renders as <img src>; a video file cannot render and the card goes blank |
The cover rule was learned the hard way. On 2026-08-25, collecting Figure AI's Helix 02 and Helix laundry-folding posts, the original hero was a video and the .mp4 was written straight into cover_image; both list cards rendered black. The current handling: when the original only has video, extract a frame with ffmpeg, then eyeball it (downsample with scripts/rw_tools.py small first — never feed a vision model the original file).
Spec gates: video ≤720p, images ≤1280px
Source videos are often 4K, which slows playback and burns bandwidth if ingested raw. The rule is "cap the ceiling, never the quality": prefer a source whose long side is ≤1280px; when only higher resolutions exist, transcode on the server with ffmpeg. If the source is below 720p, keep it as-is — never reach for a lower rung. Images follow the same rule: anything above 1280px on the long side is downscaled proportionally with Pillow (format and alpha preserved) before ingest, because oversized images blow up the vision context of the downstream review step.
# probe a manually downloaded video, transcode when the long side >1280 (nice it, don't saturate prod CPU)
ffprobe -v error -select_streams v:0 -show_entries stream=width,height -of csv=p=0 in.mp4
nice -n 19 ffmpeg -i in.mp4 -vf "scale='min(1280,iw)':'min(1280,ih)':force_original_aspect_ratio=decrease" \
-c:v libx264 -preset veryfast -crf 20 -c:a aac -b:a 128k -movflags +faststart out.mp4One counter-intuitive detail: scale must use min(BOX,iw) semantics. Writing scale=1280:1280 is unconditional interpolation and will upscale small media — a 712x294 demo GIF was once blown up to 1400x578 in production, blurrier and roughly 13MB heavier.
A bare < inside a formula eats half the article
Bodies are stored as pre-rendered HTML in body_zh/body_en, injected with dangerouslySetInnerHTML and rendered by KaTeX/Mermaid. That means <, > and & inside KaTeX must be HTML entities: z_{<t} has to be written z_{{<t}, otherwise the browser treats the bare < as a tag start, the formula is truncated there, and the rest silently disappears. Commands such as \leq and \geq contain no bare angle brackets and need nothing. The same trap is worse on the paper line, so both share the rule.
Publishing path: DB + API, not SSH file copies
Since 2026-09-01 publishing goes scripts/publish_blog.py → POST /api/blog/media + POST /api/blog/publish → the Postgres articles table (with search-embedding refresh). The server enforces the ≤1280px image cap, the script reads back to verify, and any failed step exits non-zero printing the server error; rerunning is safe (whole-record upsert, media overwritten by name). /api/* is globally Cache-Control: no-store, the domestic CDN does not cache it, so a publish is live for every user immediately.
export AGENTS_API_KEY="$(ssh -p 2222 root@43.162.123.152 \
'docker exec robotinfo-api sh -c "echo \$AGENTS_API_KEY"')"
python3 scripts/publish_blog.py <article-id>
python3 scripts/publish_blog.py <article-id> --skip-media # body/metadata onlyThe old "SSH the files → cp on the server → docker exec seed_blog.py" chain is retired (no transaction, no validation, easy to drift) and survives only as an offline fallback. List and detail pages are force-dynamic, so a new article needs no frontend rebuild; only changes under frontend/ wait for CI to produce an image, then docker compose pull && up -d. Never build on the server.
Publishing is not the finish line
99% of page weight is images (measured median 2.64MB of image payload per detail page, max 19.89MB, while the body HTML is 9KB), so step 10.5 runs the two webp tiers: build_card_thumbs.py --only articles (800px/q78 list cards) and build_body_thumbs.py --only articles (1400px/q80 body and hero images). Both are idempotent, run inside the api container, and require no container restart and no deploy. The webp files are derivative siblings — the original bytes are untouched, and click-to-zoom swaps back to the original.
Step 11 adds the social kit: POST /api/social/card?kind=article&id=<id>&both_langs=true renders two 1200x630 OG cards, then an agent-written social rewrite is stored (which also renders the 1080x1350 knowledge card), and finally the six-platform English bundle is fetched. OG cards cannot be backfilled later at leisure — when og:image 404s, social crawlers give up on that card permanently and never retry. The whole task is clocked in and out with scripts/agent_report.py start/finish blog-crawler --task-key blog:<slug>, success or failure, and failures show red on /agentsview.
What transfers, what does not
To be straight about it: this is an internal skill. It is bound to our directory layout, API endpoints, Caddy cache rules and container names, so it cannot be dropped into someone else's agent, and there is no public one-line install command. What does transfer is the method — full translation instead of summary, localise all media, covers must be images and must be eyeballed, publish through DB+API rather than file sync. Those four are site-independent and any bilingual content operation can adopt them as-is.