Skip to content
←Back to the radar

SOTA · embodied

Safety & Alignment

Jailbreak resistance, hallucination rate, tool misuse, long-task boundaries

Safety and alignment is the only domain where more capability means more problem: progress in the other twelve domains is unambiguously good news, while progress here is simultaneously progress in risk. Its relation to AGI is structural: a system that can autonomously complete economically valuable tasks necessarily holds permission to change the external world, and once permission is granted the cost of an error moves from "the answer was bad" to "the thing already happened". The criteria for safety are not "does it have a safety statement" but three observable quantities: jailbreak rate and refusal consistency, tool abuse and permission boundaries during long tasks, and the proportion of hallucinated or unverifiable claims. All three can be tied to evidence; slogans like "responsible AI" cannot. Coverage status stated plainly: the top of this ladder currently carries a single bridged asset (genesis-world at 30k stars), whose value lies in "rehearse in simulation before an irreversible action", which is safety methodology rather than a safety guardrail; dedicated alignment evaluations, jailbreak benchmarks and guardrail layers (red-teaming tooling, runtime guardrails) are not yet in the library. This is the thinnest cell today, and along with math it is a gap that needs dedicated work.

RULERJailbreak rate, hallucination rate, tool misuse, long-task boundaries

The ladder

1
Embodied AI

Genesis World: The Unified Multi-Physics Simulation Platform for Physical AI

Genesis AI's open simulation platform for physical AI: one Pythonic API over a multi-physics engine (rigid, FEM, MPM, PBD, SPH, Stable Fluid, IPC), the Nyx path tracer, and Quadrants, a cross-platform Python-to-GPU compiler. It is built to be the evaluation engine for robot foundation models rather than a data factory: 30,000 parallel environments at 43M FPS, and under a zero-shot real-to-sim protocol, sim-vs-real evaluation agreement of Pearson 0.8996 with mean maximum rank violation 0.0166. Apache 2.0, 29.9k stars.

30kGitHub stars
Genesis World: The Unified Multi-Physics Simulation Platform for Physical AI

Latest demos

10

Papers & deep reads

4

Open source

1

Boundaries & failure modes

The boundaries here are harder to write than in any other domain because this one measures failure rather than success. One, the asymmetry of evaluation: a single successful jailbreak constitutes a vulnerability, while ten thousand successful refusals prove nothing, so any "jailbreak rate of 0.x%" reading only means "under the attacks tried"; the attack surface is an open set and the number is optimistic by construction. Two, inconsistency in refusal is more dangerous than refusal itself: the same class of request changes policy after rewording, a language switch, a role framing, or being buried in a long context, and an automated adversary in production will find that inconsistency. Three, tool abuse and permission boundaries currently have almost no guardrails: once an agent holds a person's credentials, an overreaching action is indistinguishable from a human action in the logs, and most runtimes today have neither transaction boundaries nor rehearse-then-commit - which is why it is listed as the biggest production risk in the `agents` domain even though the problem belongs to safety and lives in agent implementations. Four, hallucination and unverifiable claims: models fill in facts that do not exist using training priors and do not signal uncertainty, which is the most expensive failure class in medical, legal and financial contexts; the countermeasure is not to claim elimination but to render confidence grades explicitly (A confirmed / B contested / C vendor-claim / D needs-reproduction) on every conclusion, so a reader can tell which sentence was verified. Five, safety is coupled to capability: alignment training often comes with capability regression and vendors rarely report both numbers, so any claim of "safer" must be followed by "measured on which capability axis". Six, the data boundary: with no independent safety board wired in, water-level judgement in this domain rests on methodology and cross-domain readings, and none of it is given confirmed.

As an Amazon Associate, we earn from qualifying purchases.