Riemann-1.0: An Embodied World Action Model for Physical AI

Overview

Riemann-1.0 is a fully causal autoregressive World Action Model (WAM) that unifies executable robot policy and action-conditioned world simulation within a single architecture. Developed by Riemann Dynamics, it transforms 232K+ hours of heterogeneous embodied experience — egocentric human videos, handheld-gripper demonstrations, and robot trajectories — into a causal action-video model that follows real robot interaction: history predicts the next action chunk, actions condition future visual dynamics, and new observations refresh the context for closed-loop control.

The system achieves state-of-the-art results across simulation benchmarks (99.0% LIBERO, 94.3% RoboTwin 2.0, 62.6% RoboCasa365) and real-world manipulation (85.0% average success rate, 94.43% progress success rate), exceeding the strongest open-source baseline by 15 SR points on average in real-world tasks.

The Challenge: Scaling Embodied Experience

Scaling World Action Models depends fundamentally on scaling embodied experience rather than robot trajectories alone. However, embodied experience is inherently heterogeneous across three data sources that differ substantially in observation modalities, action representations, temporal granularity, and supervision fidelity:

• Egocentric human videos (200K+ hours) offer the broadest diversity of real-world interactions, object-centric manipulation skills, and long-horizon task compositions. They provide scalable interaction knowledge but lack executable action labels.
• Handheld-gripper and wearable demonstrations (12K+ hours) provide structured end-effector trajectories that naturally bridge human interactions and robot-compatible control, reducing the embodiment gap.
• Robot trajectories (20K+ hours) provide precise embodiment-specific states and directly executable actions for downstream policy learning, but are fragmented across embodiments, camera layouts, and control conventions.

Rather than being redundant, these sources complement each other: human videos provide scalable interaction knowledge, handheld-gripper demonstrations reduce the embodiment gap between humans and robots, and robot trajectories provide executable control supervision. Together, they establish a natural supervision hierarchy for Progressive Embodied Pretraining.

Data Infrastructure

Riemann-1.0 builds a unified embodied data infrastructure that automatically transforms heterogeneous embodied experience into a common action-level video–state–action representation. The infrastructure consists of three tightly coupled components: multi-source embodied experience collection, a unified embodied data engine, and progressive supervision with multi-source data balancing.

Unified Embodied Data Engine

Although the three data sources provide complementary supervision, their native formats are fundamentally different. Egocentric human videos provide visual observations with weak language supervision, handheld-gripper demonstrations provide structured end-effector trajectories and gripper states, while robot datasets differ substantially in embodiment-specific states, action spaces, coordinate systems, and control frequencies.

The data engine converts all data into a common action-level trajectory representation consisting of language instructions, embodiment identities, visual observations, states, actions, and semantic metadata. For each embodiment, state and action streams are temporally aligned, normalized, and organized under embodiment-specific canonical definitions, while validity masks handle variable-dimensional actions without forcing heterogeneous embodiments into a shared physical action space.

For egocentric human videos, the pipeline applies a hierarchical VLM-based annotation system that decomposes long recordings into task-level and action-level clips with structured semantic annotations. To bridge visual interactions and executable actions, it reconstructs continuous 3D hand trajectories using MANO-based hand reconstruction and VGGT camera pose estimation, enabling hand trajectories to be organized in a consistent coordinate system under egocentric camera motion.

Progressive Supervision and Data Balancing

The processed corpus forms a continuum of supervision with progressively increasing action fidelity. Large-scale egocentric human videos first provide broad interaction dynamics and weak action supervision through pseudo actions generated by a frozen Latent Action Model. Human videos with reconstructed 3D hand trajectories, handheld-gripper demonstrations, and heterogeneous robot trajectories subsequently provide continuous real-action supervision. Finally, high-quality robot trajectories specialize the model for embodiment-specific action prediction.

The raw corpus exhibits severe long-tail distributions across data sources, scenes, tasks, skills, objects, and robot embodiments. Rather than balancing raw data volume, Riemann-1.0 constructs a unified semantic taxonomy and performs semantic-aware sampling across multiple semantic dimensions, preserving the scale advantage of large human datasets while ensuring sufficient exposure to long-tail manipulation skills and low-resource robot embodiments.

Model Design: Fully Causal Action-Video World Model

Riemann-1.0 adopts a fully causal autoregressive factorization over actions and visual latents. At each autoregressive step, the model predicts the current action from the preceding visual observations, robot states, and action histories. The action is then appended to the causal context and used as a condition for predicting the corresponding visual latent. This ordering follows the causal interaction process of real robots, where actions are executed before their visual consequences are observed.

The fully causal factorization is formalized as:

$$ p(a_{1:T}, z_{1:T} \mid z_0, s_0, c) = \prod_{t=1}^{T} p(a_t \mid z_{<t}, s_{<t}, a_{<t}, c) \, p(z_t \mid z_{<t}, s_{<t}, a_{\leq t}, c) $$

where $c$ is the task condition, $z_t$ is the visual latent, $s_t$ is the robot state, and $a_t$ is the action. Robot states are not generated by the model; they are directly observed from the environment or replayed from recorded trajectories and injected as conditioning signals. During policy deployment, the predicted visual latent is replaced by the real observation returned by the environment. During visual simulation, the predicted visual latent is recursively fed back to generate future action-conditioned observations.

Comparison with Existing WAM Paradigms

Existing World Action Model paradigms differ in how they model the interaction between robot actions and future visual observations, leading to different trade-offs among policy learning, visual simulation, and inference efficiency:

• DreamZero-style methods jointly denoise visual latents and actions as $p(z_{1:T}, a_{1:T} \mid z_0, s_0, c)$. This tightly couples action and visual generation but requires jointly modeling modalities with different dimensionalities, temporal resolutions, and optimization characteristics.
• Video-first methods (LingBot-VA) first generate future observations and then infer actions from the predicted visual trajectory, introducing additional inference latency.
• FastWAM-style methods decouple video and action generation using two separate DiTs, improving modularity while still conditioning action prediction on predicted visual features rather than causal interaction histories.

Riemann-1.0 instead adopts a fully causal autoregressive action-video formulation, where robot actions are either predicted online or externally specified before future visual latents are generated. This enables a single model to function both as an executable robot policy and as an action-conditioned visual world simulator.

Architecture Details

The language condition specifies the task instruction together with the robot embodiment and camera-view configuration, encoded by a T5 encoder and injected through cross-attention. A numerical embodiment ID selects embodiment-specific action/state projections and prediction heads, enabling heterogeneous robots to share a unified backbone while preserving embodiment-specific control parameterizations.

Visual observations are packed into a unified embodiment-specific multi-view canvas, resized to model resolution, encoded into latent representations using the Wan VAE, and converted into transformer tokens through a 3D patch embedding. Actions and robot states are transformed into embodiment-specific canonical representations and normalized using per-embodiment statistics, removing differences in coordinate systems, physical units, and action dimensionalities.

For heterogeneous robots, action and state vectors are mapped to an embodiment-specific canonical order, padded to fixed model dimensions $d_a \leq D_a$ and $d_s \leq D_s$. The raw embodiment ID is mapped to a compact category slot, which indexes category-specific linear projections for action and state tokens. The action prediction head is category-specific as well, so the model shares temporal and visual reasoning across embodiments while preserving the numerical action and state spaces of each robot.

Training Objective

Riemann-1.0 applies the flow-matching objective to two prediction heads. The visual-latent head predicts the velocity of noisy latent tokens:

$$ \mathcal{L}_z = \mathbb{E}_{z, \epsilon, \sigma} \left[ \| v^z_\theta(z_\sigma, \sigma, z_{<t}, s_{<t}, a_{\leq t}, c) - (\epsilon_z - z) \|_2^2 \right] $$

The action head predicts the velocity of noisy action tokens:

$$ \mathcal{L}_a = \mathbb{E}_{a, \epsilon, \sigma} \left[ \| v^a_\theta(a_\sigma, \sigma, z_{<t}, s_{<t}, a_{<t}, c) - (\epsilon_a - a) \|_2^2 \right] $$

The final objective balances visual dynamics and action prediction:

$$ \mathcal{L} = (1 - \lambda) \mathcal{L}_z + \lambda \mathcal{L}_a $$

where $\lambda$ controls the relative weight of the action objective. Separate validity masks are applied to the latent and action losses, excluding padded frames, invalid latents at episode boundaries, padded action channels, and invalid low-level action steps. The same validity information is applied during noise injection to avoid train-test mismatch where padded channels would contain random Gaussian noise during training but zeros during deployment.

Structured Attention for Causal Training

To match online execution, each prediction should not access information unavailable at inference time. Riemann-1.0 employs a structured attention mask instead of unrestricted self-attention. Clean tokens attend to previously observed clean tokens under the causal mask. Within a local generation block, bidirectional attention models intra-block structure. Noisy target tokens attend to preceding clean tokens and to other tokens within the same noisy block, but not to future clean tokens, preventing leakage from future observations while preserving teacher forcing over the observed trajectory prefix.

Progressive Embodied Pretraining

Riemann-1.0 is trained with a three-stage curriculum that progressively shifts supervision from video-only pseudo actions to real action trajectories. This design is motivated by a practical data imbalance: video-only data is much larger and visually diverse but lacks executable robot actions, whereas real action datasets are smaller but provide direct supervision over robot behavior.

Stage 1: LAM-Action Bootstrap ($\lambda = 0.1$)

The first stage uses only unlabeled human manipulation videos. Since these videos provide rich visual interaction but lack executable action labels, a Latent Action Model (LAM) is trained and then frozen as a pseudo-action annotator. LAM is trained as a latent-action VAE over pairs of adjacent video frames. Given two frames $(x_t, x_{t+1})$, the encoder produces a posterior distribution over a 32-dimensional latent action space, and a decoder reconstructs the next frame from the current frame and the sampled latent action:

$$ \mathcal{L}_{\text{LAM}} = \| \hat{x}_{t+1} - x_{t+1} \|_2^2 + \beta \, D_{\text{KL}}(q_\phi(z_t \mid x_t, x_{t+1}) \| \mathcal{N}(0, I)) $$

The KL term is deliberately weak to regularize the latent distribution without collapsing the transition code into an overly generic prior. After training, the posterior mean $\mu$ is used as the deterministic pseudo action, eliminating sampling noise when annotating large-scale video corpora. With $\lambda = 0.1$, this stage mainly initializes the visual dynamics backbone and teaches the model how action-like motion tokens correlate with future visual change.

Stage 2: Trajectory-Grounded Alignment ($\lambda = 0.5$)

The second stage replaces pseudo actions with action trajectories from a mixed multi-embodiment corpus, including UMI demonstrations, robot trajectories, and human videos annotated with 3D hand poses. Each trajectory is converted to the same WAM temporal interface: visual frames are packed, encoded into VAE latents, actions are grouped into dense chunks aligned with each visual latent, and states consist of the initial state plus one state after every latent transition. For heterogeneous robots, action and state vectors are mapped to an embodiment-specific canonical order, normalized, and padded. With $\lambda = 0.5$, action prediction becomes a central training signal while retaining visual-latent prediction as a dynamics regularizer.

Stage 3: Robot-Policy Enhancement ($\lambda = 0.9$)

The final pretraining stage uses only high-quality robot demonstrations with real action-state supervision, excluding human video data, LAM pseudo actions, UMI trajectories, and 3D hand-pose trajectories. The action weight increases to $\lambda = 0.9$, focusing optimization primarily on state-conditioned action prediction and trajectory consistency while the latent objective continues to regularize the learned dynamics. This stage calibrates the prior toward real actuator conventions, contact-rich execution, and policy stability.

Real-World Experiments

After pretraining, Riemann-1.0 is adapted to real-world deployment through real-robot post-training. Four representative manipulation scenarios are built on the Tianji Marvin dual-arm robot, covering order-constrained rigid-object stacking, deformable-object manipulation, long-horizon tabletop rearrangement, and long-horizon kitchen storage.

Four Real-World Tasks

Ordered cube stacking requires the robot to follow a language instruction identifying colored cubes and assembling a four-layer stack in a specified color order. Small alignment errors at lower layers are amplified by later placements, making this a test of precise visual recognition, grasp-sequence planning, and pick-and-place execution.

Clothes folding requires manipulating a deformable garment that can wrinkle, stretch, and drift in pose during contact. The robot must continuously adapt its action strategy based on the observed state, stressing deformable-object perception, bimanual coordination, and fine-grained contact control.

Desk organization requires identifying and rearranging multiple scattered desktop objects — tissues, toys, chargers, tape, blocks, and pens — into designated locations. A key challenge is inserting slender objects such as pens into a pen holder, requiring accurate perception of the pen pose and the holder opening.

Kitchen organization requires completing an end-to-end tidying task: localizing and grasping utensils, placing them into target storage, inserting plates vertically into a drying rack, stacking bowls, and placing the bowl stack on the upper rack — testing long-horizon sequencing and stable placement under cluttered conditions.

For each task, 15 demonstrations are collected using human teleoperation (three hours per task). Instead of training task-specialized individual models, all demonstration data is aggregated to jointly fine-tune a single generalist model. The action loss weight is increased to 0.95 in post-training to improve action execution performance.

Real-World Results

Riemann-1.0 achieves the best average performance among all compared models, with an average SR of 85.0% and average PSR of 94.43%. It maintains at least 80.0% SR on all four tasks and exceeds 91.0% PSR on every task:

Model	Cube Stacking SR/PSR	Kitchen SR/PSR	Clothes SR/PSR	Desk SR/PSR	Avg SR/PSR
DreamZero*	15.0/20.0	15.0/16.6	15.0/12.5	15.0/33.3	15.0/20.6
τ0-WM	15.0/26.2	15.0/20.0	15.0/24.6	20.0/57.2	16.3/32.0
LingBot-VLA	15.0/31.5	20.0/80.2	80.0/88.4	20.0/76.9	33.8/69.3
π0.5	40.0/59.0	20.0/37.2	45.0/73.5	40.0/73.1	36.3/60.7
LingBot-VA	40.0/66.6	20.0/30.0	80.0/85.0	40.0/80.1	45.0/65.4
G0.5	80.0/89.5	35.0/47.8	85.0/90.0	80.0/93.4	70.0/80.2
Riemann-1.0	85.0/91.6	90.0/98.4	85.0/92.5	80.0/95.2	85.0/94.4

On kitchen organization, Riemann-1.0 obtains 90.0% SR while the next best SR is 35.0% from G0.5 — a 55-point gap. On ordered cube stacking, it reaches 85.0% SR and 91.6% PSR, improving over the strongest baseline G0.5 by 5 SR points and 2.1 PSR points. These results suggest that post-training preserves the general manipulation prior learned during pretraining while adapting it to the target real-world robot, camera layout, and household task distribution.

Real-World Task Demonstrations

Compositional Generalization and Out-of-Domain Evaluation

Beyond the four post-training tasks, Riemann-1.0 is tested on held-out instructions to evaluate whether the deployed policy can reuse learned manipulation skills in new configurations. Two settings are evaluated:

Compositional generalization uses seen object categories but recombines them into unseen task goals — for example, placing a specified cube into a target receptacle such as a bowl or plate, or following multi-step instructions specifying object color, receptacle color, receptacle type, and execution order.

Out-of-Domain (OOD) evaluation uses unseen scenes, objects, and task goals not included in the real-world post-training set, with no additional demonstrations or parameter updates — for example, placing a Rubik's cube into a storage box or placing a towel into a basin.

Setting	Task	Riemann-1.0	π0.5	LingBot-VA
Compositional	Cube-to-bowl/plate placement	80.0%	60.0%	30.0%
Compositional	+ color-order constraint	50.0%	30.0%	0.0%
OOD	Rubik's cube to storage box	100.0%	50.0%	30.0%
OOD	Towel-to-basin placement	70.0%	50.0%	30.0%
Overall average		75.0%	47.5%	22.5%

Riemann-1.0 achieves an overall average SR of 75.0% across all four held-out tasks, compared to 47.5% for π0.5 and 22.5% for LingBot-VA. The model completes all Rubik's-cube-to-storage-box trials and retains 70.0% SR on towel-to-basin placement, demonstrating that the pretrained manipulation prior and real-world post-training transfer to new task compositions and out-of-distribution household scenarios rather than merely fitting the training tasks.

Simulation Benchmarks

Riemann-1.0 is evaluated on three simulation benchmarks, following the standard protocol of each with task success rate as the primary metric. One visual latent corresponds to 16 low-level action steps across all benchmarks.

RoboCasa365

On RoboCasa-365, Riemann-1.0 is pretrained on 300 tasks and finetuned on the Target-50 split. It achieves the best performance across all three categories — Atomic-Seen, Composite-Seen, and Composite-Unseen — improving the average success rate from 54.2% to 62.6% over the strongest baseline ABot-M0.5. Notably, it improves Composite-Seen and Composite-Unseen by 11.7 and 10.7 percentage points respectively, demonstrating strong compositional generalization.

Model	Atomic-Seen	Composite-Seen	Composite-Unseen	Average
GR00T-N1.5	60.6	35.0	33.3	43.7
Fast-WAM	59.1	36.4	33.2	43.5
LingBot-VA	63.5	37.3	32.1	45.1
ABot-M0.5	70.6	44.3	45.6	54.2
Riemann-1.0	74.2	56.0	56.3	62.6

RoboTwin 2.0

On RoboTwin 2.0, Riemann-1.0 is trained on 50 bimanual tasks with clean scenes (50 demonstrations per task) plus 25,000 demonstrations from heavily randomized scenes. It achieves the highest average success rate of 94.3% across 50 bimanual tasks.

Model	Clean	Randomized	Average
π0.5	82.7	76.8	79.8
LingBot-VLA	86.5	85.3	85.9
Fast-WAM	91.9	91.8	91.8
G0.5	93.7	92.8	93.3
ABot-M0.5	94.0	94.2	94.1
Riemann-1.0	94.6	94.0	94.3

LIBERO

On LIBERO, Riemann-1.0 reaches the overall average of 99.0%, achieving 99.6% on Spatial, 100.0% on Object, 97.6% on Goal, and 98.6% on Long suites.

Method	Spatial	Object	Goal	Long	Average
π0.5	98.8	98.2	98.0	92.4	96.9
GR00T-N1.6	97.7	98.5	97.5	94.4	97.0
Fast-WAM	98.2	100.0	97.0	95.2	97.6
Being-H0.5	99.2	99.6	99.4	97.4	98.9
G0.5	98.4	100.0	98.6	98.6	98.9
Riemann-1.0	99.6	100.0	97.6	98.6	99.0

Autoregressive Online Inference

The training attention mask naturally enables online autoregressive inference with a growing KV cache. At the start of an episode, the first packed multi-view frame is encoded into $z_0$, and the initial state $s_0$ is embedded as a clean state token. Together with the text embedding, these tokens initialize the cache.

At each denoising step, the model samples a Gaussian noisy action tensor and denoises it using the flow scheduler while attending to the cached clean history and text condition. The resulting clean action chunk is appended to the cache and executed for $a_{pf}$ low-level control steps. The environment returns new multi-view observations, which are packed, encoded into $z_t$, and appended together with $s_t$ to the cache as clean context for the next step.

During real-robot execution, the next latent is obtained by encoding the actual observation returned by the environment. During simulation, the same causal context together with the generated action chunk is fed to the latent head, and the predicted visual latent is decoded as the simulated camera observation. For long-horizon rollouts, the cache is maintained within a sliding temporal window — once full, the cache is reset and the most recent observation becomes the new context frame, enabling continuous online control while preserving causal consistency.

World Simulator: Action-Conditioned Visual Rollout

Riemann-1.0 can also be used as an action-conditioned visual simulator. In this mode, the model receives the current visual observation, task prompt, robot state, and a candidate future action trajectory, and predicts the visual consequences of executing that trajectory. The action trajectory is represented with the same embodimentpecific action interface used during policy training, then embedded as action tokens and inserted into the causal action-video sequence.

At inference time, the current observation is encoded into the VAE latent space as clean visual context. The future action chunk — provided by the policy head, sampled from a candidate plan, or taken from a recorded trajectory — conditions the latent head to roll out future visual latents, which are then decoded back into RGB video. For longer horizons, the generated or observed visual output can be appended back to the context and the procedure repeated autoregressively.

This gives Riemann-1.0 a data-driven simulator interface: actions are treated as controllable inputs, while future camera observations are produced as the simulated consequences. The examples cover humanoid robots, dual-arm systems, single-arm platforms, dexterous-hand robots, and simulation benchmarks, demonstrating that Riemann-1.0 learns a shared action-conditioned visual dynamics prior that generalizes beyond a single robot body.

Conclusion

Riemann-1.0 addresses two central challenges in scaling robot foundation models: how to unify heterogeneous embodied experience across egocentric human videos, handheld-gripper demonstrations, and robot trajectories, and how to jointly model robot actions, states, and world evolution in a causal form aligned with real interaction. By combining a unified embodied data infrastructure with Progressive Embodied Pretraining over 232K+ hours of embodied experience, Riemann-1.0 transfers large-scale interaction knowledge from weak supervision to executable robot control.

Across simulation and real-world evaluations, it achieves state-of-the-art performance: 62.6% on RoboCasa365, 94.3% on RoboTwin 2.0, 99.0% on LIBERO, and 85.0% SR with 94.43% PSR on long-horizon real-world manipulation. These results show that Riemann-1.0 provides a scalable path for transforming large-scale embodied experience into generalizable robot manipulation capabilities.

Source: Riemann Dynamics Official Website · Paper PDF