1X World Model: Simulating the Future to Evaluate Robot Policies

1X World Model: Simulating the Future to Evaluate Robot Policies

At 1X, we build robots that help humans in the most diverse environment imaginable: people's homes. To deploy our Redwood AI model safely and reliably, we need to anticipate its policy behavior across all that it can encounter — from retrieving a rarely-used kitchen gadget tucked away in a cluttered drawer, to navigating a living room unexpectedly rearranged overnight. Physically evaluating each policy across these varied scenarios would take several lifetimes. How can we accelerate the evaluation of generalist robot models?

Today, 1X shares progress on the 1X World Model (1XWM): a bridge between the world of atoms and the world of bits. The 1X World Model enables NEO to anticipate the outcomes of robot actions and their consequences on the world.

This approach significantly accelerates experimentation, allowing evaluation of robotic policies' reliability and effectiveness in a fraction of the time, without requiring intensive environment-specific engineering within traditional physics-based simulators or mock real-world sets.

Action-Controllable World Model

The world model is trained on sequences of video frames, robot observations, and input action trajectories. Inputs are encoded to latent representations, and the model predicts the latent encodings of future frames. It also predicts the state value of the final frame to evaluate task success and completion.

Most video generation models are text-to-video (T2V), using a language prompt to generate video. However, world models for simulating robots need to be action-controllable — steered by exact robot trajectories rather than loose directives like "Grab the mug" or "Wipe the countertop." 1X demonstrates the action-controllability of 1XWM by providing it with a few initial frames of real footage along with multiple subsequent action trajectories. From this anchor point, 1XWM simulates the consequences of those exact actions, including the physics of objects like doors being opened and cloths being wiped across a countertop.

Scaling Data: Does 1XWM Improve?

To study whether 1XWM improves with data scale, it is trained to predict not only future states and images but also whether the task attempt succeeded or failed. Clear improvements are observed across the board as the number of tasks and diversity of robot behaviors scale up, tested on Airfryer, Arcade, and Shelf tasks.

Task-Specific Data Matters

When confronted with an unfamiliar task and environment, the world model often struggles to model object interactions exactly without knowledge of their specific properties. Training on task-specific data allows the model to update based on the subtle dynamics of the task at hand.

For example, when trained on smaller amounts of data, 1XWM hallucinates the air fryer tray and body as a single unit, pulling the entire unit off the counter. After adding interaction data with the air fryer, 1XWM gains a better understanding of how the tray separates, and even learns to model subtle interactions such as the confinement of the tray movement within the base.

Positive Transfer Between Tasks

Training on both shelf and arcade data improves accuracy compared to training on shelf alone. This positive transfer of accuracy and task understanding from one task to another reinforces belief in 1XWM's scaling capability. The more task-oriented robotics data accumulated, the more accurately future outcomes can be predicted.

Evaluation Engine: Forecasting Policy Performance

An aligned world model can solve the evaluation problem by forecasting the actions of candidate robot policies. Given 1XWM generations from each policy on datasets of initial states, their respective performances can be compared. Importantly, datasets of production-setting states can be curated, and counterfactual results generated from states that an autonomous policy has previously failed in.

For every set of model checkpoint weights, 1XWM predicts future states and success likelihood that are distributionally aligned to actual real-world futures. This provides insight into model performance at scale and enables architecture and checkpoint selection decisions with an instant feedback loop.

Correlation Between Predicted and Real Performance

In ablation studies, 1X examined the decision to include proprioception (robot joint states) as input to the robot policy. Real-world evaluation was then run on the most and least promising checkpoints according to 1XWM. There is indeed a correlation between predicted success rates and true task scores.

Given a true real-world success rate gap of 15% between two policies, a World Model with 70% accuracy can accurately predict the better policy with 90% success.

Given a consistent predicted performance gap across checkpoints, and the ability to evaluate policies on identical starting states, confidence in such verdicts is even higher. In another experiment, two different image encoders for a policy were compared — the predicted better ViT-L model indeed performed better in the real world.

Limitations and Future Work

1XWM currently struggles to model interactions with held-out objects not seen in training data. As robots are deployed in homes, evaluation needs to move away from task-specific testing towards production-level evaluation capable of handling a wider, more ambiguous array of full-body manipulation tasks and objects. Improving the generalization capability and accuracy of 1XWM will be the first step towards this goal.

Implications: Synthetic Data and Evaluation

The implications of accurate hallucinated rollouts extend far beyond rigorous evaluation for humanoid robots. When data generated by 1XWM — the joint distribution over all sensor readings and actions observed by the robot — becomes indistinguishable from real data, a transformative milestone is reached. This moment has already happened for LLMs, and 1X believes it will soon be true for synthetic robotics data.

Data and evaluations are the cornerstones of solving autonomy, and 1XWM provides a unified path for tackling both challenges.

---

Source: 1X Tech — 1X World Model