π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities

Paper: π0.7: a Steerable Generalist Robotic Foundation Model with Emergent Capabilities
Authors: Physical Intelligence (90 authors, including Sergey Levine, Chelsea Finn, Karol Hausman, Danny Driess, Karl Pertsch, Lucy Xiaoyang Shi)
Links: arXiv:2604.15483 (v2, 2026-04-24) · Official blog
Code status: π0.7 code and weights are not open-sourced; the official openpi repository currently covers only the earlier π0 / π0.5 family

One-Line Summary

Physical Intelligence expands the context seen by a robot policy from a single "what to do" sentence into a multimodal task brief: subtask language instructions, subgoal images synthesized by a world model, and episode metadata describing how fast, how well, and how mistake-free each trajectory was executed. This "diverse prompting" recipe lets π0.7 absorb failure episodes, autonomous evaluation rollouts, and even human video into one training mixture, yielding a steerable generalist that matches RL-tuned specialists without any task fine-tuning, folds laundry zero-shot on a robot that never folded anything, and learns brand-new appliances such as an air fryer through live verbal coaching.

Background and Motivation

Large language models are powerful not merely because they memorize their training corpus, but because they can recombine capabilities from that corpus in new ways to solve problems they have never seen. This compositional generalization is arguably the hallmark of generalist intelligence. In the physical world, however, vision-language-action (VLA) models have grown larger and more capable while compositional generalization has remained elusive: most models still need task-specific fine-tuning to fluently execute instructions they were explicitly trained on, let alone recombine skills into new tasks.

A natural way out is to train on larger and more diverse data. But diversity in robot data creates a distinctive problem: trajectories from different operators, different strategies, and different quality levels get mixed together, and naive training makes the model average over behavioral modes, often making things worse. Failed episodes, clumsy autonomous exploration, and demonstrations of uneven quality actually contain valuable information about state distributions and error recovery — discarding them wastes data, while dumping them in naively drags performance down.

π0.7 borrows the "prompt expansion" idea from image generation: instead of cleaning the data, attach a sufficiently detailed context to every example so the model learns "what to do under which conditions." In robotics, however, richer text alone is not enough — some decisive details are hard to verbalize (what a cleanly folded T-shirt looks like), and some are policy-level attributes (the overall quality of an episode). So π0.7 expands the context into a multimodal prompt: language instructions plus subgoal images, episode metadata, and control mode. This seemingly modest change fundamentally alters what kind of data the model can digest, and ultimately gives rise to a set of emergent capabilities.

Figure 1: π0.7 is a steerable generalist robot foundation model. Its training prompts contain not only task descriptions but detailed language, generated subgoal images, and episode metadata — telling the model not just what to do but how to do it.

Where π0.7 Sits in the VLA Landscape

Generalist manipulation policies are usually initialized from pretrained vision-language models or video-generation models, then extended with components such as memory, hierarchical planning, and goal-image conditioning. π0.7 unifies all three inside one model: it inherits the MEM-style video history encoder for memory, receives subtask instructions from a learned high-level policy for hierarchy, and conditions on goal images — building on the π0.6-MEM architecture. The authors are explicit about the nature of the contribution: they do not propose a new architecture or model design so much as a methodology for letting VLAs exploit far more diverse data sources, plus an empirical analysis showing this yields strong signs of compositional generalization.

Conditioning manipulation policies on goal images has a long history — user-provided goal images, generated goals from separate models, or chain-of-thought visual planning. π0.7's use of world-model subgoals is complementary rather than novel in itself; what is new is the scale of the empirical claims: zero-shot transfer of dexterous skills like laundry folding to a completely different robot, and generalization to novel object interactions such as operating an air fryer, going significantly beyond the quantitative gains reported by prior goal-conditioning work. On data, prior work has separately shown benefits from web data, egocentric human video, or autonomous robot experience; π0.7 incorporates all of these simultaneously and finds that diverse data combined with detailed prompting produces compositional generalization and strong out-of-the-box behavior.

Method Overview: The Prompt as a Multimodal Task Brief

The standard VLA training objective maximizes the log-likelihood of an action chunk given the observation history and context:

$$\max_{\theta}\;\mathbb{E}_{\mathcal{D}}\left[\log\pi_{\theta}(\mathbf{a}_{t:t+H}\mid\mathbf{o}_{t-T:t},\mathcal{C}_{t})\right]$$

The observation $\mathbf{o}_{t}=[\mathbf{I}_{t}^{1},\ldots,\mathbf{I}_{t}^{n},\mathbf{q}_{t}]$ consists of $n$ camera images and the proprioceptive state $\mathbf{q}_{t}$, while $\mathbf{a}_{t:t+H}$ is a short chunk of future actions. Conventionally the context $\mathcal{C}_{t}$ is a single human-annotated language instruction $\ell_{t}$ (e.g., "clean the kitchen"). π0.7 expands it into five components: $\mathcal{C}_{t}=(\ell_{t},\hat{\ell}_{t},\mathbf{g}_{t},m,c)$ — task description, subtask instruction, multi-view subgoal images, episode metadata, and control mode. Each component is randomly dropped during training, so any subset can be used at test time.

Figure 3: Prompt overview. The context of π0.7 mixes subtask instructions, subgoal images, and episode metadata; every component is trained with dropout and can be combined freely at test time.

The Four Prompt Extensions

Subtask instructions: semantic segmentation of behavior

Following π0.5, the prompt includes — beyond the overall task description $\ell_{t}$ — an intermediate instruction $\hat{\ell}_{t}$ describing the current semantic subtask (e.g., "open the fridge door"). Data is collected across diverse scenarios, segmented, and annotated with detailed text. This design yields a bonus: because the model is trained to follow diverse language, a human can coach the robot step by step through entirely new tasks (loading a sweet potato into an air fryer it has never seen); the coaching transcripts can then fine-tune a high-level semantic policy that completes the task fully autonomously.

Subgoal images: drawing what language cannot say

"Open the fridge door" does not specify how to grasp the handle. Subgoal images depict the desired near-future state of the scene, conveying execution details that language struggles to express — environment/object-level outcomes (best seen in the base view) and arm/gripper outcomes (best seen in wrist views). π0.7 uses multi-view subgoals $\mathbf{g}_{t}=[G_{t}^{1},\ldots,G_{t}^{n}]$ generated at runtime by a lightweight world model $g_{\psi}$, trained with a conditional flow-matching objective:

$$\max_{\psi}\;\mathbb{E}_{\mathcal{D}_{g}}\left[\mathcal{L}_{\mathrm{CFM}}\left(\mathbf{g}_{t}^{\star},\;g_{\psi}(\mathbf{o}_{t},\hat{\ell}_{t},m)\right)\right]$$

The ground-truth subgoal is the segment end frame $\mathbf{g}_{t}^{\star}=\mathbf{o}_{t_{\mathrm{end}}}$, and $\mathcal{D}_{g}$ is the subset of segments with especially high-quality subtask labels. The world model is initialized from BAGEL, a 14B mixture-of-transformers image understanding/editing/generation model with web-scale pretraining; by mixing web data, egocentric human video, and other video sources, the semantic and physical concepts it acquires are "injected" into π0.7 through the subgoal images — a neat channel for transferring web-scale knowledge into a low-level policy.

When training the VLA, subgoal images are supplied as a mixture of real future frames and generated images: with probability 0.25 the end-of-segment frame (consistent with the world model's prediction target), with probability 0.75 a future frame sampled uniformly 0–4 seconds ahead, plus a large amount of world-model-generated images to bridge the real/generated train-test gap. Because subgoal conditioning turns action prediction into an easier "inverse dynamics" problem and speeds up training, subgoals are added to only 25% of each batch.

Episode metadata: explicit labels for data quality

This is what lets π0.7 digest low-quality data. Every training trajectory is annotated with a metadata set $m$:

• Overall speed: episode length in timesteps, discretized in bins of 500 steps (e.g., 1750–2250 becomes "2000 steps"). Speed often correlates with quality — fast episodes tend to have fewer mistakes.
• Overall quality: an execution quality score from 1 to 5, with 5 the best.
• Mistake: whether the robot made a mistake within the action segment (e.g., a missed grasp or a wrong subtask), coarsely annotated by humans.

During training the model sees ground-truth speed, quality, and mistake labels; data diversity supplies enough signal for it to correlate metadata with target actions. At test time the usage is inverted: speed is set to the 15th percentile of episode lengths for the task, quality is always 5, and mistake is always false — effectively commanding the model to "execute near-maximally fast, at top quality, with zero mistakes." This is the direct source of steerability.

Control mode: one model, joint or end-effector actions

Training includes both joint-level and end-effector actions, declared in the prompt with a text identifier $c\in\{\texttt{joint},\texttt{ee}\}$ and chosen per task at runtime. The appendix shows that for prior models the two control modes yielded little difference on cross-embodiment tasks.

The dropout recipe at training time

The per-component dropout is carefully designed: 25% of examples carry subgoal images; among those, the subtask instruction is dropped 30% of the time (a visual subgoal is often richer than, and can substitute for, the text); episode metadata is dropped entirely 15% of the time, with each of its three sub-fields additionally dropped 5% individually; the control mode is never dropped. This dropout lets π0.7 accept any prompt combination at test time and even supports classifier-free guidance (CFG) over the context.

Architecture and Training Recipe

Figure 2: Architecture overview. π0.7 is a 5B-parameter VLA: a 4B VLM backbone, a MEM-style video history encoder, and an 860M action expert. At runtime, subtask instructions come from a high-level semantic policy of the same architecture, and subgoal images come from a lightweight BAGEL-based world model.

π0.7 extends the π0.6 VLA architecture and the MEM memory system, totaling about 5B parameters:

• VLM backbone: initialized from Gemma3 4B (including a 400M vision encoder). It takes up to four camera images (front, two wrists, optional rear), each with up to six history frames sampled at 1-second stride, resized to 448×448. The history encoder follows the MEM design — spatiotemporal compression with a fixed token count regardless of history length. Entire history is dropped with probability 0.3, as is the rear view when present.
• Attention structure: a block-causal mask — observation tokens and subgoal tokens attend bidirectionally within themselves, subgoal tokens additionally attend to observations, and subsequent text tokens use causal attention.
• Proprioception: the state $\mathbf{q}_{t}$ (including history states) is embedded via a linear projection into the backbone dimension — unlike π0.6, which discretized states into text tokens. Each history state is an individual token, masked out when its frame is dropped.
• Action expert: an 860M-parameter transformer predicting continuous actions with a flow-matching objective; timestep information is injected via adaptive RMSNorm. It processes a fixed 50 action tokens (a 50-step chunk), which attend bidirectionally to each other and to the backbone activations.
• Knowledge insulation (KI): the VLM backbone is additionally supervised with FAST tokens under a discrete cross-entropy loss; the action expert reads all backbone activations but its gradients do not flow into the backbone, so the VLM keeps learning through a stable discrete objective.
• Real-time chunking (RTC): training-time RTC simulates inference delays of 0–12 timesteps (up to 240 ms on a 50 Hz robot) to keep action trajectories smooth under latency.

Figure 4: Some of the robot platforms used in the experiments, including bimanual mobile manipulators, static bimanual systems, and UR5e arms.

The training data is where this work departs most from the classic VLA pipeline. Beyond demonstrations spanning many platforms and environments (in-house lab-like and home-like settings, plus real homes), π0.7 makes heavy use of suboptimal data: failure episodes, success episodes riddled with mistakes, and autonomous rollouts produced by prior models during evaluation — including the RL-post-trained $\pi^{*}_{0.6}$. The latter amounts to "distillation": the generalist inherits the skills of RL specialists back into a single pretrained model. The mixture also includes open-source robot datasets, egocentric human video, and web-sourced non-robot auxiliary tasks (object localization and attribute prediction, visual question answering, text-only prediction, video captioning). All autonomous data from generalization-focused evaluations is explicitly excluded from training to avoid test-set leakage.

The world model's training recipe largely follows BAGEL: each example pairs a subtask instruction and three camera inputs with three target images taken at the segment end. Camera inputs pass through both a ViT (semantic understanding, 448×336, patch size 14) and a VAE (fine-grained detail, 512×384, patch size 16); ViT tokens are processed by a 7B LLM backbone while VAE tokens go through a 7B generation backbone. Label quality — especially temporal segmentation of the subtask language — turns out to matter greatly for subgoal quality, so the robot-data subset is filtered accordingly, and open-source image-editing and video datasets are mixed in to preserve the model's semantic knowledge. At test time subgoals regenerate every $\Delta=4$ seconds, matching SuSIE.

Runtime: Asynchronous Inference and Classifier-Free Guidance

At deployment π0.7 combines context freely per task, with no task-specific post-training. Episode metadata is fixed: speed at the 15th percentile of the task's episode lengths, quality 5, mistake false. The subtask instruction comes from a high-level semantic policy (fine-tuned from the same architecture) or from a human coach. When subgoal images are used, they refresh whenever the semantic intent changes (a new $\hat{\ell}_{t}$) or after $\Delta=4$ seconds, whichever comes first. The whole pipeline runs asynchronously: visual subgoals and subtask instructions are generated in separate threads, and the VLA always consumes the latest available results (Algorithm 1 of the paper). Actions are generated with 5 denoising steps over a 50-step chunk, of which $\hat{H}\in\{15,25\}$ steps are executed.

Because each context component is dropped during training, π0.7 supports classifier-free guidance on any of them. Each denoising step follows:

$$\nabla_{\mathbf{a}}\log\pi_{\theta}(\mathbf{a}_{t:t+H}|\mathbf{o}_{t},\mathcal{C}_{t})+\beta\left(\nabla_{\mathbf{a}}\log\pi_{\theta}(\mathbf{a}_{t:t+H}|\mathbf{o}_{t},\mathcal{C}_{t})-\nabla_{\mathbf{a}}\log\pi_{\theta}(\mathbf{a}_{t:t+H}|\mathbf{o}_{t},\mathcal{C}_{t}^{\text{uncond}})\right)$$

where $\mathcal{C}_{t}^{\text{uncond}}$ is the context with the guided component removed and $\beta$ is the guidance weight. The paper applies CFG to the episode metadata to elicit peak dexterity, using moderate values $\beta\in\{1.3,1.7,2.2\}$.

The runtime workflow of Algorithm 1, drawn out: three information streams (policy, high-level semantic policy, world model) cooperate asynchronously:

flowchart TB
    A["task instruction l
+ episode metadata m
+ control mode c"] --> B["high-level semantic policy
same Gemma3-4B architecture"]
    B --> C["subtask instruction
l-hat refreshes on intent change"]
    C --> D["world model g-psi
BAGEL 14B, 4xH100 TP"]
    A --> D
    D --> E["subgoal images g-star
3 views, 25-step denoise, 1.25s"]
    C --> F["pi-0.7 VLA 5B
4B VLM + 860M action expert"]
    E --> F
    G["observations o
4 cameras, 6 history frames"] --> F
    F --> H["5-step flow matching denoise
50-step action chunk"]
    H --> I["execute H-hat = 15 or 25 steps
50Hz joint or ee control"]
    I -.next subtask.-> B
    E -.regenerate every 4s
or on new subtask.-> D

On inference efficiency, π0.7 and the high-level policy share a single H100: after post-RTC optimizations the minimal configuration (3 cameras, 5 denoising steps) runs in 38 ms, rising to 127 ms worst case with the MEM vision encoder and subgoal images enabled. The 14B world model is the real bottleneck — iterative denoising over a nearly 10,000-token sequence is expensive, so the team applies 4-way tensor parallelism across 4×H100, 8-bit quantized matmuls, and a modified SageAttention, bringing 25 denoising steps (with both text and image CFG) down to 1.25 seconds while the policy keeps executing asynchronously.

Experimental Results

Figure 5: Selected evaluation tasks; two of the longer-horizon ones are visualized as full film strips.

1. Out-of-the-box dexterity: a generalist matching RL specialists

The team evaluates over a dozen highly dexterous tasks: the espresso-making, box-building, and laundry-folding tasks from the $\pi^{*}_{0.6}$ release (directly comparable against RL specialists), the "Robot Olympics" tasks (making a peanut-butter sandwich, turning a shirt inside-out, driving through a door), plus slicing a whole zucchini, peeling vegetables (zucchini, cucumber, carrot), and the long-horizon trash-bag replacement. The single generalist π0.7 matches the corresponding task specialists out of the box on every task, and even exceeds RL-specialist throughput on the two hardest ones — laundry folding and box building. Against SFT specialists built on π0.6, π0.7 is likewise competitive across the board. On memory-heavy long-horizon tasks, out-of-the-box π0.7 matches or beats the fine-tuned π0.6-MEM specialists.

Figure 6: Out-of-the-box dexterity. π0.7 performs a wide range of highly dexterous tasks without any task fine-tuning, matching or beating RL/SFT specialists.

Ablations reveal where the performance comes from: removing all autonomous evaluation data (no eval data) or removing episode metadata (no metadata) significantly degrades every task. Evaluation data varies wildly in quality; only with metadata to disambiguate high- and low-quality behavior can expert trajectories be distilled into the generalist. The synergy between diverse data and detailed context is the central empirical claim of the paper.

2. Instruction following: understanding complex commands in unseen homes

In 4 kitchens and 2 bedrooms absent from training data, the team runs 14 scenarios of 3–6 step instructions: tidying items, interacting with furniture, cleaning spills. π0.7's overall instruction-following success rate significantly exceeds π0.5 and π0.6. On deliberately complex referential instructions ("pick up an object I would use to eat soup", "pick up the fruit on the largest plate"), π0.7 again leads, and improves further when conditioned on world-model subgoal images (denoted π0.7 (GC)) — web-scale semantic understanding flows into the policy through the subgoals.

Figure 9: Broad instruction following in unseen environments — 14 scenarios with multi-step instruction sequences; π0.7 significantly outperforms prior models.

Even more interesting are the "against dataset bias" experiments: if the robot always did the same thing in a given scene, a trained model tends to ignore language and blindly copy the data. The team constructs two reversed tasks — "Reverse Bussing" (trash into the bussing bin, dishes into the trash — the exact opposite of training data) and "Reverse Fridge to Microwave" (training only ever moved food fridge→microwave; testing demands the reverse). π0.7 substantially outperforms prior models on both, showing it genuinely attends to the instruction; on the reverse fridge task the generated subgoal images prove critical.

3. Cross-embodiment transfer: laundry folding on an arm that never folded anything

The team scales the embodiment gap step by step: Table Setting (data from several robot types — all models transfer well) → Bag in Backpack / Organize Tupperware (all data from the larger UR5e bimanual platform, tested on a smaller static bimanual one — π0.5 collapses while π0.6/π0.7 remain strong) → Shirt Bagging (data from the small static bimanual, tested on a single-arm UR5e — the target robot has one heavier, larger arm and must adopt an entirely different strategy; here π0.7 significantly outperforms all prior models).

Figure 12: Cross-embodiment transfer. As the embodiment gap grows, π0.7's advantage widens; for dexterous tasks like shirt folding, subgoal images further boost performance.

The hardest case is dexterous transfer: all folding data was collected on lightweight static bimanual robots, with zero folding data on the UR5e bimanual system, whose workspace, inertia, and teleoperability differ markedly. Yet π0.7 successfully folds towels and T-shirts on the UR5e — and spontaneously discovers strategies adapted to the target embodiment. On the source robot, human operators tilt the end-effector to pin fabric against the table before lifting; on the UR5e the model switches to vertical grasps better suited to that arm's kinematics. Generated subgoal images further improve transfer by letting the world model "imagine" plausible grasps and fabric configurations for the target robot.

Figure 13: Cross-embodiment transfer produces emergent strategies adapted to the target embodiment. (a) On the source robot two arms cooperate to hold the bag open; the UR5e switches to a single-arm pick-and-place. (b) The source robot uses tilted grasps; the UR5e uses vertical grasps.

Why does this matter practically? Collecting human demonstrations on high-payload industrial arms is substantially more expensive and harder than on lightweight platforms. If dexterous skills can be taught on cheap, easy-to-teleoperate robots and then transferred zero-shot to industrial arms, the economics of data collection change considerably — and the emergent-strategy behavior means the transfer is not a brittle motion replay.

To quantify this, the paper runs a human subject study with 10 experienced teleoperators (mean 375 hours of experience, all in the top 2% of the operator fleet) who — like the policy — had never attempted shirt folding on the UR5e bimanual system:

Condition	Task progress	Success rate	Setting
Human experts (10 operators)	90.9%	80.6%	zero-shot, first time on the UR5e
π0.7 (GC, with generated subgoals)	85.6%	80.0%	zero-shot, no UR5e folding data whatsoever

Table 1: Zero-shot cross-embodiment shirt folding — the policy matches expert human teleoperators (Sec. IX-C and Appendix A-F).

The paper also runs a data-bucket ablation on the "Laundry (T-shirts and Shorts)" task: human-collected data is split into four buckets by fold quality and speed, training with/without episode metadata (8 models total). The findings:

Data bucket	Composition	π0.7 (with metadata)	π0.7 (without metadata)
Bucket 1	top 30% by quality+speed	baseline	baseline
Bucket 2	top 50%	keeps improving	minor fluctuation
Bucket 3	top 80%	keeps improving	starts degrading
Bucket 4	all data (lowest avg. quality)	still the best	clearly worse

Table 2: Mixed-quality data ablation. With metadata the model keeps benefiting from larger-but-dirtier data; without metadata it degrades as low-quality data grows (Figure 18, left).

Figure 18: Scaling generalization with diverse context and data. Left: with metadata, more (even lower-quality) data keeps improving performance; without metadata it degrades. Right: removing the most task-diverse 20% of data substantially hurts generalization.

A second controlled comparison removes either "the most task-diverse 20%" or "a random 20%" of the data (equal data volume). On several unseen short-horizon tasks, removing the most diverse slice causes significant performance loss while random removal does not — π0.7 genuinely converts task diversity into compositional generalization.

Coaching: Learning New Long-Horizon Tasks with Words Alone

For complex unseen tasks — cooking a sweet potato in an air fryer, unloading the air fryer, toasting a bagel — no training episodes exist, though similar appliances appear in human data and external datasets. A person can simply walk the robot through the task with step-by-step instructions ("pick up the sweet potato", "open the air fryer"). π0.7 follows such coaching far more effectively than prior models, and even better with generated subgoal images. Crucially, none of the models have any action-level data for these tasks; both environment and task are entirely unseen in the coaching episodes.

Figure 14: Language coaching in action. Step-by-step verbal instructions "teach" π0.7 a new task; the same instructions can later train a high-level policy that runs the task fully autonomously.

Coaching data can then endow the model with new capabilities: the step-by-step transcripts train a high-level language policy that prompts π0.7 as it works, yielding fully autonomous execution of the five tested long-horizon tasks at roughly the same performance as the coached episodes — no additional teleoperation data required.

Limitations

• Zero-shot success still lags in-distribution: seen tasks often exceed 90% success, while unseen tasks or unseen task–robot combinations sit in the 60–80% range — still a gap from reliable deployment.
• "Seen vs. unseen" cannot be strictly defined: the training set is so large and diverse that tracing a behavior to specific episodes is impractical; a "new" task may well be a novel recombination of skills seen elsewhere. The authors acknowledge this mirrors the difficulty of understanding LLM generalization — and argue such "remixing" is precisely what compositional generalization is.
• Closed evaluation: all experiments run on Physical Intelligence's own robot fleet, internal data, and protocols; no third-party reproduction exists. Quality and mistake labels rely on coarse human annotation, whose cost and consistency the paper does not detail.
• Inference cost: the 5B policy needs one H100, and the 14B world model needs 4×H100 tensor parallelism to hit 1.25-second subgoal generation — a nontrivial hardware floor for scaled deployment.

Conclusion and Outlook

The core contribution of π0.7 is not a new architecture but a methodology: with a rich enough context, "uneven data quality" stops being a training hazard and becomes a conditional generation problem. Metadata, subgoal images, and detailed language together tell the model under what conditions and in what manner each trajectory was executed — turning failures and suboptimal rollouts from poison into nourishment, and distilling RL specialists back into a single generalist. The payoff is a set of emergent capabilities: out-of-the-box parity with task specialists, zero-shot cross-embodiment transfer of dexterous skills, instruction following that overrides dataset bias, and language coaching plus high-level policy bootstrapping for brand-new long-horizon tasks.

The authors point to next steps where π0.7 exploits its own steerability to learn efficiently on the test task — finer-grained verbal coaching, or even autonomous reinforcement learning — and further, to let the model "think through" possible ways of executing a task, ground those thoughts in actions via diverse prompts, and reflect on outcomes to revise the plan. When prompting and following become strong enough, acquiring robot capabilities shifts from "collect action data for every new task" to "prompt it, coach it, explain to it."

A final implementation note worth highlighting: the paper's appendix reports that π0.7's attention patterns show the action expert selectively attends to the most relevant context modalities depending on the task phase, and that the entire runtime stack — high-level policy, VLA, and world model — was engineered to run asynchronously so that no component blocks the 50 Hz control loop. The result is a system where "prompting a robot" starts to resemble prompting a language model: specify the task, optionally the manner, and let the foundation model figure out the movements.

Quotes

"Models that can enable compositional generalization at scale would transform how we approach robotic learning, making it possible to prompt, coach, and explain things to a robot rather than needing to collect additional action data."

"In robotics, simply captioning the data with more detailed text is not enough — the details that determine task success and proficiency might be more subtle (e.g., information about the overall quality of the episode), or might simply be hard to express with language alone (e.g., the particular appearance of a cleanly folded t-shirt)."