Skip to content
RobotWorld
← Tags

#Block Causal Attention (2)

WaterfallPagination
LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics

LeVJEPA: Efficient & Scalable Video Pretraining without the Heuristics

LeVJEPA performs video self-supervised pretraining with a single encoder, a single loss and one fixed hyperparameter (λ=0.02): an invariance loss plus SIGReg regularization provably rule out representation collapse, with no target encoder, predictor, stop-gradient or pixel reconstruction. It uses 5.6–20.8× less training compute than V-JEPA 2, leads by 7.6 points on ImageNet-1K under a FLOP-matched budget, and gets block-causal attention for free — paving the way to streaming perception and autoregressive world models.

Lukas Kuhn, Lucas Maes, Giuseppe SerraAug 27, 2026
JEPASIGRegvideo-SSLAug 27, 2026
Patch Policy: Efficient Embodied Control via Dense Visual Representations

Patch Policy: Efficient Embodied Control via Dense Visual Representations

Pretrained dense visual features from Vision Transformers (ViTs) are powerful yet have been underutilized in robot learning. Modern robot policies either compress each observation into a single global token, or rely on visual backbones trained from scratch, sacrificing both fine-grained spatial detail and the benefits of large-scale visual pre-training. While there exist policies that do operate on dense patch features like large vision-language-action models (VLAs), they tend to be heavy and slow, inheriting the full cost of a billion-parameter vision-language model (VLM) backbone. We close this gap with Patch Policy, a minimal architectural extension that enables transformer-based policies to consume dense pre-trained patch tokens directly without the computational overhead of a full VLM. At its core is a block-causal attention mask that preserves the temporal causality of standard policies while letting the model attend over many patch tokens per observation, alongside other state information. Patch Policy is lightweight, fast, and highly effective. Across four simulated and three real-world environment suites, our method achieves a 40% relative improvement over policies using state-of-the-art global-pooled representations. Furthermore, it surpasses fine-tuned OpenVLA-OFT by 18% while using roughly 0.7% of the parameters. We believe Patch Policy provides a pipeline for the robotics community to readily leverage continuing progress in visual representation learning, without sacrificing the training efficiency or inference speed required for high-frequency, reactive control. Videos can be viewed at https://patch-policy.github.io

Gaoyue Zhou, Zichen Jeff Cui, Ada LangfordJul 20, 2026
Patch PolicyDense Visual RepresentationDINOv2Jul 20, 2026