Human-JEPA: A Human-Centric Vision Model that Perceives and Anticipates

Paper Information

Field	Value
Title	Human-JEPA: A Human-Centric Vision Model that Perceives and Anticipates
Authors	Hui Wei, Licai Sun, Guoying Zhao
Affiliation	ELLIS Institute Finland, University of Oulu, Finland
arXiv	2608.21160 (August 2026)
Code	No public code repository provided

One-Sentence Summary

Human-JEPA specializes a general-purpose video JEPA for human-centric perception and anticipation through anchored forecasting, surpassing pixel-anchored image specialists at 2.7× fewer parameters while its predictor head is the first that does not degrade anticipation.

Background and Motivation

The human visual system perceives and anticipates seamlessly. Watching someone reach across a table, we simultaneously register the body, the pose, who the person is, and predict where the hand will land before it arrives. Machine vision for humans requires both halves: perception of the present and anticipation of the future, and neither can substitute for the other. Decisions are made before an action completes, so anticipation matters just as much as perception.

The strongest existing human-centric vision models come from pixel-anchored image encoders pretrained on human corpora. Sapiens2 combines masked reconstruction with a self-distilled contrastive objective at billion-parameter scale, achieving state-of-the-art parsing and pose at 4K resolution. HAP shows that body-part priors improve masked person modeling. Hulk and UniHCP unify multiple human vision tasks behind shared heads. These models are the strongest possible readers of a photograph, yet they represent no motion and predict no future—the second half of understanding humans is unreachable.

Meanwhile, video self-supervised learning has produced the V-JEPA family. V-JEPA 2.1 is pretrained on general internet video, learning latent-space video prediction. But it is a general model, never specialized for human data. The question arises: can one continue-pretrain a released video JEPA to make it human-centric without destroying its existing capabilities? This seems straightforward—just continue training on human video—but multiple traps are hidden in the execution.

The challenge is more severe than expected. Naive continued pretraining causes a "silent collapse"—dense perception ability quietly disappears during training while the training loss decreases throughout, with no anomaly signal to warn. Standard multiblock masks spend the adaptation budget on a temporally local copy task rather than motion modeling, costing five points on action recognition. Person-level masking strategies (deleting whole people, inputting skeleton priors) are not just unhelpful but harmful—all nine person-level objectives are falsified by a causal probe. These findings constitute the starting point for Human-JEPA's design.

Human-JEPA's design is therefore deliberate: keep the architecture and initialization unchanged (because the initial abilities are the asset), and make two critical changes at the masking and anchoring level—an anchored context stream pins dense perception to a frozen copy of the initialization, and a forecasting mask family replaces block inpainting with a pure past-to-future split. An image co-training branch continually re-grounds the appearance features that video-only prediction lets fade. The entire training process requires no human labels or pose annotations, being fully self-supervised.

Preliminaries

JEPA (Joint Embedding Predictive Architecture) learns representations by predicting target-region embeddings in latent space, avoiding both the augmentation-dependent positive pairs of contrastive learning and the low-level biases of pixel/token reconstruction. I-JEPA first demonstrated this paradigm for images; the V-JEPA family extended it to video. In V-JEPA 2.1, a student encoder $f_{\theta}$ sees only unmasked context tokens, a predictor $g_{\phi}$ predicts the latent representation of masked tokens supervised by an EMA teacher $f_{\bar{\theta}}$, and a context loss additionally supervises predictions at visible positions with multi-layer targets. The EMA teacher updates follow $\bar{\theta} \leftarrow \alpha \bar{\theta} + (1-\alpha) \theta$, where $\alpha$ is the momentum coefficient.

Human-centric vision models have evolved from task-specific designs toward general-purpose representations. UniHCP and PATH jointly train a single ViT across multiple human-centric datasets with task-specific queries; Hulk extends this with a multimodal generalist framework formulating heterogeneous 2D, 3D, skeleton-based, and vision-language tasks as translations between output spaces. Sapiens/Sapiens2 build high-fidelity backbones via large-scale self-supervised pretraining on hundreds of millions of human images. DINOv3 improves dense representations through a Gram-anchoring objective. All operate on single images or labeled multi-task data; none represents how people move or interact over time.

The three paradigms of self-supervised learning—contrastive, distillation, and masked data modeling—each have trade-offs. Contrastive methods (SimCLR, MoCo) depend on large-batch negatives; distillation methods (BYOL, SimSiam, DINO) avoid collapse through stop-gradient and momentum encoders; masked modeling (MAE, data2vec) learns by reconstructing occluded portions. JEPA sits at the intersection of distillation and masked modeling: it predicts in latent rather than pixel space, avoiding pixel reconstruction's low-level biases while not requiring negative pairs from contrastive learning.

Method

3.1 Anchored Continued Pretraining

Human-JEPA starts from a released video JEPA—encoder $f$ and predictor $g$ pretrained on general internet video. The goal is to make it human-centric through continued pretraining without changing architecture or size, so adopting it is a checkpoint swap for an existing user. The input is a curated corpus of human video and person crops; the output is a frozen transferable encoder with a predictor trained in that encoder's own representation space. The objective is self-supervised throughout, with no human labels and no pose annotations at any stage.

The V-JEPA 2.1 training objective combines a masked prediction term and a context (dense) loss:

$$\mathcal{L} = \underbrace{\big\lVert\hat{z}_{\mathrm{msk}} - \mathrm{sg}\,[f_{\bar{\theta}}(x)]_{\mathrm{msk}}\big\rVert_1}_{\text{masked prediction}} + \lambda\,\underbrace{\big\lVert\hat{z}_{\mathrm{ctx}} - \mathrm{sg}\,[\,A(x)\,]_{\mathrm{ctx}}\big\rVert_1}_{\text{context (dense) loss}}$$

where $\hat{z} = g_{\phi}(f_{\theta}(x_{\mathrm{ctx}}))$, $\mathrm{sg}$ is stop-gradient, $A$ is the context-target network, and $\lambda$ balances the two loss terms. Standard training uses $A = f_{\bar{\theta}}$, letting dense targets drift with the student. Human-JEPA sets $A = f_0$, a frozen copy of the initialization, so the dense stream is distilled from the one network whose dense quality we want to keep, removing the target drift that causes collapse.

The key insight is that when $A = f_{\bar{\theta}}$, the teacher drifts with the student, and the semantic content of the dense target changes continuously. This means the dense stream loses a stable reference point—the student can gradually drift toward a "easier to predict but less informative" representation space, and the teacher follows into the same space, causing the loss to decrease while representation quality degrades. By fixing $A = f_0$, the dense stream is always anchored to the initial high-quality representation, and the student cannot "cheat" through drift.

An image branch trains the same objective on LUPerson-T person crops rendered as one-frame clips, carrying full-body appearance at a scale video cannot provide. This branch turns out to be where the identity ability originates—a simple yet critical addition that solves the appearance-feature fading problem. In ablation, the image branch alone restores pose AP from the collapsed 0.110 to 0.626, matching the anchored arm's gain.

3.2 Forecasting Masks and Mixing the Families

Standard multiblock masks repeat a spatial block across every frame. The visible context then always contains the same spatial surround at other instants, making the shortest path to a correct prediction a copy of appearance across space and time. Under continued pretraining this is destructive—the adaptation budget is spent strengthening a copy task rather than a model of motion.

Human-JEPA replaces blocks with a pure past-to-future split: the context is the first half of the tubelets and the target is the second half:

$$\mathcal{L} = \big\lVert\,g_{\phi}\big(f_{\theta}(x_{t<T/2})\big) - \mathrm{sg}\,[f_{\bar{\theta}}(x)]_{t \geq T/2}\,\big\rVert_1 + \lambda\,\big\lVert\hat{z}_{\mathrm{ctx}} - \mathrm{sg}\,[f_0(x)]_{\mathrm{ctx}}\big\rVert_1$$

The first term now demands a continuation rather than an interpolation, since nothing about the future is visible and the only route to the target is a model of how the scene evolves. The second term is unchanged from Eq. 1 and keeps the dense stream anchored throughout. The design principle is that the temporal structure of what is hidden matters more than its spatial placement on the person.

To understand why block masks are harmful, consider this scenario: if the context contains spatial position $(x, y)$ at frame $t_1$, and the target is the same position $(x, y)$ at frame $t_2$, then the simplest prediction strategy is to copy the feature from $t_1$ to $t_2$—this is an appearance-copy task, not a motion-modeling task. The past-to-future split eliminates this shortcut entirely, since no information about the target position is visible in the context.

Because the two families sit at opposite ends of an appearance-versus-dynamics trade-off, per-batch mixtures are also supported: at step $i$, a batch draws the forecasting family with probability $p_i$ and the block family otherwise. $p_i$ can be constant or follow a two-phase schedule (forecasting first, blocks late). Mixing acts across batches, never within one, because batched training requires a fixed masked-token count per batch. The family draw is seeded by the step index so all data workers agree. The two ends of the sweep are exactly the block recipe and the forecasting recipe, so a single knob traverses the whole space. The released recipe is the pure forecasting endpoint of Eq. 2; mixing ships as a configuration knob rather than part of the objective.

Data Pipeline

A person detector inspects four sampled frames of each of 536,699 Kinetics-700 clips; a loose gate keeps candidates and records keypoints of every detected person into a per-shard manifest; a full-body gate then selects 164,431 clips (30.6 percent) by replaying thresholds over manifests. A second offline gate keeps 82,696 clips where at least two solid people are spatially engaged, of which 24,524 were recovered from source archives. AIST++ adds dance motion at 5 percent weight. The image branch trains on 958k LUPerson-T crops. Video sampling weights are 0.6 full-body, 0.35 interaction, 0.05 AIST++.

The data pipeline design emphasizes efficiency: four frames are sampled for detection to reduce computation, then manifests record keypoints for subsequent gate replay without re-detection. Evaluation subsets were frozen before any model saw them, and the released manifests distribute clip lists and gate parameters rather than video, so the corpus is Kinetics-bounded by construction.

Figure 1: Human-JEPA (blue, outlined) leads the human specialists on pose, identity, and anticipation across seven frozen-probe tracks under one protocol.

Training Configuration

Every arm continues pretraining from released V-JEPA 2.1-L for 200 epochs (60k steps) at global batch 192 video clips plus 576 person crops, bfloat16 on 4 GPUs, learning rate $10^{-4}$ with warmup and cosine decay, gradient clipping at 1.0, and a pre-backward loss-spike guard that rejects anomalous batches. Video input is 16 frames at 4 fps and 256 pixels; the forecast mask keeps the first half of tubelets. The learning rate follows cosine annealing: $\eta_t = \eta_{\min} + \frac{1}{2}(\eta_0 - \eta_{\min})(1 + \cos(\pi t / T))$, where $\eta_0 = 10^{-4}$ and $T$ is total steps.

graph TD
    A["V-JEPA 2.1-L
Initialization"] --> B["Anchored Continued Pretraining"]
    B --> C["Frozen Anchor A=f0
Prevents dense collapse"]
    B --> D["Image Branch
LUPerson-T crops
Re-grounds appearance"]
    B --> E["Forecasting Masks
Past-to-future split
Specializes temporal repr."]
    C --> F["Human-JEPA
0.3B params"]
    D --> F
    E --> F
    F --> G["Perception: Pose 0.620 AP
ReID 0.4635 mAP"]
    F --> H["Anticipation: cosine 0.873
Predictor head +0.06"]

Experiments

Perception Results

Under frozen probes with an identical protocol, Human-JEPA at 0.3B parameters faces Sapiens2 at 0.8B. Pose reaches 0.620 AP, ahead of every pixel-anchored specialist including the 2.7× larger Sapiens2-0.8B by 2.9 AP—of which the base family position contributes 2.3 and the specialization 0.6. This attribution is kept explicit throughout the paper. Person re-identification reaches 0.4635 mAP on Market-1501, turning the base's deficit against Sapiens2 into a lead and gaining 2.7 mAP over the base itself.

The frozen-probe protocol's core design ensures all models use the same head, training budget, frozen evaluation subsets, and feature interface, with no number copied from another paper. Dense parsing uses ATR with linear and DPT heads at 512 pixels; surface correspondence uses DensePose with a DPT head; pose uses a COCO keypoint probe reporting AP; action uses attentive probes on NTU RGB-D 120. ReID uses a frozen BNNeck probe: concatenated token means of four evenly spaced encoder depths on 256×128 crops feed a 512-dimensional reduction head trained with label-smoothed cross-entropy and batch-hard triplet under PK sampling.

Figure 2: Top—frozen ATR linear canaries at matched epochs; the full recipe (green) rises to the frozen baseline, the naive control (orange) collapses after 90 epochs, and the anchor-only run (gray) never collapses but plateaus. Bottom—training losses, indistinguishable and anomaly-free.

Anticipation Results

The released predictor head does +0.06 on NTU-120 early action recognition over the encoder alone—the only released head in the comparison that does no harm. The base V-JEPA 2.1's own shipped head costs it 2.98 points. The contrast is stark: the base model's head was trained on general video and introduces noise when applied to human actions, while Human-JEPA's head was trained on human video in the encoder's own representation space.

On 20 held-out Kinetics-700 validation clips, the encoder sees the first half and the predictor is asked for the latents of the second half: predictions agree with the actual continuation at 0.873 cosine, against 0.798 for holding the last observed latent fixed and 0.735 for scoring against a different clip's future. The margin holds on every clip and widens with distance—from 0.057 half a second ahead to 0.087 two seconds ahead. The forecast retrieves the exact moment it was asked to predict 40 percent of the time against a 0.6 percent chance rate, the signature of a forecast rather than a copy.

The cosine agreement metric can be formalized as: $c = \frac{\hat{z}^T z^*}{\lVert \hat{z} \rVert \cdot \lVert z^* \rVert}$, where $\hat{z}$ is the predicted latent and $z^*$ is the actual continuation's latent. The static reference baseline is $c_{\text{static}} = \frac{z_{\text{last}}^T z^*}{\lVert z_{\text{last}} \rVert \cdot \lVert z^* \rVert}$, holding the last observed frame's latent fixed. Human-JEPA's $c = 0.873$ significantly exceeds $c_{\text{static}} = 0.798$, demonstrating that the predictor learned a motion model rather than simple copying.

Model	Encoder-only top-1	Encoder + shipped head	Head effect
V-JEPA 2.1-L (base)	78.91	75.93	-2.98
Human-JEPA (ours)	79.93	79.99	+0.06

Table 1: NTU-120 early action recognition—the base's shipped head costs 2.98 points, while ours is the only one that does no harm.

Ablation Study

The mask family sets the cost. At a matched 34-epoch pilot, replacing block masks with the pure past-to-future split and touching nothing else moves action by +8.6 NTU points overall and +7.0 on the mutual slice, at no dense or pose cost. At full scale, every block-masked arm pays about five points of action, while the forecasting recipe returns to within noise of the base (70.4 versus 70.8; 79.0 versus 79.6 mutual). The same decision also controls ReID—a task family the recipe never saw: block masks collapse it by 17 mAP while forecasting gains.

The collapse is silent and selective. Naive continued pretraining ($A = f_{\bar{\theta}}$, video only) destroys dense human perception—pose AP falling from 0.614 to 0.110 and DensePose from 0.649 to 0.353—while training loss decreases throughout. Matched-epoch probes show the control indistinguishable from the anchored run for 90 epochs before collapsing over the next 40, with nothing anomalous in any trainer-visible signal. The collapse is selective for appearance: under a DPT readout, the control's body and geometry classes recover while garment and appearance classes remain collapsed.

Counterintuitively, lower learning rates make it worse (canary 0.520 at lr $10^{-4}$ versus 0.328 at $2 \times 10^{-5}$, both unanchored). This seems paradoxical—lower learning rate should be gentler—but the explanation is that the freshly initialized predictor converges more slowly at low learning rate and feeds structured but wrong gradients into the encoder for longer. Within each family, the lower-rate run reaches a lower final training loss and a worse canary: the optimizer succeeds at the wrong thing. Anchoring works at both rates because it removes the easy basin rather than racing the descent into it.

Figure 3: Naive continued pretraining collapses dense perception while training loss improves. The image branch and anchor each recover specific parts—anchoring prevents collapse, the image branch continually re-grounds appearance.

Arm	Params	ATR-lin	DensePose	Pose AP	NTU
A0: V-JEPA 2.1-L (frozen)	0.3B	.6230	.6490	.6142	70.8
C: naive continued PT	0.3B	.2947	.3534	.1105	66.5
C + image branch	0.3B	.6223	.6367	.6262	65.4
anchor only (no image)	0.3B	n/a	n/a	.554	61.9
A1: anchored block-mask	0.3B	.6235	.6384	.6279	65.6
Human-JEPA	0.3B	.6210	.6333	.6200	70.4

Table 2: Naive continued pretraining collapses dense perception while its training loss improves. Each ingredient recovers a specific part. Gray marks collapsed cells.

Falsification of Person-Level Objectives

Nine person-level objectives were systematically tested. Masking whole people ($|M|=1$) drops dense parsing to 0.7357, below the random-mask band of 0.7423±0.0010. Deletion does not teach partner reading (mutual 56.4 vs 56.5), because deleting a person from all of space-time removes them from the target side of every token the loss touches, so two-person co-motion is never a training signal. Any pose input makes it worse: a four-anchor skeleton trajectory scores 52.9, a single-anchor identity skeleton 51.5, against the 56.5 unconditioned control—because a masked person's latent target is dominated by body configuration, so the skeleton supplies most of the answer.

The causal partner-ablation probe serves as judge: on 256 held-out two-person clips per checkpoint, it predicts the masked person's tokens with the partner visible and again with the partner's exact token support removed from the context, reading the relative error increase as a direct causal measure of partner use. Across nine designs, two representation levels (token grid and person slots), and three loss families (point L1, per-person L1, contrastive), partner use never exceeds 0.72 percent on wild clips. A person-slot InfoNCE arm that solves its retrieval task emphatically (78 percent over roughly 768 candidates, chance 0.13 percent) shows the least partner use of any arm, so benchmark-shaped proxies dissociate completely from relational learning.

These results demonstrate that making "people" the unit of prediction—an intuitively natural design—is falsified in every constructed form. The probe, not any benchmark, is the instrument that makes these distinctions visible. On scripted NTU mutual validation clips, which no wild-trained arm ever saw, given coupling, the scene-forecasting model reads the largest partner use of any wild-trained arm (2.02 percent), while supervised mutual-action co-training suppresses partner use to the floor (0.37 percent) even as it solves its training task.

Quantitative Cost Analysis

The paper is unusual in its commitment to quantifying every cost of specialization rather than only reporting gains. Against the frozen base, averaged over two pretraining seeds, the recipe buys 0.6 AP of pose and 2.7 mAP of ReID while paying 0.9 points of dense parsing (ATR linear from 0.6230 to 0.6210), 1.9 points of DensePose (from 0.6490 to 0.6333), a residual 2.7-point action tax on NTU-120 (2.1 on the mutual slice), well under the block-mask recipe's 4.8 but not zero, and 2.9 points of general action on K700. Every cost is reported with its attribution—whether it comes from the base family position, the specialization, the mask family, or the data curation.

The two-by-two factorization in Table 6 completes the picture. With neither the anchor nor the image branch, the model collapses (pose AP 0.1105, a 50-point drop). The image branch alone holds the frozen baseline and matches the anchored arm's pose gain (0.6262), despite training on uncurated video—which also shows curation is not the active ingredient at matched objective, with the caveat that uncurated Kinetics is still human-dominated. The anchor alone never collapses—its canary holds a plateau through the exact window where the control loses 25 points—but it erodes slowly to 0.709 dense and 0.554 pose, concentrating on garment and appearance classes, the same selectivity in slow motion. The verdict is asymmetric and the mechanisms complementary: the anchor removes the drifting context target that causes collapse, and the image stream continually re-grounds the appearance features that video-only prediction lets fade.

Implications for Model Specialization

Beyond the specific results, Human-JEPA offers a general recipe for specializing foundation models without destroying their base capabilities. The three design decisions—freeze the initial quality as an anchor, use an auxiliary branch to maintain what the main objective lets fade, and choose the temporal structure of masking to match the target domain—transcend the human-centric setting. Any domain that requires adapting a pretrained model to new data distributions while preserving base capabilities could benefit from this framework.

The falsification of all nine person-level objectives is particularly instructive. The authors constructed variants including whole-person deletion with one or two masks, skeleton trajectory priors with four or one anchor, identity-conditioned priors, person-slot InfoNCE, and person-level forecasting in the style of Causal-JEPA. Each was designed to be a natural way to teach the model about people specifically. That all nine failed—and were shown to fail through a causal probe rather than a benchmark—cautions against intuitive masking strategies in representation learning. The probe itself, which measures partner use by predicting masked-person tokens with and without the partner visible, is a methodological contribution beyond the specific model.

The selectivity of the collapse—appearance classes degrade while body geometry classes recover—provides a mechanistic understanding of what continued pretraining on video changes about a model. Video prediction, by its nature, emphasizes dynamic features (motion, pose changes, interaction patterns) at the expense of static appearance features (clothing, texture, individual identity cues). The image branch counteracts this by continually re-presenting full-body appearance at a scale video cannot provide. This complementarity between temporal specialization and appearance maintenance is a principle that likely generalizes: any specialized training that de-emphasizes certain feature classes needs an auxiliary objective to preserve them.

The two-phase schedule (forecasting first, blocks late) is offered as an ablation rather than the released recipe. The reasoning is that early training benefits from the forecasting mask's motion modeling, while late training might benefit from the block mask's appearance completion. The released recipe uses pure forecasting throughout, keeping the design simple. The fact that a single knob—the probability of drawing the forecasting versus block family—traverses the entire design space between two endpoints is an elegant property that makes ablation systematic and reproducible.

The released predictor head's neutrality is itself a notable result. In V-JEPA 2.1, the shipped head was trained on general video and degrades human action recognition by 2.98 points. Human-JEPA's head, trained on human video in the encoder's own representation space, adds 0.06 points—a small but nonzero gain that makes it the only shipped head in the comparison that does no harm. This finding has practical implications: when deploying a video model for human-centric tasks, the choice of predictor head matters, and a head trained in the specialized encoder's space is safer than one shipped with the base model.

Limitations

First, Human-JEPA concedes high-resolution dense parsing. The authors candidly acknowledge this—ATR linear drops slightly from 0.6230 to 0.6210, DensePose from 0.6490 to 0.6333. Sapiens2's advantage at 4K resolution and billion parameters in pixel-level parsing remains unchallenged; Human-JEPA chose "people in time" over "people in high-resolution pixels."

Second, a residual action tax remains. While forecasting masks reduce the block-mask tax from 4.8 to 2.7 points, it is not zero. NTU-120 drops from 70.8 to 70.4, K700 from 54.7 to 51.8. The authors report this as a cost of specialization rather than a preserved property. The K700 column also carries an honesty note: both the base model and ours saw Kinetics video during pretraining, so the K700 column measures general-action retention, not specialization gain.

Third, the boundary of the anticipation claim. At the current observation fraction, the rollout adds essentially nothing over the encoder alone—the value demonstrated is representation consistency, not extra anticipatory signal. A pre-registered variable-horizon test does not change this verdict at shorter horizons. This means Human-JEPA's predictor head currently contributes mainly "no harm" rather than significantly advancing prediction capability—an honest boundary statement.

Conclusion and Outlook

Human-JEPA demonstrates that a released general-purpose video JEPA can be safely specialized into a human-centric model through anchored forecasting—a frozen anchor prevents collapse, an image branch continually re-grounds appearance, and a forecasting mask family releases the temporal representation. A 0.3B-parameter model leads pixel-anchored specialists on pose and re-identification at 2.7× fewer parameters, and its predictor head is the first that does not degrade anticipation. Nine person-level objectives were systematically falsified by a causal probe—the natural next step of making people the unit of prediction was falsified in every constructed form.

This work has profound implications for embodied AI. Humanoid robots need to understand human behavior patterns—perceiving pose and identity, predicting the next action—to collaborate safely and effectively with humans. Human-JEPA provides a paradigm for acquiring both capabilities simultaneously under a self-supervised framework without any human annotations. Its design principles of anchored forecasting (freeze initial abilities, complement with image branch for appearance, use temporal split to release dynamics) may apply to model specialization in other domains as well.

Pixel-anchored pretraining owns static appearance at high resolution; Human-JEPA owns people in time. The two halves are thus united.

Golden Lines

"Pixel-anchored pretraining owns static appearance at high resolution; Human-JEPA owns people in time."

"You keep what you anchor."

"The optimizer succeeds at the wrong thing, drifting toward features that are easy to predict and less informative."