Turning Egocentric Video into 3D Hand Actions

TL;DR

We built an open-source pipeline that turns raw egocentric video into 3D hand-action trajectories for robot policy training — without any sensors beyond a monocular RGB camera. The final system reconstructs 21-joint hand poses at 15.53 FPS on an H100 with 52.04 mm Action MPJPE, combining VGGT camera estimation, HaWoR temporal hand reconstruction, metric window alignment, and targeted trajectory corrections.

Why egocentric video?

Most current robot-learning systems are trained by behavior cloning: given the current observation, robot state, and sometimes a task instruction, the model learns to predict the next action chunk the robot should perform. That means every training example needs this state-action pairing.

Teleoperation gives the cleanest version of that pair. An operator executes demonstrations on the robot itself, so its cameras, joint states, and actions (what the model will then predict) are recorded in the target embodiment. The main problem with teleop data is scalability. A full robot must be available at every collection site, operators need training, and human-controlled robot arm demonstrations can be much slower than a person performing the task directly.

To achieve both scale and diversity, labs have been trying to leverage egocentric data. These are first-person videos where the person wearing the camera performs tasks directly with their hands. They are cheap to acquire, naturally diverse, and captured at full human speed, and websites like YouTube have countless hours of them.

The scale of ordinary egocentric video comes with a tradeoff: it gives us pixels of a human demonstration, but not the robot actions that a policy should predict next. To turn egocentric video into action data, we need to reconstruct the 3D position and movement of the operator's hands in physical space.

That is the problem we tackle in this post: how to turn raw egocentric video into 3D hand-action trajectories, without any additional sensors. We will build the pipeline from open-source components, optimize its performance, and ensure that it runs at 15+ FPS on an H100.

The problem: recovering physical action

Action targets

An egocentric video records a human performing an action, but a robot policy ultimately needs commands specific to its own body, such as end-effector poses or joint positions. These robot-specific commands can be obtained by first extracting the physical 3D motion of the person's hands, including how the wrists and fingers move through metric space while the camera moves with the wearer. We use this as a common representation of the demonstrated action that can be adapted to different robot embodiments.

We focus on the RGB-only version of this problem: reconstructing 3D hand actions from monocular video, with camera calibration when provided. We do not use measured depth, LiDAR, stereo, or inertial data as inputs, even when they are available. Evaluating how much these additional sensor streams improve the reconstruction is left to future work.

The resulting trajectory is directly usable for robotic policy training if treated as a co-embodiment. It could also be transformed into aligned robotic actions but this is a separate problem, which we do not evaluate here. Robot retargeting introduces robot kinematics, inverse kinematics, joint limits, and controller design on top of the visual reconstruction problem.

Trajectory representation

There are multiple ways to represent a hand trajectory. A compact representation can treat the hand like a gripper, while a detailed representation can retain the motion of the wrist and fingers.

Prior work spans this range. HumanEgo converts the hand into a virtual parallel-jaw gripper (a 6-DoF pose plus a single opening value). EgoScale begins with 21 human hand keypoints, but retargets the finger motion into the joint angles of a specific 22-degree-of-freedom robot hand and represents arm motion as changes in wrist pose. ViTra keeps a human-hand representation, using frame-to-frame changes in wrist position and orientation together with MANO finger-pose parameters.

We choose the same general level of detail as ViTra, but use a more directly geometric target. We represent each hand as 21 metric 3D joint positions — one wrist joint and 20 finger joints — rather than MANO parameters or robot-specific joint angles. This keeps the target robot-agnostic and makes the resulting trajectory directly measurable, while allowing smaller representations to be derived later. We also report wrist position accuracy separately.

Separating hand and camera motion

Choosing the 21 joints tells us what each hand pose contains. We must also determine how those poses relate across a video recorded by a moving camera.

Hand predictions are initially expressed in the coordinate system of each frame's camera. These coordinate systems move with the wearer's head. If the wearer keeps their hand perfectly still but turns their head, the hand's camera-space position changes even though its physical position does not. To recover the demonstrated action, we need to combine each frame's hand pose (in camera space) with that frame's camera pose (the camera's position and orientation in a shared world frame) to obtain the hand's world-space trajectory.

This gives the pipeline two major sub-problems:

• Hand reconstruction: detecting hands in each frame and predicting their 3D joint positions in camera space.
• Camera reconstruction: estimating the camera's trajectory through a shared 3D world frame with correct metric scale.

When both are solved, world-space hand joints are obtained by transforming camera-space joints through the camera-to-world pose at each frame.

Baseline pipeline: HaWoR

We start from HaWoR, an end-to-end pipeline for reconstructing 3D hand trajectories in a common world coordinate system from egocentric RGB video.

Pipeline components

The HaWoR pipeline begins with the WiLoR hand detector. It processes each frame independently and predicts a bounding box, confidence score, and left- or right-hand label for each detected hand. Boxes below a chosen confidence threshold are discarded. HaWoR then links the remaining boxes across consecutive frames when their handedness and image locations are consistent. This produces separate left- and right-hand tracks: sequences of accepted crops that appear to belong to the same hand.

HaWoR then reconstructs each hand track with a temporal network that uses neighboring frames rather than treating every crop independently. For every frame in the track, the network predicts the wrist's position and orientation relative to the camera, together with MANO hand-shape and finger-pose parameters. MANO decodes the shape and finger pose into a 3D hand mesh, while the predicted wrist pose places that mesh relative to the camera. This produces the 21 camera-space joint positions $\mathbf p^C_{t,h,j}$, where $t$ is the frame, $h$ identifies the left or right hand, $j\in1,\ldots,21$ identifies the joint, and the superscript $C$ denotes camera coordinates.

The pipeline next uses DROID-SLAM to estimate how the egocentric camera moved. DROID-SLAM does this by finding the same visual points in consecutive frames and measuring how their positions changed. This works for the static background, but hands move independently and could be mistaken for camera motion. HaWoR therefore uses the reconstructed hand meshes to identify and ignore the hand pixels, then estimates camera motion and scene depth from the background.

To verify that masking matters, we reran DROID-SLAM twice on the same 1,500-frame HOT3D sequence with identical settings. The only change was whether wearer-hand pixels were masked. Against the HOT3D camera ground truth, masking reduced mean camera-trajectory error from 0.247 m to 0.220 m.

While this monocular SLAM approach can recover camera rotation and the shape of the camera path, it cannot determine the path's real-world scale. The same image motion could result from a camera moving one meter through a large scene or ten centimeters through a smaller version of it. DROID-SLAM therefore recovers the camera trajectory only up to an unknown global scale: its translations are internally consistent, but are not yet measured in meters.

To recover this scale from RGB alone, Metric3D estimates a dense depth map in metric units for each SLAM keyframe. HaWoR compares that depth with DROID-SLAM's unscaled depth on static-scene pixels, excluding the rendered hand masks. Each keyframe provides an independent estimate of the conversion from DROID-SLAM's internal scale to meters. Because Metric3D's depth predictions are noisy, these estimates vary across keyframes. HaWoR takes their median and applies that single scale to the entire sequence. The resulting camera-to-world transform at frame $t$ consists of a rotation $R^{C\rightarrow W}_t$ and metric translation $\mathbf t^{C\rightarrow W}_t$, both defined relative to the fixed shared world frame $W$.

We can now combine the hand and camera trajectories. For joint $j$ on hand $h$, its world-space position at frame $t$ is obtained by applying that frame's camera-to-world transform to its camera-space position:

$$ \mathbf p^W_{t,h,j} = R^{C \rightarrow W}_t \mathbf p^C_{t,h,j} + \mathbf t^{C \rightarrow W}_t. $$

The rotation expresses the camera-space point using the world axes, and the translation adds the camera's position in the shared world. Repeating this for every directly reconstructed joint and frame places all observed hand poses in one shared 3D trajectory, even while the wearer moves their head. This only works once the camera trajectory has metric scale: otherwise the camera translation and the metric hand reconstruction would use incompatible units.

Part 1 — Camera reconstruction: VGGT

The baseline HaWoR pipeline depends on DROID-SLAM and Metric3D for camera reconstruction and metric scale. DROID-SLAM is accurate but slow. To reach our throughput target, we replaced this stage with VGGT, a feed-forward network that predicts camera poses, depth, and point maps for a window of frames in a single forward pass.

VGGT takes a fixed window of $K$ frames and predicts camera poses, depth, point maps, and intrinsics for that window together. It has no persistent world state beyond those $K$ frames. To process a longer video, our first pipeline simply split it into consecutive windows with no overlap and concatenated their outputs.

That strategy treated every window as if it already used the same world coordinate system. It did not. Each window could reconstruct the local scene correctly while choosing a different global position, orientation, or scale from the window before it. Concatenating those independent reconstructions therefore introduced trajectory discontinuities in both the camera and, after world fusion, the hands.

Long videos need one coordinate system

To improve coherence between VGGT camera-pose windows, we introduced overlap. Instead of ending one window and starting the next immediately afterwards, we processed a short set of source frames in both windows.

More precisely, imagine two consecutive windows, A and B, that share the same overlap frames. VGGT reconstructs those frames twice: once as part of window A, and once as part of window B. The images are identical, but the predicted cameras and 3D points are expressed in two different local world coordinate systems.

We use the duplicate reconstruction of those overlap frames to estimate how window B's coordinate system must be rotated, translated, and potentially rescaled to match window A's coordinate system. For a camera center at overlap frame $t$, window A predicts $\mathbf c_t^A$ and window B predicts $\mathbf c_t^B$. They represent the same physical camera location in two different coordinate systems. We fit a transform that makes those corresponding locations agree:

$$ \mathbf c_t^A \approx s\mathbf R\mathbf c_t^B + \mathbf q. $$

Here $\mathbf R$ is a 3D rotation, $\mathbf q$ is a 3D translation, and $s$ is one scale factor. After fitting it from the overlap, we apply the same transform to every 3D prediction from window B, including camera centers, scene points, and hand joints.

We tested two transformation families. SE(3) is a rigid transformation: it rotates and translates one reconstruction into the other but does not change its size. Sim(3) additionally estimates one uniform scale factor, allowing one reconstruction to be enlarged or reduced before it is rotated and translated. This is useful for monocular reconstruction, where independently processed windows can assign different scales to the same scene.

We also tested two sources of alignment evidence. Camera-center alignment uses the camera positions predicted for the corresponding frames in the overlap. Depth-derived alignment uses each window's predicted depth and intrinsics to back-project corresponding overlap pixels into paired 3D scene points.

Depth-derived Sim(3) produced the best alignment result. We kept this alignment method fixed and varied only the overlap size. Increasing the overlap consistently lowered action MPJPE, at the cost of a slower pipeline because more frames were processed twice. We chose a 40-frame overlap, the best configuration in the sweep, because its throughput was still comfortably within our compute budget.

Longer context helps

Beyond overlap, the second important parameter is the window length. Larger windows give VGGT more temporal context for estimating camera motion and scene geometry, and create fewer coordinate-system boundaries across a video. Those benefits come at a higher GPU-memory cost. We fixed the window length at 195 frames.

Input resolution

The final parameter to choose is VGGT's input resolution. VGGT was trained with inputs up to 512 pixels, so a higher-resolution input should preserve more visual detail: small scene features, sharper depth boundaries, and more reliable image correspondences for camera reconstruction. The trade-off is higher GPU-memory use. The higher-resolution bucket consistently improved the action score. Its measured throughput was still above our 15 FPS requirement, so we retained 416 px.

After all these experiments, our VGGT-based camera-reconstruction system used 200-frame windows, 40-frame overlap, a 416-pixel input bucket, depth-derived Sim(3) alignment, and linear blending. The full pipeline still ran at 15.53 FPS end to end. More importantly, it reached 55.60 mm Action MPJPE, improving on every other camera-reconstruction system we tested, including the original HaWoR pipeline at 59.12 mm.

Part 2 — Hand detection and tracking

The geometry system from the previous section can only turn a hand trajectory into a stable world-space trajectory after we reconstruct that hand in each camera frame. To do that, we first need to find the hands in the video, track them across frames, and then infer MANO parameters from the resulting image crops.

The baseline detector and tracker

HaWoR starts with the WiLoR hand detector. For every frame it predicts hand bounding boxes, a confidence score, and a left/right label. HaWoR then uses BoT-SORT to associate boxes over time. We kept the WiLoR detector, but began with a lighter association rule. For each side, we retain the highest-confidence box in a frame and link it only to nearby boxes of the same side. This is deliberately conservative: a false recovery is more costly than a short gap.

Detector selection

Our first question was whether a detector trained on a broad hand dataset could replace WiLoR. We compared WiLoR with a YOLOv10n detector trained on HaGRID, a large hand-image dataset. Both used a confidence threshold of 0.25. HaGRID missed almost half of the visible egocentric hands, despite its low threshold. Since it was far below our 75% coverage requirement, we did not run the full hand-reconstruction pipeline with it.

Confidence threshold

Every WiLoR box has a confidence score. Lowering the threshold gives HaWoR more crops to reconstruct but increases the risk of false predictions, while raising it yields fewer direct predictions and more interpolation. The best action score occurred at a threshold of 0.75. Our metric penalizes a method that predicts only easy frames: for missing frames, we linearly interpolate the hand trajectory before comparing it with ground truth.

Recovering short gaps

A strict 0.75 threshold leaves short gaps where WiLoR produces a weaker but plausible box. To fix this, we only admit a low-confidence proposal when it occurs inside a short same-side gap with high-confidence WiLoR boxes immediately before and immediately after it. For each proposed box, we linearly interpolate the two high-confidence anchor boxes and calculate IoU between that expected box and the weak proposal. If the IoU exceeds a threshold, we accept the weak box.

Part 3 — Post-processing

After world fusion, the raw hand trajectories have two correctable issues: per-clip bone-length scales can drift, and wrist depth along the original camera ray is the least-constrained prediction. We apply a bounded clip-level bone-scale correction that adjusts the scale of each hand's MANO mesh to match the median bone length across the clip, then optimize the wrist depth along the original camera ray using temporal smoothing. The camera translation is filtered over three frames. Finally, we apply the bounded clip-level bone-scale correction and optimize wrist depth along the original camera ray with an acceleration weight of 0.2 and detector-confidence weighting.

Evaluation: Action MPJPE

Mean per-joint position error (MPJPE) is a standard metric for evaluating 3D pose estimates. It measures the Euclidean distance, in millimeters, between each predicted joint and its ground-truth position, then averages across the joints. We extend the same idea to the one-second, camera-relative hand trajectories. We call this trajectory-level metric Action MPJPE.

We start a one-second action chunk at every possible frame. Each start frame produces separate left- and right-hand chunks. We include a hand chunk when HOT3D provides its starting camera pose and marks the hand as reliably visible in at least one future frame. Within an included chunk, only the future frames where the hand is visible contribute to the score.

For each chunk starting at frame $t$, we anchor the prediction and ground truth separately. We transform the predicted future joints from the pipeline's reconstructed world frame into the coordinate system of its predicted camera at frame $t$. Independently, we transform the HOT3D future joints from the HOT3D world frame into the coordinate system of the ground-truth camera at frame $t$, aligning both trajectories to the camera axes of the same physical observation so we can directly measure the Euclidean distance between corresponding joints. Action MPJPE first averages these errors over the visible frames and 21 joints within each hand chunk, then averages all eligible hand chunks equally:

$$ \begin{aligned} E_{\mathrm{action}} = \operatorname*{mean}_{(t,h)\in\mathcal C} \left[ \operatorname*{mean}_{\substack{ i\in\mathcal V_{t,h}\\ j\in\{1,\ldots,21\} }} \left\| \hat{\mathbf p}_{t,h,i,j} - \mathbf p_{t,h,i,j} \right\|_2 \right] \end{aligned} $$

Here $\mathcal C$ contains the eligible left- and right-hand chunks, while $\mathcal V_{t,h}$ contains the HOT3D-visible future frames for hand $h$ in the chunk beginning at $t$. The index $j$ runs over the hand's 21 joints. We report the result in millimeters. Lower Action MPJPE is better.

End-to-end performance

System	Action MPJPE (mm) ↓	Throughput (FPS) ↑	Direct Coverage ↑
VGGT baseline (initial)	90.73	~16	—
HaWoR (original)	59.12	3.34	—
VGGT + depth-Sim(3) alignment	55.60	15.53	—
Final system	52.04	15.53	~75%

Compared with the original HaWoR pipeline, the final system reduces action error by 12.0% while increasing measured throughput from 3.34 to 15.53 FPS. Compared with our initial VGGT-based baseline, action error falls by 31.3%. The lower direct coverage remains a real limitation. Interpolation does not excuse missing predictions; it assigns them a trajectory derived from surrounding observations so they still incur error rather than disappearing from the evaluation.

Failure analysis

Where the remaining error comes from

The final question is which parts of the pipeline still produce the largest errors. To separate camera reconstruction from hand reconstruction, we repeated the action evaluation using only frames with a direct HaWoR prediction and visible ground truth, deliberately excluding non-predicted frames. We then replaced the camera trajectory and each camera-space hand component with ground truth and used Shapley attribution to distribute their interactions.

On frames where the pipeline reconstructs a hand, its camera-space prediction is the dominant remaining error source. Wrist translation, particularly monocular depth, is the largest component. Replacing only the camera trajectory with ground truth reduces the error from 39.20 to 34.87 mm, while replacing the complete hand-model prediction reduces it to 12.67 mm.

Depth is the largest coordinate error

Using the same direct-prediction subset, we decomposed the action error along the axes of the camera at the start of each action. The horizontal x-axis contributes 11.52 mm, the vertical y-axis contributes 10.86 mm, and the depth z-axis contributes 16.82 mm. Depth accounts for about 43% of the 39.20 mm direct-only action error, although the two image-plane axes contribute more when combined.

Camera motion matters more than window boundaries

Although camera pose contributes less error than hand reconstruction, we also studied the effect of VGGT window boundaries on prediction quality. VGGT window boundaries remain visible but are not dominant: actions crossing a boundary score 13.29 mm, compared with 12.42 mm elsewhere. Larger camera motion has a much stronger relationship with camera error. Actions with less than 2 cm of camera translation score 5.74 mm, while actions above 10 cm reach 25.09 mm. Similarly, error rises from 5.79 mm below 2 degrees of camera rotation to 16.50 mm above 10 degrees.

Problematic environments

Finally, we tested the open-source pipeline on data from real-world deployments. The HOT3D benchmark does not cover several deployment-specific challenges, including wearer identification, gloves, and wrist-mounted cameras. Although the pipeline works fairly well in clean environments without heavy occlusion, the most significant issues on real-world data are typically related to camera-space hand prediction. In particular, the detector sometimes identifies hands belonging to people other than the camera wearer, or fails entirely when gloves or wrist-mounted cameras are present.

Conclusion

In this blog, we introduced, improved, and diagnosed the current state of open-source hand-tracking pipelines.

The strongest system was not a single new reconstruction model. It combined conservative hand detection, temporal HaWoR reconstruction, long-window VGGT camera estimation, metric window alignment, and narrowly targeted trajectory corrections. Together, these changes reduced the one-second action error of our initial VGGT-based system from 90.73 to 52.04 mm while keeping the complete pipeline above 15 FPS.

The benchmark also exposes where current open-source systems still fail: camera-space hand reconstruction, reliable detection, and camera-pose estimation during larger head movements. Solving these remaining limitations is critical to making egocentric video a practical source of robot-training data.