HITTER: A HumanoId Table TEnnis Robot via Hierarchical Planning and Learning

Paper Information

• Title: HITTER: A HumanoId Table TEnnis Robot via Hierarchical Planning and Learning
• Authors: Zhi Su, Bike Zhang, Nima Rahmanian, Yuman Gao, Qiayuan Liao, Caitlin Regan, Koushil Sreenath, S. Shankar Sastry (University of California, Berkeley)
• arXiv: 2508.21043v1 (cs.RO, August 28, 2025)
• Code: Not publicly released at the time of writing. The Sastry group maintains related repositories (sastry-group/robot_table_tennis, sastry-group/LATTE-MV), but this paper provides no code link.
• Hardware: Unitree G1 general-purpose humanoid, racket mounted on the right wrist via a 3D-printed connector

One-Sentence Summary

HITTER decomposes humanoid table tennis into two layers: a model-based planner that predicts the ball trajectory and decides where, how fast, and when to strike, and a reinforcement-learning whole-body controller trained with two human swing references, which drives a Unitree G1 to sustain rallies of up to 106 consecutive shots against a human opponent.

Background and Motivation

Humanoid robots have made striking progress in locomotion and motion imitation, yet most work still concerns control in free space or static contact with the ground. Interacting with fast-moving objects through manipulation is a different beast: it couples whole-body coordination across dozens of joints with a perception-action loop that must close within extreme time scales. Table tennis makes these demands concrete. A ball travelling above 5 m/s leaves only sub-second reaction windows, so the robot must perceive, predict, plan, and strike within a few hundred milliseconds, blending fast arm swings and waist rotation with quick stepping and balance recovery.

Among racket sports, table tennis is arguably the hardest for a humanoid: compared with badminton or tennis, the distances are shorter, exchanges are faster, and the reaction window is narrower. The paper argues that humanoid table tennis therefore serves as a unique robotics testbed, for three reasons: (a) it is highly dynamic and demands rapid trajectory prediction and strike planning; (b) consecutive rallies require agile motion and balance recovery after every stroke; and (c) human-like striking motions matter both for effectiveness and naturalness.

The prior art splits into two camps. On the robotic table-tennis side, most systems use specialized arms with model predictive control or precise hitting-velocity control; the DeepMind Gemini Robotics line reached amateur-level performance with a hierarchical architecture and sim-to-real adaptation. A notable gap: earlier humanoid attempts used impedance control but kept the robot standing still, which capped the effective hitting range. On the humanoid whole-body control side, the field has moved from imitating single motions with large-scale RL, through decoupled upper/lower-body policies, to general motion trackers such as BeyondMimic. HITTER positions itself at the intersection: a whole-body controller purpose-built for fast dynamic interaction with the environment.

Architecturally, the authors adopt a hierarchical framework that separates high-level planning from low-level control. The planner supplies fast trajectory prediction and strike planning, which gives the controller a stable command interface and improves sample efficiency during training. The RL controller is trained on consecutive strikes so that it learns agile motion and reliable balance recovery, and only two human reference motions are injected into training — yet the resulting strikes look natural and human-like. The system is fully autonomous, with no teleoperation, and runs on a general-purpose humanoid without specialized hardware.

Fig. 1: System overview. (a) The racket is mounted on the right wrist with a 3D-printed connector; the ball is covered in reflective tape for motion capture. (b) Nine OptiTrack cameras track the ball position, robot base position, and base forward vector. (c) The model-based planner predicts striking position, velocity, and time. (d) The RL whole-body controller $\pi_{WBC}$ is trained in simulation and deployed on the real robot.

Method

1. System Setup and State Estimation

The sensing backbone is a motion capture rig: nine OptiTrack cameras track the ball position at 360 Hz with millimeter-level accuracy. The ball is wrapped in reflective tape and the robot base carries multiple markers rigid-body-modelled for a 6D pose. The playing field is a regulation table (2.74 m × 1.525 m, surface 0.76 m above the ground), with the coordinate frame defined at the table center on the top surface, the $x$-axis along the long side and the $z$-axis pointing up; the humanoid stands on the $x<0$ side. The RL policy outputs desired positions for all 29 joints at 50 Hz, converted to torques by a PD controller.

Because the mocap system provides position only, the planner first estimates the ball's velocity. It performs a least-squares fit of a second-order polynomial to the ball position over time, independently per coordinate, using the nearest 31 position measurements. When a bounce on the table is detected, the buffer is cleared so pre-bounce data never contaminates the fit. Evaluating $\mathbf{p}(t)$ and its derivative $\dot{\mathbf{p}}(t)$ at the current timestep then yields smooth estimates of both position and velocity.

2. Ball Trajectory Prediction: A Hybrid Dynamics Model

Trajectory prediction uses a hybrid dynamics model adopted from prior robotic table tennis work: continuous flight dynamics interrupted by discrete impact dynamics at the table surface:

$$\Sigma:\ \mathbf{a}=-k\|\mathbf{v}\|\,\mathbf{v}+\mathbf{g},\quad \mathbf{p}\notin\mathcal{S}$$

$$\Sigma:\ \mathbf{v}^{+}=\mathbf{C}\,\mathbf{v}^{-},\quad \mathbf{p}\in\mathcal{S}$$

The flight phase models quadratic air drag with coefficient $k$ plus gravity $\mathbf{g}$; spin is assumed small enough that Magnus effects are neglected. The impact phase applies the restitution matrix

$$\mathbf{C}=\mathrm{diag}(C_{h},\,C_{h},\,-C_{v}),$$

with horizontal and vertical restitution coefficients $C_{h}$, $C_{v}$, over the impact set $\mathcal{S}=\{\mathbf{p}\mid p_{z}=0\}$. The three parameters $k$, $C_{h}$, $C_{v}$ are identified from 15 recorded trajectories in which the ball is launched, bounces once on the opposite side, and exits the table, by fitting $\|\mathbf{a}-\mathbf{g}\|=k\|v\|^{2}$ together with $\|v_{x}^{+}\|,\|v_{y}^{+}\|=C_{h}(\|v_{x}^{-}\|,\|v_{y}^{-}\|)$ and $\|v_{z}^{+}\|=C_{v}\|v_{z}^{-}\|$. With this model, explicit step-by-step time integration predicts the ball's future position and velocity from the current estimate; given a predefined virtual hit plane at $x=-1.37$ m, the hitting time and position fall out directly.

3. Racket–Ball Interaction: From Desired Landing to Racket Velocity

The goal is a valid return rather than a precise landing point, so the interaction model is deliberately simple: at impact the racket plane is perpendicular to its velocity vector, the coefficient of restitution $C_{r}$ acts along the racket normal, tangential friction is neglected, and post-impact flight is governed by gravity alone. Given the desired landing point $\hat{\mathbf{p}}_{l}$ (set to the center of the opponent's side), the desired hitting position $\hat{\mathbf{p}}_{\mathrm{racket}}$, and a predefined flight time $\Delta t$, the desired outgoing ball velocity is

$$\mathbf{v}_{o}=\dfrac{\hat{\mathbf{p}}_{l}-\hat{\mathbf{p}}_{\mathrm{racket}}}{\Delta t}+\frac{1}{2}\mathbf{g}\Delta t.$$

Combining $\mathbf{v}_{o}$ with the predicted incoming velocity $\mathbf{v}_{i}$ under the restitution model yields the desired racket velocity

$$\hat{\mathbf{v}}_{\mathrm{racket}}=\dfrac{\mathbf{v}_{o}\cdot\mathbf{u}+C_{r}\mathbf{v}_{i}\cdot\mathbf{u}}{1+C_{r}}\mathbf{u},\qquad \mathbf{u}=\dfrac{\mathbf{v}_{o}-\mathbf{v}_{i}}{\|\mathbf{v}_{o}-\mathbf{v}_{i}\|},$$

where $\mathbf{u}$ is the unit vector along $\mathbf{v}_{o}-\mathbf{v}_{i}$. The triplet $(\hat{\mathbf{p}}_{\mathrm{racket}},\hat{\mathbf{v}}_{\mathrm{racket}},t_{\mathrm{strike}})$ is the planner's hand-off to the learning-based controller.

4. Human Motion References: Two Clips Are Enough

To keep the strikes human-like, only two reference motions are used — forehand and backhand — both generated by the same pipeline. A video clip of a human performing the swing is reconstructed into a SMPL motion clip with GVHMR, then retargeted to the humanoid with GMR, producing base pose and joint positions at 30 Hz. Following BeyondMimic, the clip is interpolated from 30 Hz to 50 Hz to match the control frequency; base linear/angular velocities and joint velocities come from central differencing, and forward kinematics gives each body's pose $\mathbf{T}_{b}$ and twist $\mathbf{V}_{b}$. Only bodies above the pelvis (pelvis, torso, arms) are tracked, with the pelvis as the anchor body $b_{\mathrm{anchor}}$. Each processed clip contains 94 frames (1.88 s), with the strike occurring at the 43rd frame (0.86 s) — a number that recurs in the agility analysis below.

5. MDP Design: Separate Base and Racket Commands

A deliberate departure from prior work is splitting the command space. Instead of commanding the global racket position and velocity at strike time, HITTER issues two commands: $\hat{\mathbf{p}}_{\mathrm{base},xy}$, the desired base position in the world frame, which encourages the robot to arrive at the target in time; and $[\hat{\mathbf{p}}_{\mathrm{racket}},\hat{\mathbf{v}}_{\mathrm{racket}}]$, the racket position relative to the base plus the racket velocity, both in the world frame. The desired base orientation always faces forward. To support consecutive strikes and swing switching, each episode lasts 10 s; after every swing, the next swing type is sampled uniformly, and the racket target position, racket target velocity, and base target are sampled conditionally on the swing type. The striking plane is fixed 0.4 m in front of the robot, so only the $y$ and $z$ coordinates of the racket target are sampled, and the forehand and backhand target regions are non-overlapping.

6. Reward and Asymmetric Actor-Critic

The reward decomposes into imitation, goal-tracking, and regularization terms:

$$r=w_{i}r_{i}+w_{g}r_{g}+w_{r}r_{r},$$

where $r_{i}$ encourages imitation of the upper-body reference motion, $r_{g}$ rewards tracking of commanded goals, and $r_{r}$ regularizes. $r_{i}$ and $r_{r}$ are dense throughout the episode, while $r_{g}$ contains sparse, high-weight terms: the racket position/velocity/orientation tracking rewards activate only in a short window around the hitting time, and the base position tracking reward activates only before the strike, letting the policy prepare the transition to the next target after contact.

Training uses an asymmetric actor-critic: the critic sees privileged information the actor cannot access at deployment. Table I lists the observation spaces — base angular velocity $\bm{\omega}_{\mathrm{base}}\in\mathbb{R}^{3}$, projected gravity $\mathbf{g}_{\mathrm{base}}\in\mathbb{R}^{3}$, base forward vector $\mathbf{e}_{\mathrm{base},x}\in\mathbb{R}^{2}$, target base offset $\hat{\mathbf{p}}_{\mathrm{base},xy}-\mathbf{p}_{\mathrm{base},xy}\in\mathbb{R}^{2}$, target racket position and velocity in $\mathbb{R}^{3}$, time to strike, joint positions/velocities $\mathbf{q},\dot{\mathbf{q}}\in\mathbb{R}^{29}$ and last action $\mathbf{a}_{\mathrm{last}}\in\mathbb{R}^{29}$ — while the critic additionally receives the body poses $T_{\mathcal{B}}\in\mathbb{R}^{7|\mathcal{B}|}$ (for accurate return estimation) and the episode time remaining $t_{\mathrm{left}}\in\mathbb{R}$, since several sparse $r_{g}$ terms make the episodic return depend on how many strikes remain. Actor and critic are both MLPs with hidden layers of 512, 256, and 128. At deployment, a heuristic over $\mathbf{p}_{\mathrm{base},xy}$ and $\hat{\mathbf{p}}_{\mathrm{racket}}$ decides forehand versus backhand; this binary variable only assists computing $\hat{\mathbf{p}}_{\mathrm{base},xy}$ and is not part of the policy observation.

Observation	Actor	Critic
Base angular velocity $\bm{\omega}_{\mathrm{base}}\in\mathbb{R}^{3}$	✓	✓
Projected gravity $\mathbf{g}_{\mathrm{base}}\in\mathbb{R}^{3}$	✓	✓
Base forward vector $\mathbf{e}_{\mathrm{base},x}\in\mathbb{R}^{2}$	✓	✓
Target base offset $\hat{\mathbf{p}}_{\mathrm{base},xy}-\mathbf{p}_{\mathrm{base},xy}\in\mathbb{R}^{2}$	✓	✓
Target racket position $\hat{\mathbf{p}}_{\mathrm{racket}}\in\mathbb{R}^{3}$	✓	✓
Target racket velocity $\hat{\mathbf{v}}_{\mathrm{racket}}\in\mathbb{R}^{3}$	✓	✓
Time to strike $t_{\mathrm{strike}}\in\mathbb{R}$	✓	✓
Joint positions/velocities $\mathbf{q},\dot{\mathbf{q}}\in\mathbb{R}^{29}$	✓	✓
Last action $\mathbf{a}_{\mathrm{last}}\in\mathbb{R}^{29}$	✓	✓
Body poses $T_{\mathcal{B}}\in\mathbb{R}^{7|\mathcal{B}|}$	—	✓
Episode time left $t_{\mathrm{left}}\in\mathbb{R}$	—	✓
Reference targets $[\hat{\mathbf{q}},\hat{\dot{\mathbf{q}}}]\in\mathbb{R}^{58}$	—	✓

Table I: Observation spaces of the policy (actor) and critic; the critic receives extra privileged information during training (paper Table I).

flowchart TB
    subgraph PLAN["Model-based Planner"]
        A1["Mocap 360 Hz
ball position p_ball"] --> A2["Least-squares quadratic fit
31-sample window, buffer reset on bounce"]
        A2 --> A3["Hybrid dynamics integration
drag k, restitution C_h C_v"]
        A3 --> A4["Virtual hit plane x=-1.37 m
strike position + time"]
        A4 --> A5["Racket velocity from
desired landing p_l, flight time dt"]
    end
    subgraph CTRL["RL Whole-Body Controller 50 Hz"]
        B1["Commands: base target
racket position + velocity"] --> B2["Policy pi_WBC
29 joint position targets"]
        B2 --> B3["PD control
human-like forehand / backhand"]
    end
    A5 --> B1
    B3 --> C1["Strike, balance recovery,
ready for next rally"]
    C1 -->|"next ball"| A1

Fig. 2: HITTER's hierarchical loop (drawn from the paper's method): the model-based planner closes the perception-to-strike-plan loop, and the RL whole-body controller converts the plan into 29-joint human-like motion, iterating across rallies.

Experiments and Results

Planner Accuracy

Prediction accuracy is measured over 20 collected ball trajectories (Fig. 3). Errors decay as the strike approaches, hitting zero at contact. Quantitatively: 0.5 s before the strike, the position error falls below the critical 7.5 cm threshold — the racket's radius; 0.3 s out, the time error drops below 20 ms, one control step of the policy; at 0.1 s both errors are minimal. This low error across the whole horizon provides a stable command interface for the controller.

Fig. 3: Prediction errors of the model-based planner over 20 ball trajectories. Top: striking position error with the 7.5 cm racket-radius threshold in red dashed. Bottom: striking time error. Shaded regions indicate standard deviation.

Controller Agility

Agility is quantified by relating the initial distance between the actual base position and the commanded target $\hat{\mathbf{p}}_{\mathrm{base}}$ to the time needed to arrive within 1 cm. Across 1000 simulated roll-outs, 57 fail the 1 cm threshold, leaving 943 valid points and a 94.3% success rate. When the initial distance is under 0.75 m, nearly all trials converge within 0.8 s — shorter than the 0.86 s from command issuance to strike, so the robot reaches its position before the swing completes. Convergence time grows monotonically with distance and is symmetric left/right. In the real world, this shows up as a single quick step crossing the table when the ball is directed to the far side (Fig. 4); with velocity-command WBC baselines, the robot instead performs slower multi-step lateral motion.

Fig. 4: Agility evaluation. 943 successful simulated trials (94.3%); within 0.75 m the target is reached in under 0.8 s on average, faster than the 0.86 s strike time.

Fig. 5: Real-world rapid reaching: the robot transitions from the right side of the table to the left with a single step, maintaining balance and striking the ball.

Fig. 6: Real-world human-like striking: training with human motion references produces coordinated waist rotation during hits, mimicking how humans play.

Real-World Return Performance and Rallies

For deployment evaluation, 26 balls are thrown at the robot with projections spanning a wide area of the virtual hit plane. The robot achieves 24 successful returns, misses 1 return after contact, and completely misses 1 ball — a 96.2% hit rate and 92.3% return rate (Fig. 7). The robot also tends to use forehand strokes for incoming balls at $y<0$ and backhand for $y>0$, consistent with human play.

Metric	Value
Incoming balls	26
Successful returns	24
Hit rate	96.2%
Return rate	92.3%
Max rally vs human	106 consecutive shots
Reaction time to smashes	0.42 s (opponent hit → robot return)

Table 1: Real-world return performance, compiled from Sec. VI-C.

Fig. 7: Return performance on the virtual hit plane (each block side 0.2 m). Blue/green: forehand/backhand returns (24); orange: hit without return (1); red: complete miss (1).

Against human opponents, rallies reach up to 106 consecutive shots — the rally only ends when the humanoid hits the net. This exceeds casual human play, indicating reliable tracking and balance readiness across successive strokes; the humanoid even returns human smashes with just 0.42 s of reaction time. When two humanoids run the same policy against each other, they sustain continuous rallies in a fully autonomous match setting.

System Design Rationale

The discussion frames the architecture as two orthogonal combinations: planner + controller, and model-based + learning-based. The first modularizes long-horizon prediction away from short-horizon whole-body control, letting each module be evaluated and improved independently (planner accuracy and controller agility are measured separately). The second combines complementary strengths: in table tennis, rewards are sparse and delayed, so end-to-end RL struggles with exploration and sample efficiency; purely model-based approaches demand dynamics and perception models too accurate for a many-DoF humanoid with frequent ground contacts. The hierarchical design bridges the gap, improving sample efficiency and robustness to perception errors.

Limitations

• (author-stated) Virtual hitting plane. The system assumes a fixed hitting plane at the end of the table, constraining strike strategies and reducing effectiveness against very short or deep balls; a human opponent must avoid overly short shots the robot cannot reach. Relaxing the assumption would enable more diverse contact points and better table coverage.
• (author-stated) External motion capture. Ball position and robot base pose come from a mocap system, restricting deployment to controlled environments; vision-based sensing would remove this dependency.
• (author-stated) Spin handling and stroke repertoire. The system assumes negligible spin and returns with a flat push. Professional play involves heavy spin and diverse strokes (top-spin loops, back-spin chops, side-spin blocks); perceiving spin and generating counter-strokes remains open.
• (reader assessment) No autonomous serving. In both humanoid–human and humanoid–humanoid experiments, a human must initiate the rally; serving is listed as future work rather than a supported capability.

Summary and Outlook

HITTER demonstrates that real-world humanoid table tennis is feasible by splitting the problem along its natural seam: the physically well-structured parts (ball flight, bounce, and impact) are handled by an identified hybrid dynamics model, while the hard-to-model part (agile, human-like whole-body striking) is left to RL trained with only two human motion references. The numbers support the design: sub-7.5 cm position prediction 0.5 s ahead, sub-0.8 s base repositioning within 0.75 m, a 96.2% hit rate over 26 diverse balls, 106-shot rallies with a human, and 0.42 s reactions to smashes. The authors sketch three directions: multi-agent training plus autonomous serving, and learning to play against skilled opponents through strategic and tactical adaptation, building toward championship-quality humanoid players.

Golden Quote

"These results demonstrate real-world humanoid table tennis with sub-second reactive control, marking a step toward agile and interactive humanoid behaviors."

— HITTER, Abstract