Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

Paper Metadata

Title: Motion-Omni: End-to-End Joint Speech and Full-Body Motion for Spoken Dialogue

Authors: Chengqian Ma (Peking University), Wei Tao (LIGHTSPEED), Haoyu Zhang (The Chinese University of Hong Kong, Shenzhen), Yiwen Guo (Independent Researcher)

Links: arXiv:2609.04250 (https://arxiv.org/abs/2609.04250, v1, 28 August 2026, primary category cs.SD); project page https://step-out.github.io/Motion-Omni-Page/

Code and data status: the repository https://github.com/step-out/Motion-Omni exists but at the time of this write-up contains only a README, with the code release planned for November 2026, so it counts as pending. The Stage 3 teacher-pseudo-labelled corpus is public on HuggingFace (https://huggingface.co/datasets/ChengqianMa/Motion-Omni, 422,856 pairs / 1,402 hours), and the SwDA-500 evaluation set together with the full evaluation protocol is released.

Figure 1: the Motion-Omni framework and the conditioning topology of its four components (paper Figure 1). The framework prescribes only the input/output interface and the conditioning connections of each component; the concrete model class, parameter count and tokenizer are left to the instance. The reference instance reported here is Motion-Omni-Q7.

One-Line Summary

Motion-Omni moves full-body motion from a downstream module that runs over finished audio to a native output of a spoken dialogue model: four part-specific decoders read the Speech Generator's hidden states and emit 30 Hz VQ codes in parallel. The price is that the motion loss must be allowed to update the speech pathway; the payoff is a 4.32 s response (RTF 0.78, faster than real time), 5.4x quicker than the same-audio teacher cascade, with reference-free motion metrics within about 2%.

Background and Motivation

The paper opens by fixing terminology. A spoken motion model is one that, conditioned on the same dialogue context, produces both a spoken response and the full-body co-speech motion that accompanies it. The authors also delimit scope explicitly: motion here means communicative co-speech movement, and locomotion, dance, sports and generic action generation are out of scope. That narrowing matters, because it turns the problem from "produce plausible human movement" into "produce body movement coupled in rhythm and meaning to the words currently being spoken".

The two relevant literatures have each advanced quickly but expose incompatible interfaces. On the dialogue side, SpeechGPT popularised discrete audio tokens as a language-model vocabulary, LLaMA-Omni added a speech-unit decoder plus the InstructS2S-200K corpus that this paper builds on, GLM-4-Voice contributed the 12.5 Hz discrete speech tokenizer, Qwen2.5-Omni introduced Thinker-Talker, and Moshi achieved full-duplex interaction. On the motion side, TalkShow, EMAGE, MambaTalk and GestureLSM extended the recipe from face to 3D full body, and the Language of Motion (LOM) represents the body with four part-specific VQ-VAE codebooks covering face, hand, upper body and lower body. Spoken dialogue models produce speech but no motion; co-speech motion models produce motion only from audio handed to them by someone else.

The standard remedy is a cascade: the dialogue model speaks the whole response, and the finished audio is then passed to a motion model. This does yield both outputs, and the paper uses it as the baseline throughout, but the authors identify two structural costs. The first is latency: the motion model is a second complete inference pass that only starts once the audio is done. The second is more fundamental: no motion objective can ever update the speech or dialogue parameters, so there is no channel for joint optimisation between the two, and motion is condemned to follow audio that has already been decided.

The research question is therefore stated plainly: can both costs be removed at once, that is, can full-body motion be a native output of a spoken dialogue model, generated from the same states that produce the speech, without giving up motion quality or spoken-dialogue ability? Existing spoken motion models relax only part of this recipe. SOLAMI jointly predicts speech and body/hand tokens on an AnyGPT/LLaMA2 backbone but generates facial animation post hoc with an audio-to-face model. U-Mind uses one shared autoregressive backbone for response text, acoustic tokens and SMPL-X pose tokens, but does not specify native facial-expression output and does not isolate how motion supervision affects speech-generation states. Ex-Omni jointly optimises an LLM, a Speech Generator and a 52-dimensional ARKit facial decoder, yet addresses only the face. ViBES combines a frozen GLM-4-Voice speech expert with trainable face and body experts; freezing preserves the speech model but prevents motion gradients from updating it. MIBURI causally generates full-body gestures and facial expressions from Moshi's streaming states but does not report motion-loss adaptation of Moshi. The distinctive combination claimed here is explicit face, hand, upper-body and lower-body output together with end-to-end co-adaptation of three distinct modules, supported by a model-agnostic pseudo-labelling curriculum.

Three obstacles stand in the way. First, making motion native requires jointly optimising speech and motion objectives without degrading either: the two outputs live at heterogeneous rates (12.5 Hz speech units against 30 Hz motion) and share parameters through which the two losses can interfere, so the architecture must bridge the rate mismatch and the training procedure must stop motion gradients from damaging the dialogue pathway. Second, joint training needs supervision at a scale no existing corpus provides. Many spoken dialogue models are trained towards a consistent target voice, so the motion supervision must be paired with responses in that same voice; captured audiovisual corpora such as Seamless Interaction record many speakers and cannot supply this at scale. Third, to the authors' knowledge no public benchmark is designed for stochastic open-ended full-body spoken dialogue: existing full-body motion benchmarks assume fixed supplied speech, and spoken-dialogue evaluations with controlled generated speech cover facial animation only. Neither checks whether a model's own valid but unpredictable spoken response comes with appropriate whole-body motion.

Preliminaries

Three ingredients are needed to read the method. The first is the LOM part decomposition: the body state is not regressed as one continuous parameter vector but split by SMPL-X plus FLAME into four streams, each quantised to discrete codes by its own VQ-VAE. Table 6 of the paper gives the exact composition — face 106 dimensions (SMPL-X jaw 1x6D rotation plus 100 FLAME expression coefficients), hand 180 (30 hand joints x 6D), upper body 78 (13 upper-body joints x 6D), lower body 61 (9 lower-body joints x 6D plus 3D translation plus 4D foot contact), 425 in total. The four VQ-VAEs share a codebook size of $|\mathcal{C}|=256$ and are taken verbatim from the LOM emage_vq release (architecture VQVAEConvZero, 30 Hz, no temporal downsampling); this paper neither modifies nor fine-tunes them. Axis-angle joint rotations are converted to 6D rotation representations before concatenation.

The second ingredient is the GLM-4-Voice discrete speech unit. What the Speech Generator emits autoregressively is neither a mel spectrogram nor a waveform but a sequence of 12.5 Hz speech units over a vocabulary of 16,384 units plus three control tokens. At inference these units go through the CosyVoice chunk-aware flow-matching decoder into mel spectrograms and then through a HiFi-GAN-style vocoder into 22.05 kHz waveforms, while motion codes are decoded by the frozen LOM VQ-VAE back into SMPL-X body and hand parameters and FLAME expression coefficients.

The third is TQGF (Token-as-Query Gated Fusion), a fusion operator with learned head-wise sigmoid gates whose job is to let un-contextualised token embeddings query contextualised hidden states. Its key asymmetry is that query and key/value carry different information: the query holds one embedding per token, identifying what is being said at that instant, whereas the keys and values additionally carry speaker timbre, prosodic intent and dialogue history, some of which is useless for driving body movement. The gate selects what each part actually needs.

Method

Four components and the conditioning topology. The framework specifies a Speech Projector, an LLM Backbone, a Speech Generator and a Part-Aware Motion Generator, which together process the user's speech or text input and autoregressively generate a spoken response with synchronised full-body co-speech motion. For each component the framework prescribes only the interface and the conditioning topology, not the model. The choices made by Motion-Omni-Q7 are: a frozen Whisper-large-v3 encoder (hidden dimension 1280) mapping a 16 kHz waveform to continuous representations; a speech projector that concatenates every five consecutive frames and passes them through a two-layer MLP into the LLM embedding space, downsampling the sequence fivefold, at roughly 5M parameters; a Qwen2.5-7B-Instruct backbone where projected Whisper features replace a designated <speech> placeholder in the token sequence; and a Qwen2-style Speech Generator initialised from Qwen2.5-0.5B-Instruct, running at width $d=896$ while the backbone emits 3584-dimensional states, joined by a two-layer GELU-MLP and a reshape.

TQGF on the speech side. Because the widths differ fourfold, a reshape trick performs the up-projection and the temporal lengthening at once: the MLP widens each hidden state to $4d$ dimensions and the reshape then reads that single vector as four consecutive $d$-dimensional positions, so the conditioning stream $\mathbf{Z}$ is four times as long as the text sequence. This is Equation (4):

$$\mathbf{Z}=\mathrm{Reshape}\big(\mathrm{MLP}(\mathbf{H}_{\text{llm}})\big)\in\mathbb{R}^{4T_{t}\times d},\qquad \mathbf{Q}=\mathbf{E}_{\text{sg}}[\mathbf{w}]\in\mathbb{R}^{T_{t}\times d}$$

Here $\mathbf{w}$ are the $T_{t}$ text tokens the LLM has already generated for its response and $\mathbf{H}_{\text{llm}}\in\mathbb{R}^{T_{t}\times 3584}$ are the backbone's last-layer hidden states at those same positions. $\mathbf{Z}$ supplies both the keys and the values of the block, while the query stream looks the same tokens up in the Speech Generator's own embedding table $\mathbf{E}_{\text{sg}}$, giving one un-contextualised vector per token. Two TQGF layers then inject $\mathbf{Z}$ into the query stream; writing $\tilde{\mathbf{Q}}$ and $\tilde{\mathbf{Z}}$ for the layer-normalised inputs, one layer computes Equations (5) and (6):

$$\mathbf{C}=\Big(\sigma(\tilde{\mathbf{Q}}\mathbf{W}_{g})\odot\mathrm{Attn}\big(\tilde{\mathbf{Q}}\mathbf{W}_{q},\,\tilde{\mathbf{Z}}\mathbf{W}_{k},\,\tilde{\mathbf{Z}}\mathbf{W}_{v}\big)\Big)\mathbf{W}_{o}$$

$$\mathbf{Q}^{\prime}=\mathbf{Q}+\mathbf{C},\qquad \mathbf{Q}^{\prime\prime}=\mathbf{Q}^{\prime}+\mathrm{FFN}(\mathbf{Q}^{\prime})$$

$\mathrm{Attn}$ is multi-head scaled dot-product attention with $h$ heads and no output projection, so $\mathrm{Attn}(\cdot)\in\mathbb{R}^{T_{t}\times hd_{h}}$; $\mathrm{FFN}$ is a position-wise feed-forward network with pre-norm and inner width $4d$; $\sigma$ is the logistic sigmoid and $\odot$ is element-wise multiplication. The projections satisfy $\mathbf{W}_{q},\mathbf{W}_{k},\mathbf{W}_{v},\mathbf{W}_{g}\in\mathbb{R}^{d\times hd_{h}}$ and $\mathbf{W}_{o}\in\mathbb{R}^{hd_{h}\times d}$, instantiated with $h=16$ heads of dimension $d_{h}=256$, so the gate acts in a 4096-dimensional space rather than the 896-dimensional working space. The authors' rationale deserves to be recorded separately: plain cross-attention would send $\mathrm{Attn}(\cdot)\mathbf{W}_{o}$ straight into the residual stream, whereas Equation (5) filters it first with $\sigma(\tilde{\mathbf{Q}}\mathbf{W}_{g})$; and because Equation (6) adds the gated branch to $\mathbf{Q}$ rather than replacing it, a closed gate falls back on the un-contextualised query instead of producing a zero-information state.

Figure 2: Motion-Omni-Q7 output on the released "positive character traits" clip (paper Figure 3(a)). Speech and motion come from the same autoregressive pass. The seven frames are the mutually most distinct upper-body poses subject to a minimum temporal gap of 10% of the clip, shown in temporal order rather than uniformly sampled.

The dual-input interface of the Motion Generator. The motion side instantiates one independent decoder per body part $b\in\{\mathrm{face},\mathrm{hand},\mathrm{upper},\mathrm{lower}\}$. Each decoder stacks $L_{\text{TQGF}}=2$ TQGF layers reusing Equations (5) and (6) but working at $d_{m}=512$, $h=8$ and $d_{h}=64$, followed by an $L_{\text{self}}=6$-layer self-attention Transformer with periodic rotary positional encoding of period $\mathcal{T}=30$, that is, one second at 30 Hz. Its two input streams are Equations (1) and (2):

$$\mathbf{Z}=\mathbf{H}_{s}\mathbf{W}_{h},\qquad \mathbf{Q}=\mathrm{Interp}\big(\mathbf{E}[\mathbf{u}]\mathbf{W}_{e}\big)$$

$\mathbf{H}_{s}$ are the Speech Generator's last-layer hidden states and $\mathbf{W}_{h}\in\mathbb{R}^{896\times d_{m}}$ projects them into the motion working dimension; $\mathbf{u}\in\{0,\dots,16383\}^{T_{s}}$ holds the speech units, $\mathbf{E}$ is a 512-dimensional look-up table initialised from the pre-trained CosyVoice flow embedding, $\mathbf{W}_{e}$ is a learned projection, and $\mathrm{Interp}$ linearly interpolates along the time axis from the 12.5 Hz speech-unit rate to the 30 Hz motion rate. That single operator is the engineering answer to the rate-mismatch problem. All four decoders read the same $\mathbf{Z}$ and $\mathbf{Q}$ and differ only in their parameters; at inference the four parts share the Speech Generator context but do not explicitly cross-condition on one another's sampled motion outputs. During training $\mathbf{u}$ are the ground-truth units of the target response; at inference they are the units the Speech Generator has just emitted.

The authors stress that conditioning on $\mathbf{H}_{s}$ in Equation (1) rather than on the decoded waveform is what removes a separate audio-to-motion stage at inference time, while giving each decoder a representation that already carries acoustic timing together with response semantics. A part-specific MLP head then produces per-frame logits over the $|\mathcal{C}|=256$ LOM codebook entries, and the self-attention block runs at motion rate under a causal mask, so each frame attends only to earlier frames.

Dimension-weighted motion loss. The four per-part cross-entropies are not summed with equal weight but weighted by the underlying feature dimensions (Equation (3)):

$$\mathcal{L}_{\text{motion}}=\sum_{b}w_{b}\,\mathcal{L}^{(b)}_{\text{CE}},\qquad w_{\text{face}}\!:\!w_{\text{hand}}\!:\!w_{\text{upper}}\!:\!w_{\text{lower}}=106\!:\!180\!:\!78\!:\!61$$

$\mathcal{L}^{(b)}_{\text{CE}}$ is the per-frame cross-entropy for part $b$ with label smoothing 0.1, and the weights are normalised to sum to one as $(106,180,78,61)/425$, proportional to the feature dimensions in Table 6. This ratio is a design decision rather than a tuned hyperparameter: it makes every VQ-code prediction carry equal importance per feature dimension, which otherwise would let the 180-dimensional hand dominate the 61-dimensional lower body.

Joint loss composition. Stage 4 is multi-task, and Appendix A.6 gives the per-batch total loss as Equation (7). Writing $\mathcal{L}_{\text{LM}}$ for the LLM next-token cross-entropy, $\mathcal{L}_{\text{sp}}$ for the label-smoothed speech-unit cross-entropy and $\mathcal{L}_{\text{mo},b}$ for the motion cross-entropy of part $b$:

$$\mathcal{L}=\mathcal{L}_{\text{LM}}+\mathcal{L}_{\text{sp}}\cdot\mathbb{1}[\text{speech task}]+\sum_{b}w_{b}\,\mathcal{L}_{\text{mo},b}\cdot\mathbb{1}[\text{motion task}]$$

The masks select supervision per sample: ASR contributes only $\mathcal{L}_{\text{LM}}$; TTS, TTSM and S2SM contribute $\mathcal{L}_{\text{LM}}+\mathcal{L}_{\text{sp}}$; TTSM and S2SM additionally contribute the motion sum; text-only T2T contributes only $\mathcal{L}_{\text{LM}}$. The Speech Projector has no dedicated loss term — it is trained through $\mathcal{L}_{\text{LM}}$ by backpropagation whenever the input contains audio, since the projected Whisper features feed the LLM and the gradient flows back through the projector. When logging training curves, each per-task loss is averaged only over batches containing at least one sample of that task, so that zero-padding from other tasks does not artificially depress the reported value.

Four-stage progressive curriculum. The Whisper encoder and the LOM VQ-VAE stay frozen throughout. Stage 1 trains only the speech projector with ASR supervision: each sample is a <speech> placeholder followed by its transcript target, projected features replace the placeholder, labels at the inserted speech positions are masked, and next-token cross-entropy applies only to transcript tokens. Because the LLM is frozen, the objective trains the projector to produce representations from which the LLM can decode the transcript. Stage 2 trains the Speech Generator on TTS-style pairs with the LLM still frozen: the text-side instruction asks the model to say or restate a supplied sentence, the LLM produces a semantically consistent response representation, and the target is the corresponding sequence of discrete speech units.

Stage 3 attaches the Motion Generator and trains it jointly with the Speech Generator while keeping the LLM and projector frozen, on the TTSM corpus from the data construction pipeline, consumed under a four-substage curriculum exposing progressively larger quality quantiles (12.5% / 25% / 50% / 100%), each warm-started from the previous checkpoint. A failure record here is central: in an initial pilot that kept the Speech Generator frozen, the motion loss plateaued above the level reached by joint training and the rendered motions were clearly misaligned with the audio. Jointly updating the Speech Generator lowered the loss further and improved audio/motion alignment, so it is used throughout Stages 3a-3d. This is the empirical basis for the abstract's claim that joint training is not optional. Stage 4 unfreezes the LLM backbone, speech projector, Speech Generator and Motion Generator simultaneously and optimises a four-task ASR/TTS/S2SM/T2T mixture with a constant-with-warmup schedule, warm-started from the final Stage 3d checkpoint. The T2T component of about 213K samples comes from six text-only datasets (SODA, WildChat, Tulu-3, WizardLM, OpenThoughts, NuminaMath-CoT) and preserves general text-response planning and multi-turn dialogue behaviour during joint fine-tuning; without it, responses more often repeated the user's input.

Data construction pipeline. Training a spoken motion model needs large-scale supervision pairing a dialogue response with its accompanying motion, and captured audiovisual corpora with many speaker voices cannot match the single target voice the Speech Generator produces. The solution is pseudo-labelling in two steps. Step one is teacher motion supervision: a pre-trained co-speech motion teacher (LOM in this run) is executed on every response waveform, and its four per-part VQ code streams form the motion target. The teacher is a replaceable component, so a future implementation can regenerate supervision with a stronger one. Step two is dual-metric quality scoring and curriculum: every sample is scored on a weighted VQ-VAE reconstruction error $L_{1}^{\text{vq}}$ that flags motion outside the codebook's expressive range, and on a beat correlation BC score that flags teacher motion weakly coupled to the speech. After robust 5th-95th percentile normalisation each sample receives the combined score

$$s(x)=\alpha\big(1-\widetilde{L_{1}^{\text{vq}}}\big)+(1-\alpha)\widetilde{\mathrm{BC}},\qquad \alpha=0.5$$

which drives the four-substage Stage 3 curriculum and selects the Stage 4 S2SM pool. Both metrics are defined in the appendices. For $L_{1}^{\text{vq}}$, features are extracted per the Table 6 decomposition, each part sequence $\hat{m}^{(b)}_{1:T}\in\mathbb{R}^{T\times D_{b}}$ is passed through the frozen VQ-VAE encoder-quantise-decoder chain to obtain $\tilde{m}^{(b)}$, and then Equations (8) and (9) give the per-part L1 error and its dimension-weighted aggregation:

$$\ell_{b}=\frac{1}{T\cdot D_{b}}\sum_{t=1}^{T}\bigl\lVert\hat{m}^{(b)}_{t}-\tilde{m}^{(b)}_{t}\bigr\rVert_{1},\qquad L_{1}^{\text{vq}}(\hat{m})=\frac{106\,\ell_{\text{face}}+180\,\ell_{\text{hand}}+78\,\ell_{\text{upper}}+61\,\ell_{\text{lower}}}{425}$$

It measures supervision fidelity: a high $L_{1}^{\text{vq}}$ means quantisation mapped the latent to a distant codebook entry, so the code assigned by the teacher is a poor proxy for the intended motion and training on such samples injects noisy labels. A low value alone does not guarantee useful supervision, however, since the codebook may faithfully represent a motion that is poorly aligned with the driving speech — exactly the failure mode BC catches. Together they form a dual filter, one for label fidelity and one for speech-motion coupling. BC (Equation (10)) first extracts a body-level kinematic envelope $v_{t}=\lVert\dot{X}_{\text{upper},t}\rVert_{2}$ at 30 Hz, detects kinematic beats $\mathcal{M}=\{t: v_{t}<v_{t-1},\,v_{t}<v_{t+1},\,v_{t}>0.3\bar{v}\}$ as local minima above a relative threshold, extracts audio onsets $\mathcal{O}$ with librosa, and computes the forward audio-to-motion Gaussian-kernel beat distance:

$$\mathrm{BC}(a,\hat{m})=\frac{1}{|\mathcal{O}|}\sum_{t_{o}\in\mathcal{O}}\exp\!\left(-\frac{\min_{t_{m}\in\mathcal{M}}(t_{o}-t_{m})^{2}}{2\sigma^{2}}\right),\qquad \sigma=9\ \text{frames}$$

The paper additionally defines a bidirectional diagnostic $\mathrm{BC}_{\text{bi}}=\tfrac{1}{2}(\mathrm{BC}_{a\to m}+\mathrm{BC}_{m\to a})$ in Equations (11) to (13), and honestly demotes it to an exploratory measure: compared against the majority rhythm verdict on 100 human-evaluated pairs, with score differences below 0.07 treated as ties, $\mathrm{BC}_{\text{bi}}$ reaches 46% agreement against 40% for standard BC, and at this sample size neither separates from the chance reference.

The topology as a whole. Assembling the pieces above gives the topology of Figure 1, redrawn below from the paper's actual connections. Note that the motion generator's keys and values come from the Speech Generator rather than the LLM, which is precisely the choice tested by the Appendix G.1 ablation:

flowchart TD
  U[User speech 16kHz or text] --> W[Frozen Whisper-large-v3 encoder
hidden dim 1280]
  W --> P[Speech Projector
concat 5 frames plus 2-layer MLP
5x downsample, about 5M params]
  P -->|replaces speech placeholder| L[LLM Backbone
Qwen2.5-7B-Instruct
frozen Stages 1-3, fine-tuned Stage 4]
  T[Text tokens] --> L
  L -->|H_llm in R^(T_t x 3584)| G4[Eq 4: MLP to 4d then Reshape
Z in R^(4T_t x d), d=896]
  L -->|same tokens| Q4[Q = E_sg of w
un-contextualised query]
  G4 --> TQ1[TQGF x2, h=16, d_h=256
Eq 5 gated fusion, Eq 6 residual]
  Q4 --> TQ1
  TQ1 --> S[Speech Generator
Qwen2.5-0.5B init
GLM-4-Voice units 12.5Hz, vocab 16384]
  S -->|speech units u| D1[CosyVoice flow-matching decoder
plus HiFi-GAN vocoder to 22.05kHz]
  S -->|H_s last-layer hidden states
Eq 1: Z = H_s W_h, K and V| MG[Motion Generator
4 parallel part decoders
d_m=512, h=8, 2 TQGF plus 6 self-attn
periodic RoPE T=30]
  S -->|emitted units u
Eq 2: Q = Interp of E u W_e
12.5Hz to 30Hz| MG
  MG --> VQ[LOM VQ codes at 30Hz
face 106 / hand 180 / upper 78 / lower 61]
  VQ -->|frozen VQ-VAE decode| SM[SMPL-X body and hand params
plus FLAME expression coefficients]
  MG -.->|Stage 3 motion CE Eq 3
weights 106:180:78:61| S
  S -.->|Stage 4 joint loss Eq 7| L

The two dashed edges are the training-time gradient paths, and they mark the dividing line from frozen-speech designs such as ViBES: the motion loss updates the speech pathway through the Speech Generator in Stage 3, and in Stage 4 it reaches all the way back into the LLM backbone. Total parameters are about 8.3B, split as roughly 7.6B for the LLM backbone, 0.5B for the Speech Generator, 150M for the four part decoders and 5M for the speech projector. The four stages take about 960 GPU-hours in total on 4 GPUs with DeepSpeed ZeRO-2, BF16 mixed precision and gradient checkpointing; the effective batch size is 128 in every stage, and Stage 4 tightens gradient clipping to 0.5 with a three-layer NaN guard.

Experiments

Evaluation set and setup. Speech and motion evaluation uses SwDA-500, an external 500-prompt dialogue-text set derived from the Switchboard Dialog Act Corpus, covering all 66 SwDA topic descriptions with 7 or 8 semantically complete speaker turns per topic, keeping prompts of moderate length and removing transcription artefacts not intended to be spoken. The authors state clearly that SwDA-500 provides real conversational wording but is not a paired real-motion benchmark: every motion reference below is teacher-generated or baseline-generated under matched prompts. On data scale, Table 1 reports Stage 1 ASR at 1,572,119 samples / 4,295.8 hours, Stage 2 TTS at 1,642,715 / 4,517.0, Stage 3 TTSM at 422,856 / 1,402.3 and the Stage 4 mixture at 724,508 / 1,929.4, totalling 4,362,198 samples and 12,144.6 hours. All sources are English-filtered with langdetect and capped at 10,000 characters per dialogue.

Does motion training damage dialogue ability? This is the first question any joint-training proposal must answer. On Seed-TTS-Eval test-en (1,088 samples, corpus-level WER computed by Whisper-large-v3), Motion-Omni-Q7 reaches 2.62%, the lowest among the omni-modal LLMs compared. The authors also keep dedicated TTS systems in a separate ranking, since those only synthesise a supplied sentence whereas omni-modal LLMs must additionally plan the response.

Model	Category	WER (%) test-en
Human	Reference	2.14
Qwen3-TTS-12Hz-1.7B	Dedicated TTS	1.24
CosyVoice 3-1.5B RL	Dedicated TTS	1.45
F5-TTS	Dedicated TTS	2.04
CosyVoice 2	Dedicated TTS	2.57
Qwen2.5-Omni	Omni-modal LLM	2.72
Ex-Omni	Omni-modal LLM	2.67
Motion-Omni-Q7 (ours)	Omni-modal LLM	2.62

Table 1: WER on Seed-TTS-Eval test-en (excerpt of paper Table 2). On the UTMOSv2 naturalness proxy the model scores 3.77, above GLM-TTS (3.31), VoxCPM1.5 (3.17), F5-TTS (3.15) and CosyVoice (3.06), below Qwen3-TTS (4.05) and CosyVoice 3 (3.92). On VoiceBench the nine-subset Overall is 47.63, above LLaMA-Omni (41.12), Ex-Omni (43.57), Mini-Omni2 (33.49) and Moshi (29.51).

Motion, facial geometry and lip synchronisation. The comparison arms are controlled cascades named speech source plus audio-to-motion model. Rows prefixed MO-audio use this model's own audio, isolating the motion pathway under identical speech; rows prefixed Qwen2.5-Omni attach the same motion generators to a stronger external speech model; LOM rows are teacher-reference cascades because LOM also supplies the reference distribution for teacher-based FGD. Metrics fall into two classes: reference-free ones (Diversity, BC, LSE-C, LSE-D) need no motion reference, while teacher-referenced ones (FGD, MSE, LVD) use LOM-generated motion as a pseudo-reference and therefore favour systems that invoke LOM at inference time.

System	Div.	BC	LSE-C	LSE-D	FGD	MSE	LVD
Qwen2.5-Omni + MambaTalk	11.92	7.17	6.853	7.920	4.08	7.092	6.851
MO-audio + MambaTalk	12.07	7.03	6.797	7.955	4.15	7.128	7.019
Qwen2.5-Omni + GestureLSM	13.61	7.44	-	-	3.29	-	-
MO-audio + GestureLSM	13.55	7.49	-	-	3.17	-	-
Qwen2.5-Omni + EMAGE	11.67	7.31	6.578	8.151	3.75	7.553	7.205
MO-audio + EMAGE	10.95	7.32	6.643	8.094	3.29	7.627	7.291
Qwen2.5-Omni + LOM	13.82	7.56	7.136	7.128	-	6.581	6.243
MO-audio + LOM	13.07	7.67	6.985	7.011	-	-	-
Motion-Omni-Q7 (ours)	13.67	7.59	7.011	7.023	3.03	6.723	6.427

Table 2: motion, facial geometry and lip-sync metrics on SwDA-500 (paper Table 3). FGD and BC are reported in units of 10^-1 and Diversity is unscaled. The proposed model places first or second on seven of the eight metrics; the rows that surpass it are the LOM teacher-reference cascades, which invoke at motion-inference time the same model that supplied their training targets.

The table has to be read by metric class. Among systems that do not run LOM at motion-inference time, Motion-Omni-Q7 obtains the highest beat correlation (7.59), the highest diversity (13.67) and the best score on every facial and lip-sync metric. Its teacher-reference FGD of 3.03 is also the lowest, but that metric measures distance to the LOM-generated distribution by construction, so its meaning is limited. The reference-free gap to the same-audio teacher cascade is about 1%, and that small gap is what supports the architectural choice: conditioning motion on Speech Generator hidden states rather than on the waveform costs no visible motion quality.

Figure 3: first row of the cross-system comparison, Motion-Omni-Q7 (paper Figure 4(a)). For this comparison the authors deliberately avoid selecting frames at which their own model happens to move; instead seven evenly spaced timestamps are sampled over the middle of the clip and the same timestamps are applied to every system.

Figure 4: second row, MO-audio + LOM, the teacher cascade (paper Figure 4(b)). Same timestamps, same renderer, same avatar and crop box; every tile is cropped with one shared box computed as the union of avatar bounding boxes over all frames, so differences between tiles are differences in motion rather than in framing.

Figure 5: third row, MO-audio + GestureLSM (paper Figure 4(c)).

Figure 6: fourth row, MO-audio + EMAGE (paper Figure 4(d)). In the human study this is exactly the arm against which the proposed model gains its largest margin.

Human evaluation. Four trained young-adult male annotators (mean age about 26, all fluent in English and experienced in evaluating motion-generation or spoken-dialogue outputs) independently rated 25 paired clips for each of the four arms. All received the same rubric beforehand, the interface hid system names and randomised A/B order, and each pair was judged from rendered video plus audio on three motion dimensions: R1 rhythm, R2 semantic alignment and R3 body naturalness, with per-pair verdicts by majority vote. The two MO-audio arms hold speech fixed, so differences there reflect the motion pathway alone: against MO-audio + EMAGE the pooled margin is +25 (45 wins, 10 ties, 20 losses), while against MO-audio + LOM, that is against the teacher itself, the outcome is 25 wins, 27 ties and 23 losses, with ties the single most frequent verdict. The authors are restrained about scale: with four annotators and 25 pairs per arm they do not read the smaller margins as evidence of a reliable preference. They attribute the R3 advantage over EMAGE to a domain mismatch in the baselines' training data — the BEAT2 corpus used by LOM and EMAGE contains monologue and public-speaking recordings as well as conversational clips, so their generated motion sometimes exhibits presentation-style behaviours such as pacing, turning sideways and exaggerated arm swings, which annotators perceive as unnatural in a dialogue setting. Motion-Omni is trained on dialogue-paired pseudo-motion, so its style is calibrated to a conversational register.

Latency. Total response time $T_{\text{resp}}$ is the wall-clock seconds from receiving the user audio to completing all audio and motion generation, and the real-time factor is $\mathrm{RTF}=T_{\text{resp}}/T_{\text{user}}$ where $T_{\text{user}}$ is the user-utterance duration; $\mathrm{RTF}<1$ means the system responds faster than real time. All measurements use SwDA-500 on a single GPU and evaluate offline response generation.

Model	$T_{\text{resp}}$ (s)	RTF	Slowdown
Qwen2.5-Omni + LOM	99.08	18.34	22.9x
Qwen2.5-Omni + EMAGE	48.65	8.79	11.3x
MO-audio + LOM	23.35	4.39	5.4x
MO-audio + EMAGE	4.63	0.84	1.1x
Motion-Omni-Q7 (ours)	4.32	0.78	1.0x

Table 3: response latency on SwDA-500 (paper Table 5). The matched comparison is MO-audio + LOM, which uses the same speech model and the teacher motion model but invokes the latter as a separate audio-to-motion stage, and is consequently 5.4x slower.

Paper Figure 2 places these systems on the quality and latency axes together, and the conclusion is that two cascades each come close on one axis only, with no system achieving both: MO-audio + EMAGE matches the response time but gives up 0.27 BC, while MO-audio + LOM exceeds BC by 0.08 at 5.4x the response time. These numbers are the hardest support for the claim that removing the separate audio-to-motion stage is what buys the speed.

Two ablations. The first isolates the fusion design: all variants start from the same Stage 2 checkpoint and use the same Stage 3a data and optimisation schedule, with the only changed factor being how the motion query incorporates contextual information. TQGF with Speech Generator K/V reaches the best BC of 0.912, plain cross-attention with Speech Generator K/V reaches 0.905, FiLM 0.852, TQGF with LLM K/V only 0.827, concatenation 0.809 and GLU 0.799. The authors draw only two scoped conclusions: under the same TQGF operator, Speech Generator context is more useful than direct LLM context (variant 6 versus 1), and gated fusion is modestly better than plain cross-attention in this fixed Stage 3a setup (variant 6 versus 4). They state explicitly that the experiment does not isolate every internal component of TQGF.

The second ablation is stage-wise teacher-reference FGD: 0.3974 at Stage 3a, 0.3258 at 3b, 0.3079 at 3c, 0.3075 at 3d and 0.3040 after Stage 4 joint fine-tuning. It decreases monotonically through the curriculum and remains lowest after Stage 4, which is direct evidence that the quantile-based curriculum works and that unfreezing the LLM in Stage 4 does not come at the cost of motion quality.

Limitations

The Limitations section is candid, and the first item is the most fundamental: motion quality is bounded by the LOM VQ-VAE codebook and by the teacher-generated pseudo-labels used in this instantiation, so motion outside that distribution cannot be expressed. This is not merely a data-quality complaint but a structural ceiling — the model's motion vocabulary is those four 256-entry codebooks, and if the teacher cannot supply better targets, the student will not exceed the teacher. Related to this, only the Motion-Omni-Q7 instance has been trained; stronger backbones, larger motion teachers and continuous motion heads are left to future work. The framework claims that all components are swappable, but that swappability currently rests on architectural argument rather than on a second trained instance.

The third limitation is that the model does not condition on explicit speaker identity or emotion, and the training data is entirely English, restricting generalisation to other languages and motion cultures; the ethics statement repeats that the learned motion style reflects the conversational register of English data and should not be assumed to transfer. The fourth is that Motion-Omni-Q7 ingests the user utterance fully before emitting the first response token, making it an offline response generator rather than a streaming interaction model; reaching the low-latency interactive behaviour of streaming systems such as MIBURI requires a different design and evaluation protocol. For digital-human deployment this matters most: RTF 0.78 means the whole response is generated faster than real time, not that the first visible motion arrives 0.78 seconds after the user stops speaking.

The fifth concerns evaluation itself. The authors concede that FGD and BC are imperfect proxies for perceived naturalness, that the four-annotator A/B/tie comparison is exploratory, and that the absence of a larger non-author human study remains a limitation. More instructive is the negative result in Appendix H. They originally intended to scale rubric scoring with a video-input LLM-as-judge (Gemini 3.1 Pro), but after five rounds of prompt iteration on a 26-clip multi-turn calibration set they abandoned it: the best fifth iteration achieves Pearson $r=+0.32$ on speech quality, $+0.18$ on R3 body naturalness and $+0.17$ on multi-turn coherence, while the remaining pilot dimensions including R1 and R2 have $r\leq 0$, and the average across the pilot rubric is only $+0.05$, an order of magnitude below the 0.50-0.70 band typically reported for LLM-as-judge on text dialogue benchmarks. Three independent root causes are identified: a 26-clip 1-5 integer scale gives a per-dimension confidence interval of roughly plus or minus 0.4 that prompt iteration cannot escape; the judge model has hard ceilings, including indistinguishable lip-sync from preview-quality video and a systematic R3 over-rating of large-amplitude motion that humans read as over-presented; and R2 involves a genuine disagreement among human raters about whether emphasis-time gestures count as semantic alignment, which no judge prompt can resolve. Following the LMSYS Arena recipe they also implemented a pairwise judge, which performed worse rather than better: on 13 pairs it agreed with the human aggregate ranking on 4 (31%) with Cohen's $\kappa=0.079$, below the simpler baseline of scoring each clip absolutely and picking the higher one at 5 of 13 (38%). The reason is clear — the two clips share the same audio, so non-motion criteria are forced to ties and the verdict is dominated by R3, exactly the dimension where the judge is most biased. The final decision was to remove the video LLM-as-judge from the main evaluation entirely and rely on human ratings only.

Two further observations are my own. First, teacher-reference FGD and the pseudo-reference MSE and LVD are all computed against a LOM-generated distribution, and LOM is also the source of the training targets, which forms a self-referential evaluation loop: the proposed model obtaining the lowest FGD inside that loop is close to a foregone conclusion. The informative signals are the four reference-free metrics and the human study, and while the authors phrase this carefully, a reader can easily retain only the headline "first or second on seven of eight metrics". Second, the Stage 3 joint update was forced by a failed pilot, yet the paper offers only a qualitative description (loss plateauing higher, rendered motion visibly misaligned) plus one FGD curve, without a full SwDA-500 metric row for the frozen-Speech-Generator arm. That ablation supports the paper's most central claim, and its evidence is thinner than the claim's importance.

Summary and Outlook

The contributions sort into three layers. Architecturally, the paper gives an end-to-end spoken motion framework in which motion is generated from the hidden states that produce speech, and motion supervision updates the LLM, Speech Generator and Motion Generator together; the 12.5 Hz to 30 Hz mismatch is bridged by a dual-input conditioning interface where keys and values come from Speech Generator hidden states and the query is the speech-unit embedding linearly interpolated up to motion rate, combined with part-specific gated fusion and dimension-weighted cross-entropy. On data, a model-agnostic pseudo-labelling route turns consistent-voice open-ended speech-instruction responses into 422,856 paired samples (1,402 hours), with a dual-metric combined score from $L_{1}^{\text{vq}}$ and BC driving a four-substage curriculum. On evaluation, SwDA-500 plus a reproducible protocol uses matched generated audio to control response content and prosody, with a shared renderer, reference-aware automatic metrics, paired human evaluation and a common latency protocol — described by the authors as the first publicly released evaluation protocol for stochastic open-ended full-body spoken dialogue.

The shape of the result is quality parity with a large speed gain: reference-free motion metrics within about 1% of the same-audio teacher cascade, response time 4.32 s (RTF 0.78) that is 5.4x faster than that cascade, speech WER of 2.62% the lowest among the omni-modal systems compared, and a VoiceBench Overall of 47.63 showing that dialogue ability survived motion training. For anyone building digital humans or virtual presenters this combination matters more than any single number, because cascade motion quality was already adequate; what blocked deployment was the latency of the second inference pass.

Looking forward, the authors frame every component as swappable, with additional instances following from re-running the recipe with stronger backbones or teachers. Four directions follow the paper's own threads: replacing LOM with a stronger motion teacher and regenerating supervision, to test whether the quality ceiling really is set by the teacher; converting the offline response generator into a streaming interaction model, which needs a new evaluation protocol and not just a new architecture; adding speaker identity and emotion conditioning, and rebuilding the TTSM pool on non-English corpora; and the evaluation methodology itself, since Appendix H shows that an off-the-shelf multimodal LLM cannot yet substitute for human annotation of rendered conversational digital-human clips, leaving a large, fine-grained human benchmark as the field's public gap.

Golden Quotes

"Conditioning motion on the Speech Generator's hidden states rather than on a decoded waveform keeps the reference-free motion metrics within about 1% of the same-audio teacher cascade while removing a separate audio-to-motion stage, which accounts for the 5.4x difference in response time." (Conclusion)

"Joint training is not optional here: with the speech pathway frozen, motion remains misaligned with the audio." (Abstract)