DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation

Paper Metadata

Field	Detail
Title	DSpark: Confidence-Scheduled Speculative Decoding with Semi-Autoregressive Generation
Authors	Xin Cheng, Xingkai Yu, Chenze Shao, Jiashi Li, Yunfan Xiong (five co-first authors) and 28 others
Affiliations	DeepSeek-AI; Peking University
arXiv	arXiv:2607.05147v1 (2026-07-06; primary cs.AI, cross-listed cs.CL)
Code	github.com/deepseek-ai/DeepSpec, MIT license, released: full data-prep / training / evaluation pipeline plus every draft checkpoint behind Table 1 for Eagle3, DFlash and DSpark (HuggingFace deepseek-ai/dspark_qwen3_{4b,8b,14b}_block7, dspark_gemma4_12b_block7)
Deployment	Live in the production serving engines of DeepSeek-V4-Flash (preview) and DeepSeek-V4-Pro (preview); replaced the MTP-1 baseline two weeks after the V4-preview release
Benchmarks	Math (GSM8K, MATH500, AIME25), code (MBPP, HumanEval, LiveCodeBench), open chat (MT-Bench, Alpaca, Arena-Hard)

Background and Motivation

Autoregressive decoding makes inference latency scale linearly with output length: every token costs a full forward pass, and the arithmetic intensity of single-token decoding is so low that the GPU spends most of its time waiting on memory. Speculative decoding is the principled fix. A lightweight drafter proposes $\gamma$ candidate tokens, the target model verifies the whole block in one parallel forward pass, rejection sampling keeps the longest prefix that agrees with the target distribution, and a bonus token tops it off. Because the acceptance rule reproduces the target distribution exactly, the speedup is not bought with output quality.

Equation (1) pins down the cost structure the rest of the paper works against:

$$L=\frac{T_{\text{draft}}+T_{\text{verify}}}{\tau}$$

where $L$ is average wall-clock latency per generated token and $\tau$ is the number of tokens accepted per round. Only three levers remain: shrink $T_{\text{draft}}$ (faster drafting), raise $\tau$ (better drafting), and cut the effective $T_{\text{verify}}$ (smarter verification). Most prior work pulls exactly one of them. DSpark pulls all three, which is why it reads as a systems paper wearing an algorithms paper's clothes.

The two drafter families trade these levers against each other. Autoregressive drafters (the EAGLE line, Medusa-style tree expansions) condition each token on the prefix actually sampled so far, which models well and yields high $\tau$, but $T_{\text{draft}}\propto\gamma$ forces them into small blocks and shallow networks. Parallel drafters (Medusa, DART, DFlash) emit all $\gamma$ positions in a single forward pass, so $T_{\text{draft}}$ is essentially block-length independent and blocks can be made large; the price is that positions are mutually independent, and acceptance decays fast along the block. The paper names this failure mode suffix decay and illustrates it concretely: when the context admits several plausible continuations ("of course" and "no problem"), an independent parallel drafter happily emits cross-mode splices such as "of problem" or "no course", because each position marginalizes over all possible predecessors instead of conditioning on the one that was really sampled. The non-autoregressive translation literature calls the same phenomenon multi-modal collision.

The second bottleneck sits on the system side, and the paper's framing there is sharper than its algorithmic framing: even with a good drafter, "verify every drafted token" is itself the waste. The ideal verification length varies along two axes. Along the data axis, structured requests such as code are intrinsically easier to accept than open-ended chat. Along the system axis, an extra verified token is nearly free at low concurrency but consumes batch capacity that other live requests could use at high concurrency. A static verification length is blind to both, so the paper argues the verification length should be a scheduling variable solved against the engine's current load, not a hyperparameter.

Together these two observations define DSpark's design goal: keep the drafting speed of a parallel backbone while buying back autoregressive dependence as cheaply as possible, then have the drafter produce a trustworthy, calibrated survival probability so verification budget is spent only on tokens with positive expected return. The authors' phrasing is "verify smarter, not longer".

Preliminaries: the DFlash Parallel Drafter Interface

DSpark instantiates its parallel backbone as DFlash, so it helps to know exactly what DFlash consumes and returns. Its central design choice is to pull rich context features out of the target model and inject them into the draft layers. During prefill, hidden states from a set of target layers $\{l_1,\ldots,l_m\}$ are concatenated and projected into the draft hidden space (Equation 2):

$$H_{\text{ctx}}=\mathrm{RMSNorm}\bigl(W_c\,[H^{(l_1)};\,\ldots;\,H^{(l_m)}]\bigr)$$

with a shared projection $W_c\in\mathbb{R}^{d\times md}$. Those context features are then concatenated with the draft block representation along the key/value sequence dimension and injected into every draft layer (Equation 3):

$$K_i=[W_i^K H_{\text{ctx}};\;W_i^K H_d],\qquad V_i=[W_i^V H_{\text{ctx}};\;W_i^V H_d]$$

All positions inside the block attend to each other bidirectionally and to the injected target context. The drafter shares the target model's embedding table and LM head, which is what makes the sequential head affordable later on. In the released configuration (config/dspark/dspark_qwen3_4b.py) the feature layers are target_layer_ids=[1,9,17,25,33], the backbone is num_draft_layers=5 deep, and block_size=7 with mask_token_id=151669 standing in for the not-yet-sampled positions of the block.

Figure 1: DSpark architecture (paper Figure 1). The parallel backbone produces hidden states and base logits for the whole block in one pass; the sequential head adds a prefix-dependent bias to those logits, and the confidence head turns the same hidden states into per-position survival probabilities that the hardware-aware prefix scheduler consumes together with the profiled engine capacity curve.

Method

Semi-autoregressive generation: split the block into a parallel stage and a sequential stage. The parallel backbone (DFlash) runs the whole block in a single forward pass and produces hidden states $h_1,\ldots,h_\gamma$ with base logits $U_1,\ldots,U_\gamma$. The sequential stage does not recompute representations; it only adds a prefix-dependent transition bias $B_k(x_0,x_{<k},x_k)$ on top of those base logits, giving the intra-block joint distribution (Equation 4):

$$P(X\mid x_0)=\prod_{k=1}^{\gamma}p_k(x_k\mid x_0,x_{<k}),\qquad p_k(v\mid x_0,x_{<k})=\frac{\exp\!\left(U_k(v)+B_k(x_0,x_{<k},v)\right)}{\sum_{u\in\mathcal{V}}\exp\!\left(U_k(u)+B_k(x_0,x_{<k},u)\right)}$$

Here $x_0$ is the anchor token left over from the previous verification round and $\mathcal{V}$ is the vocabulary. The structure of this equation is the point: the bias is added to logits before a single softmax, so every position still yields an exact softmax evaluation. The related-work section uses precisely this property to separate DSpark from CRF-NAT and CTC-drafter, which also place sequential modules over parallel hidden states but lose exact per-token probabilities to a globally normalized partition function and to latent marginalization over alignment paths respectively. Rejection sampling demands those probabilities, which is why CTC-drafter is restricted to greedy verification. Keeping the sequential correction local, in the authors' words, avoids the problem entirely. At inference time the sequential stage samples left to right from $p_k(\cdot\mid x_0,x_{<k})$.

Markov head: a low-rank factorization of first-order transitions. The simplest instantiation restricts $B_k$ to the immediately preceding token, degenerating into a first-order transition $B(x_{k-1},x_k)$. In full generality that is a $V\times V$ matrix, so the authors factorize it as $B=W_1W_2$ with $W_1\in\mathbb{R}^{V\times r}$ and $W_2\in\mathbb{R}^{r\times V}$, making the transition bias at position $k$ (Equation 5):

$$B(x_{k-1},\,\cdot\,)=W_1[x_{k-1}]\,W_2\;\in\;\mathbb{R}^{V}$$

$W_1$ acts as an embedding lookup table and $W_2$ as a logit projection. The default rank is $r=256$ (markov_rank=256 in the released config), so both storage and per-step compute stay small even with a large vocabulary. Back to the earlier example: once position 1 samples "of", the Markov head raises "course" and suppresses "problem" at position 2, and the cross-mode collision never reaches the verifier. In code this is deepspec/modeling/dspark/markov_head.py::VanillaMarkov.sample_block_tokens, a for step_idx in range(proposal_len) loop that calls apply_step_logits(base_logits[:, step_idx, :], token_ids=prev_token_ids, ...), samples, and feeds the result back as prev_token_ids for the next iteration, which is a literal transcription of Equations (4) and (5).

RNN head: an alternative with intra-block memory. The Markov head remembers nothing beyond one step, so position $k$ cannot see tokens before $x_{k-1}$. The RNN head relaxes that by carrying a recurrent state $s_k$ that accumulates the full prefix history inside the block. Each step concatenates the current state $s_{k-1}\in\mathbb{R}^r$, the previous token's embedding $W_1[x_{k-1}]\in\mathbb{R}^r$ and the backbone hidden state $h_k\in\mathbb{R}^d$ into $z_k=[s_{k-1};\,W_1[x_{k-1}];\,h_k]\in\mathbb{R}^{2r+d}$, then performs one gated update (Equation 6):

$$s_k=\sigma(W_g z_k)\odot s_{k-1}+\bigl(1-\sigma(W_g z_k)\bigr)\odot\tanh(W_c z_k),\qquad B_k(x_{<k},\,\cdot\,)=W_2^\top\tanh(W_o z_k)$$

$W_g,W_c,W_o\in\mathbb{R}^{r\times(2r+d)}$ come from a single linear layer split into gate / candidate / output thirds, and $s_0$ is initialized to zero. RNNHead._rnn_step mirrors this exactly: z = torch.cat([state, prev_embeddings, hidden_states], dim=-1), then proj = self.joint_proj(z), then chunk(3), then new_state = gate * state + (1 - gate) * candidate. The experimental verdict is that the RNN head buys only marginal gains over the Markov head, mostly on longer drafts, so the paper defaults to the Markov head given its worse implementation complexity and deployment characteristics. That negative result, a stronger sequential module not paying for itself, is one of the more informative architectural findings here.

Confidence head: it predicts conditional survival, not acceptance rate. The confidence head emits one scalar $c_k\in(0,1)$ per draft position. Its semantics are defined tightly: the probability that the draft token at position $k$ passes target verification given that every earlier token in the block was accepted. Structurally it is a linear projection followed by a sigmoid (Equation 7):

$$c_k=\sigma\bigl(w^\top[h_k;\,W_1[x_{k-1}]]\bigr)$$

The feature vector mixes the backbone hidden state $h_k$ with the previous draft token's representation from the Markov embedding table. That choice makes $c_k$ depend on the already-sampled $x_{k-1}$, which is exactly where the causality discussion in Appendix A comes from. Supervision is not a hard 0/1 label but the analytic acceptance rate $c_k^*$, computable in closed form from the total variation distance between draft and target distributions (Equation 8):

$$c_k^*=1-\tfrac{1}{2}\lVert p_k^d-p_k^t\rVert_1$$

In code, deepspec/modeling/dspark/qwen3/modeling.py::predict_confidence_step is this equation: prev_embeddings = self.markov_head.get_prev_embeddings(prev_token_ids), features = torch.cat([hidden_states, prev_embeddings], dim=-1), then self.confidence_head(features).float(). On the label side loss.py::_compute_accept_rate_3d computes 1.0 - 0.5 * (draft_probs - target_probs).abs().sum(dim=-1) clamped to $[0,1]$, matching Equation (8) term for term.

Post-hoc calibration: why a threshold heuristic is not enough and STS is. A threshold-based verification heuristic only needs confidence scores to rank draft tokens correctly. The hardware-aware scheduler in this paper needs the absolute magnitude of cumulative acceptance probability to compute the expected accepted length $\tau$; correct ranking with wrong scale distorts the throughput estimate. Neural confidence estimators are systematically overconfident, so the paper applies Sequential Temperature Scaling (STS). Because each $c_i$ is conditional, the chain rule makes the joint probability of a prefix surviving the cumulative product $\prod_{i\leqslant k}c_i$. STS calibrates that product left to right on a held-out validation set: at each position $k\in\{1,\dots,\gamma\}$ it runs a one-dimensional grid search for the temperature scalar minimizing expected calibration error of the cumulative product, while leaving all earlier positions' already-calibrated scores untouched. Temperature scaling is an order-preserving transform, so it fixes magnitude without disturbing the ranking the confidence head learned. The evaluator in the repo, deepspec/eval/dspark/confidence_head.py::PerPositionConfidenceMetrics, accumulates per-position ECE / AUROC / Brier over exactly these cumulative products and produces the reliability diagrams in the paper.

Hardware-aware prefix scheduler: verification length as global throughput maximization. Earlier methods apply a static threshold to confidence scores to pick a verification length. That works under a single-request assumption and is suboptimal in a high-concurrency production system, because the utility of verifying one more draft token depends heavily on current load. The paper restates the problem as a global optimization (Algorithm 1). A batch holds $R$ active requests, request $r$ carries per-position confidences $c_{r,1},\dots,c_{r,\gamma}$, and the scheduler assigns verification lengths $\ell_r\in\{0,\dots,\gamma\}$. Since speculative decoding only accepts contiguous prefixes, the survival probability at position $j$ is the cumulative product

$$a_{r,j}=\prod_{i\leqslant j}c_{r,i}$$

The batch size handed to the target model in one verification step, counted in tokens, and the expected number of accepted tokens are then

$$B=\sum_{r=1}^{R}(1+\ell_r),\qquad \tau=\sum_{r=1}^{R}\Bigl(1+\sum_{j=1}^{\ell_r}a_{r,j}\Bigr)$$

Writing engine throughput as $\text{SPS}(B)$ (steps per second at forward batch size $B$), the scheduling objective is expected system-level token throughput

$$\Theta=\tau\cdot\text{SPS}(B)$$

The capacity curve is profiled once offline at engine initialization and stored as a lightweight cost table. Treating $\text{SPS}$ as a function of $B$ alone is a simplification, and the paper defends it in a footnote: average context length in real serving is far below extreme values and has negligible effect on decode latency for a heavily optimized architecture like DeepSeek-V4, while prefill-decode disaggregated deployments let the decode-side load balancer roughly equalize request count and total context length across DP ranks, flattening sequence-length variance.

Why greedy is globally optimal, and why early stopping cannot be dropped. $\Theta$ looks like combinatorial search, but its structure admits an efficient greedy. Since $a_{r,j}$ is monotonically non-increasing in $j$, the marginal gain of extending request $r$ from $j-1$ to $j$ is exactly $a_{r,j}$, so sorting all candidates by $a_{r,j}$ in descending order automatically respects intra-block prefix dependence. For fixed $B$ the optimal allocation is to greedily take the highest-survival draft tokens from the global pool of $\{a_{r,j}\}$. The scheduler decides $B$ incrementally along that same greedy admission path: sort globally, admit one at a time, look up $\text{SPS}$ in the cost table at each step to update $\Theta$, and break as soon as $\Theta\leq\Theta_{\text{best}}$.

That break is not an engineering shortcut but a necessary condition for losslessness. Lossless speculative decoding strictly requires the admission decision to be non-anticipating: it may not depend on future candidate tokens. The confidence head consumes Markov features of the previous sampled token, so computing $a_{r,k+1}$ requires the realized value of $x_{r,k}$. A retrospective global search would therefore leak $x_{r,k}$ into the admission decision at step $k$ and induce selection bias. Appendix A gives a minimal counterexample: take $R=1$, $\gamma=2$, $a_1=0.8$, and a profiled curve with $\text{SPS}(1)=1.0$, $\text{SPS}(2)=0.5$, $\text{SPS}(3)=0.45$, so $\Theta_0=1\times1.0=1.0$ and $\Theta_1=(1+0.8)\times0.5=0.9$. Without early stopping the scheduler evaluates $\Theta_2$ first: if $x_1$ happens to give $c_2=0.9$, then $a_2=0.72$ and $\Theta_2=(1+0.8+0.72)\times0.45=1.134$ is the global maximum, returning $\ell=2$ and admitting the first draft token; if $x_1$ gives $c_2=0$, then $\Theta_2=(1+0.8+0)\times0.45=0.81$ and the global maximum stays $\Theta_0=1.0$, returning $\ell=0$ and refusing admission. Whether $x_1$ is admitted now depends on what $x_1$ is. Turned into a distributional bias: with vocabulary $\{A,B\}$, $p_t(A)=0.7$, $p_t(B)=0.3$, $p_d(A)=p_d(B)=0.5$, the standard acceptance probability $\min(0.7,0.5)+\min(0.3,0.5)=0.8$ matches the assumed $a_1$, yet under the retrospective scheduler $x_1=A$ is accepted with probability $\min(1,0.7/0.5)=1$ and outputs $A$, while $x_1=B$ is never admitted and gets resampled by the target model, giving $\Pr(Y=A)=0.5+0.5\times0.7=0.85$ and $\Pr(Y=B)=0.15$ against a target of $(0.7,0.3)$. Retrospective scheduling is not lossless. Stepwise early stopping confines the truncation decision to the prefix processed up to that exact step, isolating it from future tokens and recovering the target distribution exactly. The cost is that stepwise early stopping attains the global maximum if and only if $\Theta$ is unimodal, which implicitly assumes a smoothly decaying hardware capacity curve; Section 5.2 handles the case where real SPS is not smooth.

Training objective: three loss terms with positional decay weights. During training multiple anchor positions are sampled at random from each target sequence to form $\gamma$-token blocks. The target model stays frozen throughout, the drafter shares and freezes its embedding table and LM head, and only the parallel backbone, the sequential block and the confidence head receive gradients. All three losses carry a positional weight $w_k=\exp(-(k{-}1)/\gamma)$ that emphasizes early positions, which contribute most to expected accepted length under prefix verification. The cross-entropy term trains the drafter to predict the correct next token (Equation 9) and the distribution-matching term penalizes total variation distance between draft and target distributions (Equation 10):

$$\mathcal{L}_{\text{ce}}=-\sum_{k=1}^{\gamma}w_k\log p^d_k(x^*_k),\qquad \mathcal{L}_{\text{tv}}=\sum_{k=1}^{\gamma}w_k\lVert p^d_k-p^t_k\rVert_1$$

Because total variation distance is a direct proxy for acceptance rate (the stepwise acceptance probability equals $1-\frac{1}{2}\lVert p^d-p^t\rVert_1$), minimizing $\mathcal{L}_{\text{tv}}$ is equivalent to maximizing expected acceptance directly. The confidence term is binary cross-entropy against the soft label $c_k^*$ (Equation 11), and the total objective combines all three with default weights $\alpha_{\text{ce}}=0.1$, $\alpha_{\text{tv}}=0.9$, $\alpha_{\text{conf}}=1.0$ (Equation 12):

$$\mathcal{L}_{\text{conf}}=-\sum_{k=1}^{\gamma}w_k\bigl[c_k^*\log c_k+(1-c_k^*)\log(1-c_k)\bigr],\qquad \mathcal{L}=\alpha_{\text{ce}}\mathcal{L}_{\text{ce}}+\alpha_{\text{tv}}\mathcal{L}_{\text{tv}}+\alpha_{\text{conf}}\mathcal{L}_{\text{conf}}$$

Each term is checkable in the released implementation. loss.py::_build_loss_weight_mask builds $w_k$ with torch.exp(-positions.float() / loss_decay_gamma); note that the config sets loss_decay_gamma=4.0 while block_size=7, so the code treats the decay constant as an independent hyperparameter rather than tying it strictly to block length. _compute_local_l1_term evaluates $\lVert p^d-p^t\rVert_1$ and normalizes it with the same weight mask. The confidence target is explicitly detached (confidence_targets = accept_rate_3d.detach()), which prevents the confidence loss from pushing gradients back into the draft distribution. ce_loss_alpha=0.1 and l1_loss_alpha=0.9 are the first two weights of Equation (12), confidence_head_alpha=1.0 the third. Remaining training hyperparameters: lr=6e-4, warmup_ratio=0.04, weight_decay=0, bf16, global_batch_size=512, 10 epochs, max_length=4096, Qwen chat template.

Truncation at inference time. The offline evaluator turns the scheduler into one explicit rule. deepspec/eval/dspark/draft_ops.py::_confident_prefix_length walks the block and truncates at the first position whose confidence falls below the threshold, sigmoid(conf) < threshold; a threshold at or below zero disables truncation and verifies the full block. That is the static-threshold regime used for the diagnostic sweeps below, and it is also the degenerate case of the scheduler's dynamic top-$K$ admission.

Wiring the pipeline together. The diagram below redraws the decoding cycle using the paper's actual connections. Three details are easy to miss: the sequential head only adds a logits bias and never recomputes hidden states; the confidence head shares the very same Markov embedding $W_1[x_{k-1}]$ as the sequential head, which is why it must depend on an already-sampled token and why the causality argument exists at all; and the scheduler reads an offline-profiled $\text{SPS}(B)$ cost table, so its decision depends on data-side confidence and system-side load simultaneously.

flowchart TD
  P[Prompt tokens] --> T1[Target model one step
emits anchor token x0]
  T1 -->|target hidden from layers 1 9 17 25 33
Eq2 RMSNorm Wc concat| CTX[H_ctx context features]
  CTX -->|Eq3 KV injection into every draft layer| BB[Parallel backbone DFlash
5 layers, single forward pass
anchor counts as position 1]
  T1 --> BB
  BB --> HK[hidden states h_1..h_gamma]
  BB --> UK[base logits U_1..U_gamma]
  HK --> SEQ[Sequential head, Markov default
Eq5 bias B = W1 of x_{k-1} times W2
rank r = 256, left to right loop]
  UK --> SEQ
  SEQ --> DK[draft tokens x_1..x_gamma
Eq4 softmax over U_k plus B_k]
  HK --> CONF[Confidence head
Eq7 c_k = sigmoid of w dot concat h_k and W1 of x_{k-1}]
  SEQ -->|shares Markov embedding| CONF
  CONF --> STS[Post-hoc STS calibration
per position 1D grid search minimizing ECE
order preserving]
  STS --> SCH[Hardware-aware prefix scheduler
a_{r,j} = cumprod c, B = sum 1+l_r
tau = sum 1 + sum a, Theta = tau times SPS of B
global sort by a plus early stop]
  SPS[Profiled SPS of B cost table
built once at engine init] --> SCH
  SCH -->|scheduled lengths l_r| VER[Target model parallel verification
reject sampling, longest accepted prefix plus bonus]
  VER -->|accepted prefix and bonus token| OUT[Output tokens]
  OUT -->|last token becomes next anchor| T1
  VER -.->|Eq8 analytic label c_k star = 1 minus half TV distance| LOSS[Training only: L = 0.1 Lce + 0.9 Ltv + 1.0 Lconf
position weights w_k = exp of minus k-1 over gamma]
  LOSS -.-> BB
  LOSS -.-> SEQ
  LOSS -.-> CONF

Figure 2: the DSpark decoding cycle and its training loop, drawn from Section 3 and Algorithm 1. Solid edges are inference-time data flow; dashed edges are the supervision the frozen target model provides during training. The target model, the shared embedding table and the LM head are frozen throughout.

Experiments

Setup. Four target models across two families: Qwen3-{4B, 8B, 14B} and Gemma4-12B. Two drafters serve as the comparison points: DFlash for the parallel route and Eagle3 (Training-Time Test variant) for the autoregressive route. To keep the comparison honest all three drafters are retrained in the same framework on the same data. Eagle3's TTT horizon is aligned to 7 to match the block size used by DFlash and DSpark, all drafters draw on the same set of target feature layers, and depth is 1 layer for Eagle3 versus 5 layers for DSpark and DFlash. Training data is Open-PerfectBlend (the open release of PerfectBlend, 1.3M samples; chat 17.6%, math 39.4%, code 38.9%, instruction following 4.1%): only the prompts are used, and responses are regenerated by each target model with its recommended sampling parameters. Every drafter trains for 10 epochs to ensure convergence, and both generation and evaluation run in non-thinking mode. The metric is accepted length per decoding round $\tau$, including the bonus token produced by the target model, at sampling temperature 1.0 with chained drafting.

Main result: accepted length leads across the board. To isolate raw draft quality from system-level scheduling policy, the offline evaluation disables the confidence scheduler and forces every drafter to propose a fixed-length block. That caveat matters: Table 1 measures drafting quality alone, and the scheduler's contribution only appears in the production results of Section 5.

Target	Drafter	GSM8K	MATH	AIME25	MBPP	HumanEval	LCB	MT-Bench	Alpaca	Arena-Hard
Qwen3-4B	Eagle3	5.14	4.62	3.92	3.69	4.16	3.77	2.39	2.26	2.55
	DFlash	5.40	4.85	4.15	4.40	4.74	4.18	3.07	2.96	2.83
	DSpark	6.11	5.70	4.89	5.13	5.38	4.86	3.64	3.54	3.29
Qwen3-8B	Eagle3	5.30	4.77	3.91	3.96	4.33	4.17	2.66	2.54	2.54
	DFlash	5.33	4.91	4.07	4.36	4.64	4.39	3.11	2.98	2.81
	DSpark	6.17	5.78	5.01	5.16	5.52	5.17	3.72	3.58	3.21
Qwen3-14B	Eagle3	5.24	4.60	3.71	3.81	4.14	4.01	2.62	2.47	2.48
	DFlash	5.41	4.84	3.98	4.44	4.59	4.33	3.10	2.94	2.72
	DSpark	6.21	5.74	4.94	5.26	5.43	5.02	3.70	3.58	3.13
Gemma4-12B	Eagle3	5.87	5.46	4.83	4.72	5.37	4.16	3.19	3.06	2.72
	DFlash	5.45	5.04	4.22	4.39	4.95	3.70	2.98	2.84	2.59
	DSpark	6.05	5.78	5.12	5.11	5.64	4.51	3.49	3.35	2.92

Table 1: accepted length per decoding round $\tau$ (paper Table 1; higher is better). In macro averages DSpark beats Eagle3 by 30.9%, 26.7% and 30.0% on Qwen3-4B/8B/14B and DFlash by 16.3%, 18.4% and 18.3%. On Gemma4-12B DSpark takes first place on all nine benchmarks, so the advantage holds across model families.

Table 1 also exposes a strong domain effect. On Qwen3-4B, structured tasks have intrinsically longer accepted lengths (math macro 5.57, code 5.12) than open-ended chat (3.49). That intrinsic variance in data predictability is precisely why a static verification length must waste compute on tail tokens, and it is the direct motivation for confidence-scheduled verification. The Gemma4-12B block deserves separate mention: there DFlash is actually worse than Eagle3 (GSM8K 5.45 versus 5.87, trailing on eight of nine benchmarks), which shows that the parallel-drafting advantage does not transfer across families automatically. DSpark pulls all nine benchmarks back to first place, effectively repairing the failure mode of both routes at once.

Figure 3: position-level conditional acceptance rate (paper Figure 2; Qwen3-4B, averaged over benchmarks within each domain). The denominator counts only instances where the first $k-1$ draft tokens were all accepted by the target model, so this metric strips out the penalty for early prefix errors and isolates the base prediction quality at position $k$. The autoregressive drafter Eagle3 stays flat or rises, while the parallel drafter DFlash exhibits suffix decay.

Why parallel beats autoregressive: a three-part decomposition. That parallel and semi-autoregressive drafters achieve longer accepted lengths than a fully autoregressive drafter runs against the usual expectation that step-by-step generation yields higher-quality sequences. Figure 3's conditional acceptance rates decompose the effect into three parts. First, capacity at position 1: at the very first draft position both architectures predict the next token from target context alone, so any gap is purely architectural capacity. An autoregressive model is constrained by $O(\gamma)$ latency into shallow networks, while an $O(1)$ parallel drafter can afford depth; DFlash therefore sits clearly above Eagle3 at position 1 (Math 0.88 versus 0.81, Chat 0.72 versus 0.53). Speculative decoding is a strict prefix-survival process in which the first token has the highest leverage, because rejection there voids the entire block, so this initial capacity advantage is amplified disproportionately into final accepted length. Second, the independence defect at later positions: from position 2 to 7, later tokens should become easier as early tokens lock in the semantic path. Eagle3 exploits that conditional certainty (rising from 0.53 to 0.74 on Chat) while DFlash keeps decaying (Code 0.87 to 0.78, Chat 0.72 to 0.63). Third, DSpark collects both ends: it inherits the deep parallel drafter's high initial acceptance (starting at 0.93 on Math), and the lightweight sequential head suppresses the fast decay typical of parallel generation, leaving the whole block at a high and stable conditional acceptance rate.

A little autoregression goes a long way: depth, block size, latency overhead. The paper sweeps the design space along two axes. For depth it fixes block size at 7, varies DSpark from 1 to 5 layers, and compares against a 5-layer DFlash: accepted length rises monotonically with depth, the steepest marginal gain is 1 to 2 layers, and a 2-layer DSpark beats a 5-layer DFlash on all three domains. The implication is parameter efficiency: injecting local autoregression through a lightweight sequential head buys more sequence coherence than simply stacking deeper parallel layers.

Figure 4: effect of drafter depth (paper Figure 3). With draft length fixed, accepted length aggregated over the math / code / chat domains rises monotonically with the number of DSpark layers, and a 2-layer DSpark already outperforms the deeper 5-layer DFlash baseline.

Figure 5: draft length and latency overhead (paper Figure 4). Left three panels: with 5 layers fixed, draft length (proposal length $\gamma$ plus one anchor) is swept over $\{4,8,12,16\}$; DSpark wins at every length and the gap widens as $\gamma$ grows. Right panel: end-to-end engine latency at batch size 128 (one target verification plus one parallel draft forward plus the sequential sampling loop), averaged arithmetically over context lengths $\{512,1024,2048,4096\}$ to remove sequence-length bias.

For block size the paper fixes 5 layers and sweeps draft length over $\{4,8,12,16\}$. DSpark wins at every length and its advantage grows steadily with $\gamma$: the accepted-length gain over DFlash is math 16% / code 15% / chat 18% at $\gamma=7$ and expands to 30% / 26% / 22% at $\gamma=15$. Figure 3 already explains why: pure parallel generation suffers fast suffix decay, so long blocks have diminishing marginal utility, and mitigating that decay means the longer the block, the larger the relative gain. The RNN head only helps marginally and only on longer drafts, which is why the Markov head remains the default. The latency numbers are the most underrated part of the paper: because the target model dominates verification compute at this batch size, the sequential block is nearly free, and stretching draft length from 4 to 16 adds just 0.2%-1.3% to end-to-end step latency relative to the DFlash baseline while buying up to 30% more accepted length.

Confidence-head diagnostics: a static threshold sweep. To examine the estimator in offline isolation, the paper runs a threshold sweep on Qwen3-4B (threshold zero is equivalent to standard fixed-length verification) and leaves the hardware-aware scheduler for the production evaluation in Section 5. As the threshold rises, overall acceptance rate climbs steadily because the confidence head prunes tokens that would ultimately be rejected. Pruning is most dramatic on chat, where high-entropy distributions make fixed-length verification least efficient, taking acceptance from 45.7% to 95.7%; structured tasks are pruned more gently and retain more draft tokens, with Math rising from 76.9% to 92.5% and Code from 67.6% to 92.0%.

Figure 6: confidence threshold sweep (paper Figure 5). A threshold of zero is standard fixed-length verification; as it rises the hatched bar segments (tokens ultimately rejected) are pruned away and overall acceptance climbs steadily.

Figure 7: reliability diagram on Alpaca (paper Figure 6). The background histogram is sample frequency per confidence bin. The raw estimator is strongly discriminative but systematically overconfident; post-hoc calibration aligns prefix survival probability with empirical acceptance rate.

A static threshold is nevertheless suboptimal in a dynamic serving environment, because it ignores system load: at low concurrency the opportunity cost of verifying a low-confidence token is tiny, while at high concurrency it squanders scarce batch capacity. That is the motivation for the hardware-aware scheduler, and it requires a confidence model that is both discriminative and precisely calibrated. Figure 7 quantifies the outcome: the raw model is discriminative (ROC-AUC 0.81-0.90) but overconfident (ECE 3%-8%), and post-hoc STS brings mean ECE down to roughly 1%, making survival-probability estimates usable for scheduling.

Diagnostic	Domain	Raw / static threshold	After calibration or scheduling	Reading
Overall acceptance under threshold sweep	Chat	45.7%	95.7%	Strongest pruning; high-entropy distributions make fixed-length verification least efficient
	Math	76.9%	92.5%	Gentler pruning, more draft tokens retained
	Code	67.6%	92.0%	Structured text is intrinsically easy to accept
Confidence discrimination, ROC-AUC	All	0.81-0.90	—	Ranking ability is already strong enough
Calibration error, ECE	All	3%-8%	About 1% after STS	Scheduling needs absolute magnitude, hence calibration
Sequential head overhead	batch 128, ctx 512-4096	DFlash baseline	+0.2%-1.3% ($\gamma$ 4 to 16)	The target model dominates verification compute
	Accepted-length gain	$\gamma=7$: +16/15/18%	$\gamma=15$: +30/26/22%	math / code / chat; longer blocks pay more

Table 2: key quantitative findings for the confidence head and the sequential head, drawn from Sections 4.3.2 and 4.3.3 and Figures 4-6. Both the threshold sweep and the reliability diagnostics run on Qwen3-4B, and the offline stage deliberately keeps the scheduler off so that the estimator alone is under test.

Real-World Deployment: From Offline Algorithm to the DeepSeek-V4 Production Line

Section 4's numbers were measured on offline benchmarks with the scheduler switched off. The paper devotes an entire section (Section 5) to the engineering obstacles encountered when wedging DSpark into the production serving systems of DeepSeek-V4-Flash and DeepSeek-V4-Pro previews, and that section carries more information than the typical "system implementation" paragraph, because it describes a machine already serving real traffic.

The production draft model does not share the offline evaluation configuration. The backbone is three MoE layers with mHC, sliding-window attention of 128, a maximum block size of $\gamma=5$ (evaluation used 7), and a Markov head for sequential modeling. The confidence head trains end to end alongside the draft model and is then calibrated with STS to provide reliable scheduling signals.

Two training bottlenecks, two system-level fixes. Training a drafter needs the target model's output distributions as supervision, and running both models over the full document context blows up memory and inter-worker communication. Two optimizations inside the internal HAI-LLM framework handle this. The first is hidden-state communication: shipping the target model's full-vocabulary logits ($V\approx 10^{5}$) across parallel workers is a bandwidth killer, so the target model's forward activations are cached temporarily and only the hidden states immediately preceding the LM head are transmitted. The LM head projection then runs locally on the draft workers and only for the sampled target positions, cutting per-token communication complexity from $O(V)$ to $O(d)$, where $d$ is the hidden dimension. The second is anchor-bounded sequence packing: to decouple the drafter's compute from the target model's context length, a fixed number of draft anchors is sampled from the training sequence and those isolated prediction blocks are packed into dense batches. Packing is managed through token-level attention indices rather than standard 2D masks, which preserves exact causal masking across multiple independent sequences and anchors while avoiding the compute and memory cost of padding.

Two fundamental conflicts between the scheduler and production infrastructure. Algorithm 1 is theoretically lossless, but deploying it directly runs into two problems. First, it assumes a smooth unimodal capacity curve, whereas the real hardware $\text{SPS}(B)$ is discrete and degrades in jagged steps: crossing a batch-capacity threshold drops throughput by a notch, and stepwise greedy search with early stopping halts at the first downward edge, getting trapped in a local optimum just before the cliff. Second, it requires per-step dynamic draft lengths, which clashes with continuous CUDA graph replay and Zero-Overhead Scheduling (ZOS): ZOS needs the next step's batch size known before the current step finishes, so synchronous scheduling stalls the GPU pipeline.

The fix is to make scheduling asynchronous, and this single change resolves both correctness and utilization. Concretely, the confidence head's outputs from two steps prior approximate the verification capacity about to become available; the current step's candidate tokens are still strictly sorted by their actual, up-to-date cumulative confidence scores, and the two-step-old prediction is used only to determine the dynamic truncation length, that is, the batch capacity limit $K$. Admission thereby becomes a dynamic top-$K$ selection: $K$ carries a slight temporal offset, but the selection mechanism is fundamentally rank-preserving, so the most confident draft tokens are always verified first. Scheduling latency is fully hidden and ZOS integration is seamless.

With that asynchronous pipeline in place, the early-stopping break can be removed in favor of an unconstrained global search, which is what actually crosses SPS cliffs to reach maximum physical throughput. Retrospective search would normally leak future token information and break the lossless guarantee, which is exactly what the Appendix A counterexample warns against, but the asynchronous design forms a causal barrier: the unconstrained search evaluates only historical predictions from two steps prior, so the admission decision is isolated from the realization of the current token $x_{r,k}$, and the truncation length inherently depends only on information already available two steps earlier. Losslessness survives and the throughput across hardware cliffs is recovered.

Inference kernels: variable-length queries. Dynamic routing pushes a hard problem down to the physical execution layer, because the inference framework must efficiently support variable-length queries within a single batch. Standard decode kernels are heavily optimized for fixed query lengths, and naively processing variable-length verified prefixes leads to severe GPU under-utilization through padding and uneven workload distribution. The solution decouples physical execution from logical sequence tracking: all tokens across requests are flattened and processed identically as independent elements, while the complex intra-sequence dependencies are conveyed strictly through a marker tensor integrated into the sparse attention implementation. On the DeepSeek-V4 architecture only the index-attention and compress kernels require modification to support this variable-length routing, so the dynamic scheduler operates without introducing low-level execution overhead.

The paper also clarifies that the classic latency-versus-throughput trade-off is not actually a trade-off in its setting. In production the number of requests processed per step is frequently constrained by resource limits (fixed KV-cache capacity per request) and by the available traffic pool (RL long-tail loads), so the effective batch size persistently stays well below the GPU's compute-saturating threshold. In that regime, given a fixed concurrency limit, maximizing per-GPU total token throughput and maximizing per-user generation speed (tok/s/user) become highly correlated objectives rather than competing ones.

Figure 8: the throughput-interactivity Pareto frontier under live user traffic (paper Figure 7). Scatter points are raw telemetry sampled directly from live traffic, capturing complex real request distributions; solid lines are the fitted performance frontiers. DSpark-5 pushes the whole frontier outward relative to the MTP-1 baseline.

Results under live user traffic. The evaluation pits DSpark-5 ($\gamma=5$) against the MTP-1 baseline inside the production serving engines of DeepSeek-V4-Flash (preview) and DeepSeek-V4-Pro (preview). MTP-1 was the former production setup and was superseded by DSpark two weeks after the V4-preview release. The single-token configuration had been kept in production historically because a static multi-token drafter (MTP-3/5) strictly degrades aggregate throughput under high concurrency through excessive verification overhead. Comparing DSpark against this established baseline therefore demonstrates directly that it can safely unlock the potential of larger draft blocks in a dynamic serving environment.

Engine	SLA (tok/s/user)	Aggregate throughput vs MTP-1	Per-user speedup at matched throughput	The authors' own reading
DeepSeek-V4-Flash (preview)	80 (moderate)	+51%	+60% to +85%	Comparable operating regime; representative gain
	120 (strict)	+661% (nominal)		The baseline approaches its operational boundary and sustains only a very small concurrent batch; this point is evidence that DSpark extends the feasible interactivity frontier, not a multiplicative speedup over a well-utilized baseline
DeepSeek-V4-Pro (preview)	35 (moderate)	+52%	+57% to +78%	Same pattern reproduced
	50 (strict)	+406% (nominal)		MTP-1 again enters a low-concurrency regime; read as sustaining useful throughput under an interactivity target the baseline cannot efficiently support

Table 3: measured results at production SLA anchors, from Section 5.4 and Figure 7. The authors explicitly attach a conservative interpretation to the two nominal percentages, and that framing is worth more than the numbers themselves.

Figure 9: load-adaptive throughput and verification budgets (paper Figure 8). Top row (a, b): aggregate output throughput across levels of system concurrency. Bottom row (c, d): average target verification budget allocated per request. As concurrent load rises, the dynamic scheduler automatically restricts per-request verification length to prevent resource contention.

Figure 9 explains the mechanism behind those gains. In the moderate concurrency regimes typical of production deployment (fewer than 200 concurrent requests for V4-Flash, 150 for V4-Pro), the hardware-aware scheduler puts idle target compute to work by expanding the verification budget from MTP-1's static 2 tokens to roughly 4-6 tokens per request. Longer verification means more accepted tokens per forward pass, which converts directly into the throughput gains on the Pareto frontier. As concurrency scales and target capacity saturates, the scheduler restricts that budget in the opposite direction: average verification length decreases smoothly with load, and low-confidence draft tokens are pruned before they consume critical batch capacity. Exploiting idle compute under light traffic while protecting critical batch capacity under heavy traffic is what makes the production deployment stable.

Limitations

Stated by the authors: the fixed draft-side cost is unrecoverable. The prefix scheduler minimizes wasted verification on the target side, but DSpark still pays a fixed draft-side cost, since the parallel backbone must generate the full $\gamma$-token block first. For complex queries with intrinsically low acceptance rates, that upfront drafting compute cannot be recovered: the scheduler can decide not to verify, but it cannot decide not to draft. The authors' proposed direction is difficulty-aware early exiting inside the draft model, letting such requests bypass full-block generation entirely.

Global optimality of early stopping depends on the unimodality assumption. Stepwise early stopping in Algorithm 1 returns the global maximum throughput only if the objective $\Theta$ is unimodal, which implicitly assumes a smoothly decaying hardware capacity curve. Section 5.2 concedes that real $\text{SPS}(B)$ degrades in jagged steps, so the production version has to drop early stopping and run an unconstrained global search to get the right answer. In other words, the cleanest algorithm in the paper is not the one running in production.

"Two steps prior" is an unquantified approximation. Asynchronous scheduling uses confidence outputs from two steps earlier to set the current step's capacity $K$. The paper argues this is rank-preserving and lossless thanks to the causal barrier, but it offers no experiment on how quickly the offset becomes inaccurate under fast-changing load. If concurrency shifts sharply within two steps (traffic bursts, preemption, a cluster of long-context requests arriving together), how much throughput is lost by slicing the current step's budget with a stale capacity estimate is simply not measured.

Offline evaluation and online deployment are different configurations, and SPS models only one variable. Offline results use $\gamma=7$ with the scheduler disabled; production uses $\gamma=5$, a three-layer MoE backbone and the scheduler fully enabled. The two sets of numbers cannot corroborate each other, so a reader cannot tell how much of the offline accepted-length gain actually converts into online throughput. Meanwhile the throughput model $\text{SPS}(B)$ takes batch size as its only argument, whereas real per-step cost also depends on sequence-length distribution, KV-cache hit rate and the prefill-decode mixing ratio. Whether the curve still holds under long-context or prefill-heavy load is not discussed.

Production results have limited reproducibility. Every number in Section 5.4 comes from DeepSeek's internal serving engines, the internal HAI-LLM training framework, DeepSeek-V4 preview models and real user traffic telemetry. The open-source DeepSpec repository ships the offline training and evaluation pipeline, so a third party can reproduce the offline half of the conclusions but not the Pareto frontier.

Conclusion and Outlook

DSpark's contribution separates into three layers, and its persuasiveness decreases along them. The algorithmic layer: the semi-autoregressive drafting paradigm, a compute-heavy parallel backbone plus a nearly free sequential head, fixes suffix decay in parallel drafters, lifting accepted length 30.9%/26.7%/30.0% over Eagle3 and 16.3%/18.4%/18.3% over DFlash at a cost of 0.2%-1.3% of end-to-end step latency. This layer is clean, reproducible and ships with open weights. The calibration layer: treating the confidence head as a probability estimator that must satisfy discrimination and calibration simultaneously, evidenced by ROC-AUC 0.81-0.90 and post-STS ECE of about 1%, and showing that "ranks correctly" and "has correct magnitude" are different properties while scheduling requires the second. Its methodological value extends well beyond speculative decoding, since any system that allocates resources by a model's self-reported confidence hits the same trap. The systems layer: promoting verification length from a hyperparameter to a scheduling variable solved against real-time engine load, and replacing MTP-1 in production.

The systems layer is also the hardest to verify independently. The production numbers are impressive but welded to DeepSeek's own models, framework and traffic, and Section 5.2 concedes that what runs online is not Algorithm 1 but its asynchronous approximation. That candor is to the paper's credit, and it also means the algorithm's actual shape was forced by infrastructure rather than derived from theory. What other teams can transfer directly are three engineering conclusions: if the capacity curve is step-shaped, do not use greedy search with early stopping; make scheduling asynchronous, use historical confidence to set capacity and current confidence to set ordering, and you get utilization and losslessness together; and support variable-length verification by flattening tokens plus a marker tensor rather than by patching kernels with padding.

Looking forward, the difficulty-aware early exit the authors name is the most direct extension. Drafting cost currently treats all requests alike while request difficulty is plainly long-tailed, and a request that can exit during drafting saves more than one merely pruned during verification. A second direction is to widen the scheduling signal from "what the drafter reports about itself" to "what the serving system observes globally": today $\text{SPS}(B)$ looks only at batch size, and folding in sequence-length distribution, KV-cache pressure and queue waiting time would allow verification budget allocation finer than smooth shrinkage with concurrency. A third class of problems goes untouched here. Confidence calibration is performed on evaluation sets close to the training distribution, whereas distribution shift (new domains, new languages, very long contexts) would misalign the STS isotonic mapping, and the scheduler's response to misalignment is to change the verification budget directly, which amplifies a statistical problem into a service-quality problem.

For RobotWorld's readers the value of this paper is not robotics but the demonstration of a complete engineering loop that jointly optimizes model capability and the serving system: an algorithmic change (semi-autoregressive drafting), an observable signal (calibrated survival probability), a scheduling decision (dynamic verification budget), and validation against live telemetry (an outward-shifted Pareto frontier). Real-time control loops in embodied systems are far more latency-sensitive than text generation, and deploying vision-language-action models faces the same three conditions of limited compute, bursty requests and hard latency constraints, so the methodology travels better than the specific numbers do.

Speculative decoding research has spent years asking how to make drafts more accurate. DSpark supplies the other half of the question: once the draft is accurate, how much of it should be verified. The first answer is architecture; the second is scheduling, and scheduling needs a confidence signal that both ranks correctly and tells the truth about magnitude.

How the authors handle the +661% figure matters more than the figure itself: they volunteer that the baseline is already at its operational boundary under that SLA, so the point should be read as an extended frontier rather than a multiplicative speedup over a well-utilized baseline.