How deep should you speculate?
draftJul 20, 2026 · 2x DGX Spark (GB10, 121 GiB unified LPDDR each), TP2 over 200 Gb ConnectX/RoCE
Sweeping draft depth K=0..7 on GLM-5.2 across two DGX Sparks — where speculative decode stops paying, and why.
Holding the model, engine, drafter and workload fixed, how does decode throughput change as speculative draft depth K rises from 0 (no speculation) to 7 (the dspark speculator's native depth) — and where does the acceptance payoff stop covering the per-position verify cost?
- →K=2 is the optimum, and it is not close. Decode peaks at 24.05 tok/s at K=2 and falls away on both sides - 18.43 at K=1, 21.36 at K=3, 14.92 at K=6. By K=6 speculative decode is WORSE than not speculating at all (16.06 tok/s at K=0). The dspark speculator's native depth is 7; running it there costs you a third of your throughput.
- →Acceptance decays about 60% per position, and deeper speculation poisons the shallow positions too. Per-position acceptance runs 0.64 / 0.38 / 0.23 at K=3, and pushing from K=2 to K=3 drops position-0 acceptance from 0.689 to 0.644. That second-order effect is why the measured tok/verify gain (+0.110) is less than half what position 2's own 0.232 acceptance would predict.
- →Each extra verify position costs 14.90 +/- 2.08 ms - less than half the ~32 ms this stack had assumed. That figure had been inferred from a single K=1 vs K=2 differential on an older config; measured across eight depths it is far cheaper, which independently reproduces the s19 finding that dense/attention bytes are paid per step regardless of K.
- →K=2 is also the efficiency optimum, and power rises monotonically with depth. Draw climbs 44.0 -> 49.0 W from K=0 to K=6 while throughput falls, so tokens-per-joule peaks at 0.512 at K=2 against 0.365 at K=0 and 0.304 at K=6. Deep speculation burns more watts to go slower.
- →What the sweep could not resolve, a direct measurement could. Fitting cycle(K) = t_draft + t_verify_base + K*t_marginal gave t_draft = 2.8 +/- 9.3 ms - useless, since run-to-run scatter (11.0 ms) exceeded the quantity, and the estimate slid 29.6 -> 17.8 -> 2.8 ms as higher-K points were added (the cycle is convex in K, so one straight line under-estimates t_draft near the low-K operating point). Timing the proposer's stages directly instead gave t_draft = 15.88 ms, vindicating the low-K fit. The lesson: don't infer a small quantity from the intercept of a noisy extrapolation when you can instrument the thing itself.
Results
Background
Speculative decode trades verify work for drafted tokens: the drafter proposes K tokens, the target verifies them in one pass, and whatever it accepts comes free. More depth means more tokens per verify — but also more verify positions to pay for. Somewhere those curves cross.
The dspark speculator on our GLM-5.2 build has a native depth of 7. Our production config runs K=2, and the recipe prose says K=3. Nobody had measured the whole curve, so nobody knew which was right — or how much the wrong choice costs.
Method
Hold the model, engine, drafter, prompts and sampling fixed; vary only
DSPARK_K from 0 to 7; measure decode throughput and, crucially, tokens per
verify — because tok/s alone can't be decomposed.
Findings
The shape of the answer
Decode peaks hard at K=2 and degrades in both directions. Past K=5 speculation is actively worse than not speculating at all.
The two curves that explain it:
- tokens/verify rises smoothly and saturates — 1.00, 1.67, 2.15, 2.26, 2.38, 2.40, 2.50, 2.79. Clean diminishing returns.
- cycle time rises roughly linearly — about 14.9 ms per added position.
Payoff saturates; cost doesn't. That's the whole story, and K=2 is where they cross.
Why acceptance decays the way it does
Per-position acceptance falls about 40% per step: at K=3 it runs 0.644 / 0.383 / 0.232. Extrapolating, positions 3 and 4 would land near 0.14 and 0.08 — tokens you're paying 14.9 ms each to almost never get.
There's a subtler effect worth knowing about. Deeper speculation degrades the shallower positions. Going from K=2 to K=3, position-0 acceptance falls from 0.689 to 0.644. So the marginal token gain (+0.110) is less than half what position 2's own acceptance (0.232) would predict. Depth isn't merely diminishing — past the optimum it's mildly self-defeating.
The 32 ms that wasn't
This stack had carried a figure of ~32 ms as the marginal cost of a verify position, inferred from a single K=1 vs K=2 differential on an older config. Measured across eight depths it's 14.90 ± 2.08 ms — less than half. That independently reproduces the s19 result (dense/attention bytes are paid per step regardless of K, so a marginal position costs far less than assumed) by a completely different route.
Power: deep speculation costs watts too
Draw rises monotonically with depth, 44.0 → 49.0 W, while throughput falls after K=2. Tokens-per-joule therefore peaks at the same place throughput does. There's no regime where deeper speculation is the efficient choice.
Limitations
The original motivation was to isolate t_draft — the drafter's own
per-cycle cost, and the term a remote drafter would free (see
the hop-cost study). Fitting
cycle(K) = t_draft + t_verify_base + K * t_marginal
gives t_draft = 2.8 ± 9.3 ms, a 95% CI of [−16, +21]. Useless: residual scatter
is 11.0 ms, larger than the quantity itself.
Worse, the estimate slid monotonically as points were added — 29.6 ms (K=1,2),
17.8 ms (K=1–3), 2.8 ms (K=1–7). A drifting intercept means the model is
misspecified: the cycle is convex in K, so a single straight line
under-estimates t_draft near the low-K operating point. Which of those numbers
is right is exactly what the data can't say.
Open questions
The honest lesson: t_draft is a small difference between two large extrapolated
numbers, and no amount of black-box throughput sweeping will resolve it — it'd
take ~10 reps per arm just to reach ±3 ms. The answer is per-stage timers
inside the proposer, measuring the draft call directly. That's a small patch,
not an engine port — and it's the thread the hop-cost study
picks up.