Does speculative decode still pay at 128K context?

Background

Every speculative-decode number on this site is measured at a short prompt. The Qwen3.6 35B-A3B NVFP4 recipe headlines 106.5 tok/s at k=3 against 67.9 with no draft — a 57% gain — and that figure is taken at a 2,048-token prompt. A Spark owner running a coding agent is not at 2,048 tokens. They are 60K into a session with three files pasted in.

Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context (arXiv:2607.21535) argues that gap matters. Its observation is mechanical: "an MTP draft head typically runs full attention over the entire KV cache at every draft step, so its read grows linearly with context and comes to dominate the draft cost." The draft is only cheap because it is small in weights — but it reads the same KV cache the target does, once per draft step, and that read scales with the prompt. The paper's fix is to window the draft's attention (sliding window plus attention sink), which it reports cuts per-decode-step cost by 28–44% at 1M context in SGLang, on model families including a Qwen GDN-MoE 35B — architecturally the neighbourhood of the model we serve here.

Their numbers are theirs: a single datacentre GPU, SGLang, and a million tokens of context. We have a DGX Spark, vLLM, and a 262,144-token window. But the premise should bite harder here, not softer. The Spark is one 128 GB pool of unified LPDDR at modest bandwidth, and decode is bandwidth-bound end to end. Extra KV bytes read by the draft are not competing with spare memory traffic; they are competing with the target's own weight and KV reads for the same bus. If the draft-KV tax is real, a Spark should show it early.

This study does not implement Windowed-MTP — that needs a patch to vLLM's MTP attention path, which does not fit one session. It tests the premise underneath it: does the speculative speedup we publish at 2K survive to 131K on our hardware?

Method

One DGX Spark (Spark-2), vLLM 0.24.0, unsloth/Qwen3.6-35B-A3B-NVFP4-Fast, both arms measured back to back in a single session on 2026-07-28. Nothing from a previous day is compared against; the baseline was re-run from scratch.

Two servers, one variable between them:

• Baseline — the published recipe's serve command, no --speculative-config.
• Treatment — the same command plus --speculative-config '{"method":"mtp","num_speculative_tokens":3}', which loads the MTP draft layer that ships inside the checkpoint (vLLM resolves it as Qwen3_5MoeMTP and shares the target's embedding and lm_head weights with it).

Everything else is held fixed across all six rungs: --max-model-len 262144, --kv-cache-memory-bytes 4294967296 (a 343,699-token KV pool), --gpu-memory-utilization 0.85, --max-num-seqs 4, --max-num-batched-tokens 8192, greedy sampling, thinking off, 256 output tokens, concurrency 1.

Each server was warmed with three short requests, then swept across three prompt lengths without restarting: 2,040 / 32,779 / 131,290 tokens as actually counted by the server. vllm:spec_decode_* counters were snapshotted before and after each length, so acceptance is attributed to that length alone rather than to the server's lifetime.

Repeats were 6 / 4 / 5 per length. The paper's claimed effect is 28–44%, far outside the 1.3–2.0 tok/s run-to-run σ these machines show, so a handful of repeats is enough to resolve it — and the baseline arm turned out to be extremely tight (±0.03 to ±0.07 tok/s sem). The treatment arm is looser; the 32K rung in particular spans 90.2–102.2 tok/s and is the one number in the study to distrust.

Two derived quantities in the charts are worth defining, because they are what makes the arms comparable at all:

• cycleMs is the wall time of one target forward: for the baseline that is simply 1000 / decodeTokS; for the treatment it is tokPerVerify × 1000 / decodeTokS, i.e. the full verify cycle of three draft forwards plus one batched verify. This is the paper's own unit — per-decode-step cost — and the only one in which the draft's tax is directly visible.
• tokPerVerify is accepted tokens per verify cycle, from vLLM's counters. It is 1.0 for the baseline by definition.

Findings

The speedup did not erode. It grew.

This is the opposite of what the study set out to look for. If the draft's full-context KV read were eating the win, the right-hand column would fall monotonically. It does not: the 2K→131K move is +0.080 ± 0.032, about 2.5σ upward. The 32K dip is the noisiest arm in the study and is within about 1σ of the 2K figure; we would not defend it as a real non-monotonicity.

The mechanism for the rise is the same arithmetic that motivates speculation in the first place. A plain decode step got 40% more expensive between 2K and 131K (14.14 → 19.84 ms) as the target's KV read grew. Speculation amortizes that increasingly expensive forward over ~3.1 accepted tokens instead of 1. The more a target forward costs, the more there is to amortize — and on this model that gain outran the draft's own growing cost.

The tax is real. It is one tenth the size of the target's.

The paper's premise is mechanically correct and we measured it directly, in the cycleMs chart:

• a plain decode step grew 5.70 ms across the sweep (14.14 → 19.84 ms);
• the whole verify cycle grew 7.21 ms (31.16 → 38.35 ms).

The verify cycle contains one target forward plus three draft forwards. Subtracting the target's own 5.70 ms leaves 1.51 ms of context-driven growth attributable to the draft, spread over three draft steps: ~0.50 ms per draft step, against 5.70 ms for the target's. The draft's KV tax is about 8.8% of the target's, and it stays that fraction as context grows because both scale linearly in the same KV.

That 8.8% is not a coincidence, and it is the whole result. Qwen3.6-35B-A3B is a hybrid: 40 layers, of which 30 are linear-attention (constant state, no per-token KV growth) and only 10 are full attention (full_attention_interval: 4). The MTP head is one layer. So per decode step the target re-reads ten layers of growing KV and the draft re-reads one — a ratio of 1:10, which is what the timings show. A draft-KV tax that would dominate a deep draft on a fully-quadratic target is, on this checkpoint, a rounding error.

The paper's own worked example is a million tokens of context and a deep native draft. We have neither. What our numbers say is narrower and, we think, more useful to a Spark owner: with a shallow MTP head on a hybrid target, the tax does not reach the size where it changes your decision, even at 131K.

Acceptance did degrade — and it did not matter

Long context cost the draft real accuracy. Accepted tokens per verify cycle fell from 3.362 to 3.102, and the raw acceptance rate from 78.7% to 70.1% of draft tokens. The per-position breakdown degrades evenly rather than collapsing at the tail (434/371/298 at 2K against 359/290/240 at 131K), so this is the draft getting uniformly less certain about a longer context, not the third speculative token becoming worthless.

A 7.7% erosion in tokens-per-cycle is exactly the kind of thing that should eat a speedup. It was simply outweighed by the growth in what a target forward costs.

Speculation stayed the energy-efficient choice

At every length the MTP arm produced more tokens per joule: 2.97 vs 1.96 tok/J at 2K, 1.49 vs 1.06 at 32K, 0.96 vs 0.62 at 131K. Board power rose only marginally with the draft enabled (36.3 vs 36.0 W at 2K, 84.0 vs 81.8 W at 131K) while throughput rose ~50%. On a machine that idles around 36 W and tops out in the mid-80s under a 131K decode, spec-decode is close to free on the power budget.

The number that actually hurts at 131K is TTFT

Nothing in this study fixes prefill, and prefill is what a 131K session actually feels like: 43.7 s to first token in the baseline arm, 47.8 s with the draft loaded. Prefill throughput more than halved from 2K to 131K (6,970 → 3,003 tok/s baseline). The draft costs about 9% on TTFT — it has to prefill its own layer's KV too — which is a real but secondary price for a 1.6× decode win.

Limitations

• We tested the paper's premise, not its method. Windowed-MTP is a change to the draft's attention (sliding window plus attention sink) and would need a patch to vLLM's MTP attention path. Nothing here reproduces or evaluates their technique, and nothing here should be read as a refutation of it.
• One model, one draft depth. A single-layer MTP head at k=3 on a 30/10 hybrid is the mild end of the paper's argument, exactly as the queue entry predicted. A deep native draft (or a dense, fully-quadratic target) is where the compounding they describe should show up, and we did not test one.
• 131K is not 1M. The paper's headline numbers are at a million tokens. Our window caps at 262,144 and a 131K prefill already costs 44 s; extrapolating our slope to 1M is not something these six points support.
• The 32K treatment rung is noisy (90.2–102.2 tok/s over 4 repeats). We report it, we do not lean on it.
• Power is relative-only on GB10. nvidia-smi power.draw reads, power.limit does not, and there is no wall meter — the tok/J figures compare arms fairly but are not absolute.
• Acceptance counters are per-length but not per-repeat. They are summed across the repeats at each prompt length, so they have no error bars.

Open questions

• Does the picture invert on a non-hybrid target? The 1:10 layer ratio is doing all the work here. The same experiment on a dense, all-full-attention model with an MTP head should push the draft's share of the KV read up toward 1:N-layers and is the direct test of whether our explanation is right.
• What happens at k=5 or k=8? The tax scales with draft depth while the amortization benefit saturates as acceptance falls. There should be a context length at which increasing k stops paying, and this study says it is past k=3 at 131K.
• Is a windowed draft worth patching in anyway? Even at 8.8% of the target's tax, a sliding-window draft would recover ~0.5 ms per draft step at 131K. That is roughly 1.5 ms per cycle, or about 4% of decode — small, but it is the acceptance cost (78.7% → 70.1%) that a windowed draft would also have to avoid making worse.
• Does the speedup keep climbing to 262K? Our three points suggest the ratio is a shallow U. Filling in 200K and 262K would settle whether the rise at 131K continues or turns over.