Can you quantize the drafter instead of moving it?

<!-- Draft. Numbers live in data/case-studies.json (runs[]) and drive the charts — keep them out of this prose so the two can't disagree. Delete this note at publish. -->

Background

The spec-depth sweep established that K=2 is the throughput optimum, and direct profiling put t_draft at 15.88 ms of an 89.34 ms cycle — 17.8% of decode time spent on the drafter, on a machine whose binding constraint is memory bandwidth.

The remote-drafter study asked whether to move that work onto another box. This asks the cheaper question: what if you just make the drafter smaller? Same lever — fewer bytes read per cycle — with no network hop, no RPC, and no second machine. And the two stack.

Method

Where the bytes actually are

The speculator is 3.81B params / 7.09 GiB BF16, but not all of it is read each step:

The embed_tokens line is the one that matters for planning: it's a quarter of the checkpoint and quantizing it buys zero speed, because an embedding lookup only touches the rows it needs.

That leaves ~5.17 GiB of genuinely hot weights (~2.66 GiB/rank at TP2). Against the measured 9.91 ms draft_fwd, that implies ~288 GB/s effective — close enough to GB10's nominal bandwidth to confirm the draft forward really is a weight read, which is the whole premise.

Three things had to be built first

This experiment was mostly a build, not a run.

The speculator has no HF modeling class. Its config.json declares architectures: ["DSparkDraftModel"] and ships no modeling_*.py — the model lives inside vLLM. So AutoModelForCausalLM fails and llmcompressor cannot be used at all. The way through is to ignore the model entirely and quantize the raw safetensors: group-wise symmetric RTN is data-free, so it needs no forward pass, no calibration set, and no module tree. That's scripts/quantize_raw_w4a16.py in the case-study skill.

DFlash bypassed quantized dispatch in two places. The per-step forward called F.linear(hidden_states, self.qkv_proj.weight, ...), and the fused context-KV precompute sliced qkv_proj.weight[q_size:]. Under a quantized checkpoint .weight either doesn't exist (marlin stores weight_packed) or is transposed (fp8 PTPC transposes at load) — so the first raises and the second silently returns garbage. Both now route through the quantized path.

vLLM's compile cache doesn't know the drafter changed. Swapping the speculator produced ValueError: too many values to unpack (expected 10): the cached graph was compiled against the BF16 parameter list, and the quantized module exposes weight_packed + weight_scale where weight used to be. The cache key covers neither the drafter's quantization nor edits to the model source. Both arms of this study were recompiled from the same patched source — otherwise the two arms would differ by more than the variable under test. Details are in the DGX-Spark field notes.

The gate before the build

Before producing a checkpoint, the cheapest disproof was: do the INT4 kernels actually deliver on GB10 at the drafter's real shapes? Marlin W4A16, TP2 shard shapes, M=3–64:

The big layers show the textbook bandwidth-bound signature: same GB/s, 4× fewer bytes, 4× faster. The two 18× outliers are a trap — their implied bandwidth is four times what the machine has, which means the 18 MiB int4 weight is sitting in L2 while the 75 MiB BF16 one isn't. In the real model five layers stream and evict each other. Any per-layer microbenchmark whose implied bandwidth exceeds the machine's is measuring cache, not memory.

Findings

The result

Quantizing the four hot groups (protecting embed_tokens, lm_head and markov_head) took the checkpoint 7.09 → 4.57 GiB and bought +8.7% decode, against +9% predicted. The cycle fell 8.8 ms, against ~7.9 ms predicted from the byte cut alone. For a bandwidth-bound drafter, the naive weight-read model was close to right.

And it cost no measurable acceptance. That was the risk the whole study was designed around — speculative decode is lossless, so a degraded drafter never shows up as wrong output, only as fewer tokens accepted per verify. At ~3,800 draft tokens per arm the difference is 0.7σ. The drafter proposes about as well at 4 bits as it does at 16.

The variants that didn't pay off

Three more configurations got measured in the same session. All three land on the "don't bother" pile, and each is worth recording so the next person doesn't re-run it.

Quantizing lm_head too. It's the obvious next target — 1.77 GiB, a third of the hot bytes — and it needs only targets: ["Linear", "lm_head"], since ParallelLMHead isn't a Linear. It bought nothing measurable on decode (0.2σ) while dropping acceptance 2.5σ below the BF16 arm. lm_head has the highest quantization error of any tensor in the model, and it emits the draft logits directly, so every byte saved comes straight back as a rejected token. An earlier draft of this study guessed it was worth "another 5 points"; measuring it is what corrected that. Protect it.

Speculating deeper. A cheaper drafter should tempt you to draft more tokens per cycle. It moves the optimum, but barely: K=3 is a statistical tie with K=2 on the quantized drafter — where on the BF16 drafter K=2 won outright — and K=4 loses 9.6%, its fourth draft slot accepted about one time in ten while its verify cost is paid every cycle. K=2 stays the recommendation.

Grafting a conditioning channel. The orthogonal bet, included because it fails informatively — not a quantization result, but run in the same session. A Sparkulator retrain feeds the target's final pre-norm hidden state in as a sixth fc channel, aiming to raise acceptance rather than speed. It lowered acceptance instead: the new channel is one epoch from zero-init, still too small to earn its keep, and it has perturbed the five trained channels on the way. A second benchmark reproduced the drop independently. Adding capacity a model hasn't learned to use yet is a regression, not a feature.

Limitations

The BF16 baseline did not reproduce. Same build, same flags, one day later: decode 24.05 → 20.09 tok/s, acceptance 0.5743 → 0.5166.

Comparing the quantized drafter against the published baseline would have shown it losing on both axes — slower and dumber — which is the exact opposite of what a same-session paired comparison shows. The published figures weren't wrong so much as underpowered: they were medians of three repeats, and three repeats on this machine have a run-to-run spread of ±20%, which cannot resolve a 9% effect. The BF16 triple was [24.05, 17.42, 25.39]; its median flattered it, and the first W4A16 triple came out [17.97, 23.11, 26.33] — a monotone ramp that was still warming.

Everything above rests on 15 repeats per arm, measured within twenty minutes of each other, on the same warm state, from the same patched source. The absolute numbers are lower than the published ones and should be read as such: it is the paired difference that this study measures, not the ceiling.

The general lesson is the one the remote-drafter study kept learning in the other direction. There, each replaced assumption pushed the projected gain down — 45% → 22% → 15% → 10%. Here the discipline paid the other way: re-measuring the baseline is what turned an apparent regression back into the win the physics had predicted all along. A stale baseline is not a neutral reference point; it is an unmeasured variable.

Open questions

• RTN is the floor. No calibration was used. AWQ/GPTQ need real activations and a real forward pass, which is exactly what the missing modeling class makes awkward — but acceptance held at 4 bits, so there may be little left to win.
• FP8 is nearly free to try. This vLLM fork can quantize at load time with no checkpoint at all (quantization/online/), though the draft config path needs ~3 lines of plumbing first. Cheaper to run than INT4, and a useful middle rung.