Does quantization actually buy you speed?

<!-- Draft. Numbers live in data/case-studies.json (runs[]) and drive the charts — keep them out of this prose so the two can't disagree. Delete this note at publish. -->

Background

Quantization is sold as a free lunch: shrink the weights, fit a bigger model, go faster. But "faster" hides three separate questions — decode, prefill, and power — and they don't move together. This is a controlled look at all three on a single DGX Spark, taking one model down the precision ladder BF16 → FP8 → INT4 → NVFP4 and changing nothing else.

Method

• Model: Qwen3.6 35B-A3B — a Mixture-of-Experts model with 35B total but only ~3B active parameters per token, plus a hybrid of linear- and full-attention layers. (It's also multimodal, but we only exercise the text path.)
• Engine: vLLM 0.24 on one DGX Spark (GB10, 121 GiB unified LPDDR).
• Precision levels: each rung was self-quantized from the same BF16 base with llmcompressor — FP8 (FP8_DYNAMIC), INT4 (W4A16, group 128), and weight-only NVFP4 (NVFP4A16). One toolchain, one base, so the only variable is the number format. The MoE experts are quantized in every rung; the vision tower, router gates, norms, and linear-attention mixers are held at BF16.
• Harness: bench/harness.py against the served endpoint, cache-busted and warmed up. Prompt sizes 512 / 2048 / 8192, concurrency 1 and 4. The numbers below are the 2048-prompt, single-stream point unless noted.
• Power: mean nvidia-smi power.draw sampled at ~2 Hz across each run.

Caveat on power. On the GB10, nvidia-smi reports power.draw but returns [N/A] for power.limit and even memory.total — the unified-memory driver doesn't expose them, and there's no tegrastats or INA rail. The wattages here are a relative signal across runs on the same box, not certified module or wall power. Idle sits ~5 W; these runs sit 36–41 W.

Findings

Decode speed

Decode — steady-state token generation — is memory-bandwidth bound: each token reads the active weights, so fewer bytes per weight should mean more tokens per second. It does, and cleanly:

30.4 → 37.4 → 42.5 → 43.0 tok/s across BF16 → FP8 → INT4 → NVFP4.

FP8 buys +23%. Dropping to 4-bit buys another ~15%. But INT4 and NVFP4 land on top of each other — because both are weight-only: the 4-bit weights are dequantized back to BF16 for the actual matmul, so they move the same number of bytes and hit the same ceiling. Decode was essentially flat across prompt length at each rung (memory-bound, not compute-bound), confirming the picture.

Prefill / TTFT

Prefill — processing the prompt before the first token — is more compute-bound, and it behaves differently:

3687 → 5089 → 5954 → 5772 tok/s.

FP8 and INT4 each help, but NVFP4 comes in below INT4, and time-to-first-token bottoms out at INT4 (343 ms) rather than NVFP4 (354 ms). The compute-bound prefill phase is more sensitive to the per-weight dequantization overhead that 4-bit formats add — the thing decode barely notices.

Power & efficiency

Here's the result that pushes back on the marketing. Power barely moved — and what movement there was went the wrong way:

Quantization did not make the Spark sip power — the 4-bit schemes actually drew a couple of watts more than FP8, the cost of dequantizing on the fly. So every bit of the efficiency gain (+35% tok/J from BF16 to 4-bit) comes from doing more work per second, not from drawing less. If your goal is a cooler, lower-wattage Spark, quantization isn't the lever; if it's more tokens for the same power budget, it is.

Memory & fit

Footprint is where 4-bit clearly separates from FP8: 67 → 36 → 21 → 22 GiB of weights. FP8 halves the model; 4-bit takes it to a third. On a 121 GiB Spark that headroom is what lets you raise concurrency or stretch context — the payoff that doesn't show up in single-stream tok/s at all.

Limitations

The interesting NVFP4 story is w4a4 — quantizing activations to FP4 too, which is what actually lights up Blackwell's FP4 tensor cores and would pull NVFP4 ahead of INT4. It needs calibration, and calibrating a 256-expert MoE (llmcompressor linearizes every expert to observe it) ran the single GB10 out of memory. So what we measured is weight-only NVFP4, which is why it looks like INT4 — the FP4 tensor-core payoff simply isn't in these numbers.

Open questions

The FP4 tensor-core payoff is still open: w4a4 NVFP4 likely needs expert-offloaded or multi-node calibration to fit the observer pass, and until it runs we can't say whether true FP4 activations pull NVFP4 decisively ahead of weight-only INT4.