Qwen3.6 35B-A3B — NVIDIA NVFP4

NVIDIA's official NVFP4 build of Qwen3.6 35B-A3B serves the full 262,144-token context on one DGX Spark — but it decodes at 42.6 tok/s against 106.5 for Unsloth's NVFP4-Fast build of the same base model. Two structural reasons, both measured here: it is weight-only NVFP4, so the FP4 tensor-core MoE kernels refuse it and vLLM falls back to MARLIN; and the export ships without the MTP head, so it cannot speculative-decode at all.

Contributors

@lloyd094

Experiments run against this recipe

Case studies that used this recipe as their baseline — what was tried, and whether it changed the recipe.

• Does quantization actually buy you speed?Adopted into the recipecomparison arm

  Designated FP8 as the recommended precision for this model on one Spark: +23% decode and +38% prefill over BF16 at half the footprint (67 -> 36 GiB) and flat power. 4-bit adds ~15% more decode and cuts weights to a third, but with clear diminishing returns — it is the pick only when the memory is the binding constraint.

This page exists to answer one question honestly: NVIDIA publishes an official NVFP4 quantization of Qwen3.6 35B-A3B and Unsloth publishes an NVFP4-Fast one, so which should a Spark owner download? Measured on the same node on the same day against the same engine build, the answer is Unsloth's, by a factor of 2.5x on decode. This recipe documents the NVIDIA build anyway, because it serves correctly, it is the vendor artifact, and the reasons it loses are structural and worth understanding rather than folklore. The first reason is the kernel path. NVIDIA's artifact is NVFP4A16: the weights are 4-bit but activations stay BF16, with no input_activations block in its compressed-tensors config. vLLM's FP4 MoE backends all want FP4 activations to feed the Blackwell FP4 tensor cores, so every one of them declines and selection falls through to MARLIN, which dequantizes weights back up before a BF16 GEMM. Forcing the issue with --moe-backend flashinfer_cutlass does not work, and the error says exactly why: "kernel does not support quantization scheme QuantKey(u8,scale(f8e4m3fn,static,GroupShape(row=1, col=16)),scale2(f32,static,per_tensor),symmetric)xNone". That trailing xNone is the missing activation quantization. This is a property of the checkpoint, not of sm_121 or of the vLLM version, and it is a different failure from the one the NVFP4-Fast recipe documents for vLLM 0.26.0. The second reason is the draft. config.json here declares mtp_num_hidden_layers: 1, and NVIDIA's own recipe.yaml carefully excludes re:^mtp.* from quantization — but the export then ships zero mtp.* tensors. We checked the weight index directly: 93,426 tensors, none of them MTP. Unsloth's build of the same base model keeps all 19 of them, at BF16, costing 1.573 GiB. So the NVFP4-Fast recipe's single biggest lever — MTP speculative decode, worth +57% there — is simply unavailable on this checkpoint as published. The two effects compound: 42.6 tok/s here against 106.5 tok/s there at a 2,048-token prompt. What this build still does well is context and prefill. The full 262,144-token window serves, and prefill runs at 5,872 tok/s at 2K and 6,431 tok/s at 8K — within a few percent of the Unsloth build, because prefill is compute-bound over a large batch of tokens where MARLIN's dequantize overhead amortizes. It is decode, one token at a time, that pays the fallback kernel on every step: ITL is 23.5 ms here against roughly 9 ms for the fast build at k=3. One thing this page cannot explain. The benchmark rows on this recipe from 2026-07-06 recorded 76.8 tok/s decode for this same checkpoint. Re-run on 2026-07-27 against the same ~/venvs/vllm (reporting 0.24.0), on the same node, decode reads 42.6 tok/s under the harness. Re-serving the same checkpoint at the short 10,240-token context those rows used and timing three warm 256-token completions gave 6.00-6.03 s each, i.e. ~42.5 tok/s — so the context flag is not the cause. (That second check is curl-level, not a full harness run: the server was lost to an ssh teardown before the harness pass completed, so no rows were emitted for it.) The engine reports a healthy configuration in each case: MARLIN, FLASH_ATTN, full CUDA-graph capture. The most likely explanation is an in-place venv or driver change between those dates that the recorded engineVersion does not distinguish, but we did not establish it. The July rows are left in place and dated rather than deleted; today's are the ones this recipe quotes.

• 1 × DGX Spark (GB10, sm_121) — no tensor parallelism, no second node
• Full 262,144-token context serves, at 1.18x concurrency with the KV pool pinned to 6 GiB
• Measured 42.6 tok/s single-stream decode at a 2,048-token prompt — no speculative decode is possible
• Unsloth's NVFP4-Fast build of the same base model measures 106.5 tok/s: prefer it unless you need the vendor artifact
• Weight-only NVFP4A16 — activations stay BF16, so every FP4 MoE backend declines and vLLM falls back to MARLIN
• --moe-backend flashinfer_cutlass is rejected at startup here, and the error names the missing activation quant
• The export ships zero mtp.* tensors despite declaring mtp_num_hidden_layers: 1 — no MTP draft to speculate from
• Prefill is competitive: 5,872 tok/s at 2K, 6,431 tok/s at 8K — the loss is decode-side only
• Weights 21.88 GiB resident, slightly MORE than the Unsloth build (20.58) despite quantizing strictly more
• KV costs ~2x per token versus the Unsloth build: 51,492 tokens/GiB against 101,606
• --kv-cache-memory-bytes 4294967296 will not start; the engine needs 5.08 GiB for one full-context sequence
• Verified 2026-07-27 on vLLM 0.24.0. The 2026-07-06 rows on this page read 76.8 tok/s and did not reproduce — see the overview

Software requirements

• 1 × DGX Spark (GB10, sm_121), NVIDIA driver ≥ 580 (CUDA-13 capable)
• vLLM 0.24.0 with torch 2.11.0+cu130
• The `hf` CLI to fetch the checkpoint (~22 GB, 6 shards)
• No patches and no sitecustomize workarounds — but also no working FP4 MoE kernel; see troubleshooting
• Last verified 2026-07-27 on vLLM 0.24.0: flags below all valid, weights 21.88 GiB, KV pool 308,955 tokens / 1.18x, decode 42.9 / 42.6 / 41.4 tok/s at 512 / 2,048 / 8,192-token prompts

Key vLLM parameters

API usage

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"nvidia/Qwen3.6-35B-A3B-NVFP4",
       "messages":[{"role":"user","content":"Explain speculative decoding in two sentences."}],
       "max_tokens":256,"temperature":0}'bash

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"nvidia/Qwen3.6-35B-A3B-NVFP4",
       "messages":[{"role":"user","content":"Write an LRU cache in Python."}],
       "max_tokens":512,"temperature":0,
       "chat_template_kwargs":{"enable_thinking":false}}'bash

Troubleshooting

Should I use this or Unsloth's NVFP4-Fast build?

The MoE backend line says MARLIN, not FLASHINFER_CUTLASS. Did I misconfigure something?

config.json says mtp_num_hidden_layers: 1 but there is no draft

vLLM refuses to start: '5.08 GiB KV cache is needed, which is larger than the available KV cache memory (4.0 GiB)'

I measured this checkpoint faster than 42 tok/s before

Is prefill slow too?

Can I graft the missing MTP head on from the Unsloth checkpoint?