Qwen3.6 35B-A3B — FP8

Qwen's own FP8 build of Qwen3.6 35B-A3B serves the full 262,144-token context on one DGX Spark in about 41 GiB and, unlike the NVFP4A16 build, it gets a real FP8 kernel path — vLLM picks the TRITON FP8 MoE backend, not the MARLIN fallback. It still decodes slower: 38.3 tok/s against 42.6 for NVFP4A16 and 106.5 for Unsloth's NVFP4-Fast. The reason is bytes, not kernels — the FP8 export carries 30.1 GiB of expert planes against roughly 17.5 — and like every official Qwen3.6 export it ships no MTP head.

Experiments run against this recipe

Case studies that used this recipe as their baseline — what was tried, and whether it changed the recipe.

• Does quantization actually buy you speed?Adopted into the recipe

  Designated FP8 as the recommended precision for this model on one Spark: +23% decode and +38% prefill over BF16 at half the footprint (67 -> 36 GiB) and flat power. 4-bit adds ~15% more decode and cuts weights to a third, but with clear diminishing returns — it is the pick only when the memory is the binding constraint.

Qwen publishes its own BF16 and FP8 artifacts for this model, NVIDIA publishes an NVFP4 one, and Unsloth publishes NVFP4-Fast. This page measures the FP8 build on one Spark, at the full context, against the same node, engine, flags and prompts the NVFP4 recipe used, so the four are directly comparable. The headline is that FP8 is the *well-behaved* quantization here and it still loses. Its compressed-tensors block declares channel-wise F8_E4M3 weights **and** dynamic per-token FP8 input activations, so vLLM has a real FP8 MoE kernel to reach for and takes it: 'Using TRITON Fp8 MoE backend'. Contrast NVIDIA's NVFP4A16 build, which is weight-only, gets refused by every FP4 backend and falls back to MARLIN's dequantize-then-BF16-GEMM. On paper FP8 should therefore win. Measured, it does not: 38.27 tok/s decode at a 2,048-token prompt against 42.57 for the MARLIN-bound NVFP4 build, on the same node on the same day. The reason is bytes. Decode on a Spark is bandwidth-bound and this model's routed experts are 86% of the checkpoint, so what decode pays for is expert bits: 30.06 GiB of FP8 planes here against roughly 17.5 GiB of NVFP4 planes there. An 8-bit expert plane reads about 1.8x the bytes of a 4.5-bit one on every token, and that outweighs the kernel-quality advantage. MARLIN's dequantize overhead claws back most of the gap but not all of it, which is why the two land 10% apart rather than 80%. A better kernel on more bytes loses to a worse kernel on fewer bytes — the most Spark-specific lesson on this page. The second lever is the draft, and it is absent for the same reason it is absent on NVIDIA's build. config.json declares mtp_num_hidden_layers: 1 under text_config; the export ships 62,546 tensors and not one of them matches mtp. That is now true of both official Qwen3.6 35B-A3B exports we have checked, and only Unsloth's NVFP4-Fast keeps the head (19 tensors, 1.573 GiB BF16). So the +57% that MTP speculative decode is worth on the NVFP4-Fast recipe is simply unavailable here. Engine version does not move it. We served the identical checkpoint on vLLM 0.24.0 and 0.26.0 back to back in one session with identical flags: 38.63 / 38.27 / 37.42 tok/s on 0.24.0 and 38.97 / 38.70 / 37.76 on 0.26.0, at 512 / 2,048 / 8,192-token prompts. Both pick TRITON and FLASH_ATTN, both load 35.28 GiB and both size the KV pool at 308,955 tokens. There is no version to pin on this recipe. That null result has a second use. The NVFP4 recipe records an unexplained drop on this model between 2026-07-06 (76.8 tok/s) and 2026-07-27 (42.6). The FP8 rows show the same shape — 52.3 tok/s at a 10,240-token context on 2026-07-06 against 38.3 today — so whatever changed was not specific to one checkpoint. Serving on 0.26.0 rules the engine build out as the cause, at least for the FP8 half. We still have not established what it was, and both sets of rows stay on the site, dated. What FP8 is genuinely good for on this model is prefill and a clean deployment. Prefill measured 5,122 tok/s at 2K and 6,081 at 8K, within a few percent of both NVFP4 builds, and the checkpoint loads on stock vLLM with no patch, no backend flag, no --trust-remote-code and no environment workaround. If you want the vendor artifact at a precision every tool understands, this is it. If you want speed on a Spark, download unsloth/Qwen3.6-35B-A3B-NVFP4-Fast instead.

• 1 × DGX Spark (GB10, sm_121) — no tensor parallelism, no second node, no engine patch
• Full 262,144-token context serves, at 1.18x concurrency with the KV pool pinned to 6 GiB
• Measured 38.27 tok/s single-stream decode at a 2,048-token prompt — 38.63 at 512, 37.42 at 8,192
• Real FP8 kernel path — vLLM selects the TRITON FP8 MoE backend, not the MARLIN fallback the NVFP4A16 build gets
• …and it still loses to that fallback: 38.3 tok/s here against 42.6 for NVFP4A16 on the same node, same day
• Bytes beat kernels — routed experts are 30.06 of 35.08 GiB at 8 bits, against roughly 17.5 GiB at 4.5 bits for NVFP4
• Engine version is a non-event: vLLM 0.24.0 and 0.26.0 measured within 1% of each other in the same session
• The export ships zero mtp.* tensors despite declaring mtp_num_hidden_layers: 1 — no speculative decode
• Byte-accurate memory split: 35.28 GiB resident, 30.37 GiB F8_E4M3 + 4.71 GiB BF16, ~57 of 121 GiB working set
• Prefill is competitive: 5,122 tok/s at 2K and 6,081 at 8K, within a few percent of the NVFP4 builds
• Prefer unsloth/Qwen3.6-35B-A3B-NVFP4-Fast for serving — 106.5 tok/s with its MTP head at k=3, 67.9 without
• Verified 2026-07-28 on Spark-1, on vLLM 0.24.0 and 0.26.0

Software requirements

• 1 × DGX Spark (GB10, sm_121), NVIDIA driver ≥ 580 (CUDA-13 capable)
• vLLM 0.24.0 or 0.26.0 with torch 2.11.0+cu130 — measured within 1% of each other, so either is fine
• The `hf` CLI to fetch the checkpoint (~36 GB, 10 shards)
• No patches, no --trust-remote-code, no backend flag and no sitecustomize workaround
• Last verified 2026-07-28 on Spark-1: weights 35.28 GiB, KV pool 308,955 tokens / 1.18x, decode 38.63 / 38.27 / 37.42 tok/s at 512 / 2,048 / 8,192-token prompts on 0.24.0 and 38.97 / 38.70 / 37.76 on 0.26.0

Key vLLM parameters

API usage

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3.6-35B-A3B-FP8",
       "messages":[{"role":"user","content":"Explain speculative decoding in two sentences."}],
       "max_tokens":256,"temperature":0}'bash

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3.6-35B-A3B-FP8",
       "messages":[{"role":"user","content":"Write an LRU cache in Python."}],
       "max_tokens":512,"temperature":0,
       "chat_template_kwargs":{"enable_thinking":false}}'bash

Troubleshooting

FP8 gets a real kernel and NVFP4A16 falls back to MARLIN. Why is FP8 the slower one?

config.json says mtp_num_hidden_layers: 1 but there is no draft

Should I pin vLLM 0.24.0 like the NVFP4-Fast recipe does?

I measured this checkpoint faster than 38 tok/s before

Is prefill slow too?

The engine warns that --kv-cache-memory-bytes makes it skip memory profiling

Which Qwen3.6 35B-A3B build should I actually download?