Qwen3.6 35B-A3B NVFP4 (Unsloth Fast)

Serve the full 262,144-token context of Qwen3.6 35B-A3B on ONE DGX Spark at 106 tok/s single-stream, using the checkpoint's own MTP head for speculative decode at k=3 — a 57% gain over the same config with no draft. Nothing needs patching and nothing is tight: the whole working set is 36 of 114 usable GiB.

Contributors

@Fogle@joemuller

Qwen3.6 35B-A3B is a 35B sparse MoE (about 3B active per token) with a hybrid attention stack and a vision tower, and it fits comfortably on a single DGX Spark once quantized. This recipe serves Unsloth's NVFP4-Fast build, which quantizes only the expert FFNs to NVFP4 and leaves attention at FP8, and drives speculative decode from the MTP head that ships inside the checkpoint. The shape of the config is set by one fact: on this model KV is nearly free, so context costs almost nothing and the tuning question is not 'how much context can I afford' but 'how many speculative tokens should I draft'. Sweeping that — no draft, then k=1 through k=4 with every other flag held identical — is the whole recipe. Decode climbs 67.9 → 83.7 → 93.4 → 106.5 tok/s and then falls back to 97.2 at k=4, so k=3 is the peak. Everything runs on stock vLLM 0.24.0 with no source patches, no --moe-backend override, and no quantization workarounds. One caveat worth reading before you plan around the context number: the 262K window is real and was exercised at 250,223 tokens, but it is prefill-bound. Decode decays gently with prompt length (100.1 tok/s at 8K, 94.4 at 33K, 75.5 at 131K, 62.9 at 250K) while TTFT does not — it goes 1.2 s → 6.1 s → 46 s → 140 s, because prefill throughput falls from 6,567 to 1,791 tok/s as the attention window grows. A quarter-million-token prompt costs well over two minutes before the first token appears. That is a fine trade for a long document you ask several questions about, and a bad one for interactive work.

• 1 × DGX Spark (GB10, sm_121) — no tensor parallelism, no second node
• Full 262,144-token context, actually served — largest prompt measured was 250,223 tokens
• But long prompts are prefill-bound: TTFT is 1.2 s at 8K and 140 s at 250K — see the curve below
• Measured 106.5 tok/s single-stream decode at k=3, against 67.9 tok/s with no draft — +57%
• MTP speculative decode from the checkpoint's own draft layer: 79% acceptance, 3.37 tokens/step
• The full k sweep is published below — k=4 is slower than k=3, so the peak is real, not the end of the range
• Mixed precision: NVFP4 expert planes, FP8 attention/lm_head, BF16 embeddings and vision tower
• vLLM auto-selects the FLASHINFER_CUTLASS NVFP4 MoE backend on sm_121 — do not force marlin
• Whole working set is ~36 of 114 usable GiB; ~78 GiB stays free
• No vLLM patch needed — the MTP layer is unquantized in the checkpoint and loads as-is
• Warm start ~4 min; always warm up before measuring — cold, decode reads ~25% low

Software requirements

• 1 × DGX Spark (GB10, sm_121), NVIDIA driver ≥ 580 (CUDA-13 capable)
• vLLM 0.24.0 with torch 2.11.0+cu130 — CUDA 13 is required for the FP4 MoE kernels; a cu129 build fails outright
• The `hf` CLI to fetch the checkpoint (~24 GB)
• No patches, no custom backend flags, no sitecustomize workarounds

Quick start

1. 1

   Free the node first

   Anything still holding unified memory — a previous vLLM, a leftover Ray cluster — comes out of the same pool. Check before you start, and drop the page cache the checkpoint download left behind, or vLLM reports 'free memory less than desired GPU memory utilization' at startup.

   pgrep -af 'vllm serve|ray::' || echo 'node is clear'
   ray stop --force 2>/dev/null || true
   python3 ~/Dev/vLLM-Moet/spark/purge-cache.py ~/models/hf
   free -gbash

2. 2

   Download the checkpoint

   5 safetensors shards, ~24 GB. Only this node needs it — there is no second rank to mirror to.

   hf download unsloth/Qwen3.6-35B-A3B-NVFP4-Fast \
     --local-dir ~/models/hf/Qwen3.6-35B-A3B-NVFP4-Fastbash

   Verify 5 shards and zero *.incomplete files before serving; a partial tree fails deep into load.

3. 3

   Serve with MTP speculative decode at k=3

   This is the shipped configuration. Note what is NOT here: no --moe-backend, no --tensor-parallel-size, no --enforce-eager, no quantization flag. vLLM reads the compressed-tensors config out of the checkpoint and picks the backend itself.

   PATH="$HOME/venvs/vllm/bin:$PATH" VLLM_USE_DEEP_GEMM=0 \
   TORCHINDUCTOR_COMPILE_THREADS=2 MAX_JOBS=4 \
   vllm serve ~/models/hf/Qwen3.6-35B-A3B-NVFP4-Fast \
     --served-model-name unsloth/Qwen3.6-35B-A3B-NVFP4-Fast \
     --max-model-len 262144 \
     --kv-cache-memory-bytes 4294967296 \
     --gpu-memory-utilization 0.85 \
     --max-num-seqs 4 \
     --max-num-batched-tokens 8192 \
     --port 8000 \
     --speculative-config '{"method":"mtp","num_speculative_tokens":3}'bash

   Write "mtp", not "qwen3_5_mtp" — vLLM accepts the latter but logs 'deprecated and replaced with mtp' and rewrites it anyway.

4. 4

   Confirm what it actually chose

   Three lines in the log tell you the run is healthy. If the MoE backend line says EMULATION, or the KV pool is far off 343,699 tokens, stop and read the troubleshooting section rather than benchmarking.

   grep -E "NvFp4 MoE backend|attention backend|Model loading took|GPU KV cache size|Maximum concurrency" /tmp/vllm.logbash

   Expect FLASHINFER_CUTLASS, FLASHINFER, 22.15 GiB, 343,699 tokens, 1.31x.

5. 5

   Warm it up before you trust any number

   The first requests after a start pay Triton JIT for the speculative-decode kernels (eagle_prepare_next_token_padded_kernel, rejection_greedy_sample_kernel and friends — vLLM warns about each one). This is not a small effect and it is not optional: our first pass at k=2 measured 76.3 tok/s on the 512-token scenario and 91.2 on the identical rerun.

   for i in 1 2 3; do
     curl -s http://127.0.0.1:8000/v1/chat/completions \
       -H 'Content-Type: application/json' \
       -d '{"model":"unsloth/Qwen3.6-35B-A3B-NVFP4-Fast",
            "messages":[{"role":"user","content":"Write an LRU cache in Python."}],
            "max_tokens":256,"temperature":0}' > /dev/null
   donebash

6. 6

   Check the memory you actually used

   Read this with the API up and idle, not during weight loading — mid-load the KV pool does not exist yet and the figure flatters you by exactly the amount you are about to allocate.

   free -g
   # expect ~40 GiB used of 121 — weights 22.15 + KV 4 + overhead ~10 + OS ~4bash

Key vLLM parameters

API usage

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "unsloth/Qwen3.6-35B-A3B-NVFP4-Fast",
    "messages": [{"role": "user", "content": "Explain MoE routing in three sentences."}],
    "max_tokens": 512
  }' | jq -r '.choices[0].message.content'bash

import json, time, urllib.request, uuid

body = {
    "model": "unsloth/Qwen3.6-35B-A3B-NVFP4-Fast",
    "messages": [{"role": "user",
                  "content": f"[{uuid.uuid4().hex[:8]}] Write an LRU cache in Python."}],
    "max_tokens": 256, "temperature": 0, "stream": True,
    "stream_options": {"include_usage": True}, "ignore_eos": True,
    "chat_template_kwargs": {"enable_thinking": False},
}
req = urllib.request.Request("http://127.0.0.1:8000/v1/chat/completions",
                             data=json.dumps(body).encode(),
                             headers={"Content-Type": "application/json"})
t0 = time.perf_counter(); t_first = None; usage = None
with urllib.request.urlopen(req) as res:
    for raw in res:
        line = raw.decode().strip()
        if not line.startswith("data: "):
            continue
        if line[6:] == "[DONE]":
            break
        ch = json.loads(line[6:])
        usage = ch.get("usage") or usage
        if ch.get("choices") and ch["choices"][0]["delta"].get("content") and not t_first:
            t_first = time.perf_counter()
t_end = time.perf_counter()
print(f"ttft {1000*(t_first-t0):.0f} ms")
print(f"decode {(usage['completion_tokens']-1)/(t_end-t_first):.1f} tok/s")python

grep 'SpecDecoding metrics' /tmp/vllm.log | tail -1
# Mean acceptance length: 3.37 ... Per-position acceptance rate: 0.917, 0.784,
# 0.668, Avg Draft acceptance rate: 79.0%bash

Troubleshooting

Decode looks ~25% slower than expected, and the 512-token prompt benchmarks slower than the 2048-token one.

You are measuring cold, and on this config that is a big enough effect to reverse a conclusion. Speculative decoding brings its own Triton kernels that JIT on first use — vLLM logs 'Triton kernel JIT compilation during inference: eagle_prepare_next_token_padded_kernel' (and rejection_greedy_sample_kernel, _causal_conv1d_update_kernel, fused_sigmoid_gating_delta_rule_update_kernel). Our first k=2 pass measured 76.3 / 92.1 / 91.0 tok/s across the three scenarios; the identical rerun gave 91.2 / 93.4 / 91.7. A first scenario slower than the second is the tell. Worse, an early cold sweep made k=3 look erratic (probes scattered 79-105 tok/s) and nearly got it rejected in favour of k=2 — warm, k=3 is both the fastest and among the tightest (104.3-106.5, ~1% spread). Send a few hundred tokens of traffic after every restart before recording anything.

Which num_speculative_tokens should I use?

k=3 on this model. Measured warm at 2048/256: no draft 67.9 tok/s, k=1 83.7, k=2 93.4, k=3 106.5, k=4 97.2. Acceptance falls as you draft deeper (93.2% at k=1, 86.3% at k=2, 79.0% at k=3, 67.9% at k=4) but through k=3 the extra accepted tokens still outrun the wasted draft compute. At k=4 they do not — the fourth position is accepted 49% of the time. Do not assume more drafting is monotonically better, and do not stop at the first k that improves.

The context is 262K but a long prompt takes minutes to answer.

Expected — this config is prefill-bound at length, and the KV headroom that makes 262K affordable does nothing for prefill compute. Measured at k=3, single stream: 8,208-token prompt → 1.25 s TTFT at 6,567 tok/s prefill; 32,780 → 6.1 s at 5,408; 131,292 → 46.4 s at 2,832; 250,223 → 139.7 s at 1,791. Decode holds up far better than TTFT over the same range (100.1 → 62.9 tok/s). If you need interactive latency, keep prompts under ~32K; the long window is for documents you load once and then ask several questions about. Note prefill throughput more than halves by 131K, so the cost is superlinear in prompt length, not linear.

'free memory less than desired GPU memory utilization' at startup, even though nothing else is running.

Page cache from the 24 GB checkpoint download is squatting in the unified pool. Run `python3 ~/Dev/vLLM-Moet/spark/purge-cache.py ~/models/hf` before serving. Check `free -g` — a large buff/cache figure with little 'free' is the signature.

vLLM logs "method `qwen3_5_mtp` is deprecated and replaced with mtp".

Harmless — it rewrites the method and proceeds. Write "method": "mtp" to keep the log clean. Both produce an identical SpeculativeConfig(method='mtp', num_spec_tokens=N).

Should I force --moe-backend marlin, as other DGX Spark NVFP4 write-ups say?

No. On vLLM 0.24.0 + torch 2.11.0+cu130 this checkpoint auto-selects FLASHINFER_CUTLASS out of ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED', 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'EMULATION'] and serves correctly. The TRT-LLM fused-MoE autotuner logs 'Skipped N unsupported tactic(s)' for trtllm::fused_moe::gemm1/gemm2 — that is it pruning tactics with no sm_121 kernel and keeping the ones that work, not an error. The marlin-only advice belongs to a different checkpoint and build.

The KV pool comes up around 7.6 million tokens and 29x concurrency. Is that a misconfiguration?

No — that is what this model does when you let it. Only 10 of 40 layers are full attention (full_attention_interval: 4) with 2 KV heads at head_dim 256, so KV runs ~10.3 KiB/token. The pool is genuinely that big; it is just useless to one user. Pin it with --kv-cache-memory-bytes 4294967296. The usual Spark instinct — raise --max-model-len until concurrency approaches 1.0x — is already satisfied here at the model's full native context, so the leftover memory simply goes unused, and that is fine.

The log says "Setting attention block size to 2128 tokens to ensure that attention page size is >= mamba page size".

Expected on this architecture, not a warning to act on. The linear-attention (mamba-style) layers carry a per-sequence state whose page must fit alongside the attention pages, which forces the unusually large block size. It is why per-token KV lands where it does.

A benchmark reports far fewer tokens than max_tokens, or decode looks impossibly high.

Two separate traps on this model. First, it is a thinking model: without "chat_template_kwargs": {"enable_thinking": false} the output budget is spent on reasoning prose, and thinking tokens arrive as reasoning_content rather than content, so a naive client sees an empty stream. Second, never count SSE frames to get token counts — speculative decoding emits ~3.4 tokens per chunk at k=3, so frame counting under-reports decode by that factor. Read usage.completion_tokens.

Engine dies at load with a shape mismatch on an mtp.* parameter.

Should not happen with this checkpoint — the quant config's `ignore` list contains `re:^mtp.*`, so the MTP layer is stored unquantized, and vLLM 0.24.0 has native Qwen3_5MoeMTP support. If you hit it on a different Qwen3.6 quant, check whether that repo quantized the MTP weights; a 4-bit packed parameter is half the logical width, which reads exactly like a tensor-parallel shard but is not.