Qwen3.6 27B NVFP4 (NVIDIA ModelOpt)

Qwen3.6 27B — a dense 27B with hybrid GDN/full attention and a vision tower — in mixed NVFP4 on one DGX Spark, with the checkpoint's own MTP head speculating 5 tokens ahead: 31.8 tok/s single-stream, 2.6× the no-draft baseline, at the full 262,144-token context.

Contributors

@joemuller

Qwen3.6 27B is a dense 27B with a hybrid attention stack — 48 GDN linear-attention layers interleaved with 16 full-attention layers — plus a 27-block vision tower, and NVIDIA's ModelOpt build quantizes it unevenly on purpose: the MLP and lm_head go to NVFP4 (W4A16, group 16), every attention projection stays FP8, and the vision tower, embeddings and the MTP head stay BF16. At 21 GB it fits one Spark with room to spare, so nothing about this recipe is a memory fight. The fight is bandwidth. Because the model is dense, every decoded token re-reads essentially all of it: 17.61 GB, of which the MLP is 44% and the GDN projections another 26%. Against GB10's 273 GB/s that sets a hard ceiling near 15.5 tok/s, and the measured no-draft rate of 12.14 tok/s is a perfectly healthy 78% of it — nothing is misconfigured, the model is just expensive per token in a way a sparse MoE of the same file size is not. Two consequences shape everything below. First, decode is almost flat in context (12.19 tok/s at a 512-token prompt, 12.03 at 8192) because the KV traffic is noise next to the weights. Second, and this is the whole recipe, speculative decoding pays here far better than it does on a cheap MoE: the checkpoint ships its own MTP head that reads only 0.85 GB per draft step, 4.8% of a target pass, so drafting several tokens and verifying them in one pass is a very good trade. Measured across 13 single-stream runs per setting, decode climbs cleanly and tightly from 12.14 tok/s with no draft to 20.00 at k=1, 26.48 at k=2 and 29.61 at k=3 — and then flattens. Beyond k=3 the curve is essentially level and the measurement gets noisy: k=4, k=5 and k=6 pool to 33.10, 31.75 and 35.58 tok/s with 95% intervals of roughly ±1.2 to ±1.5, wide enough that their ordering is not stable between runs or between our two Sparks. This recipe therefore ships k=5 and publishes 31.75 tok/s — 2.6× the no-draft baseline — which is the *lowest* of the three plateau values rather than the most flattering one, because any of k=4 to k=6 is a defensible choice and pretending to a precise optimum here would be false precision. Three honest limits. The figure is the greedy, code-workload number the site's spec-decoding profile prescribes; on open-ended prose at temperature 0.7 acceptance falls to roughly 37-43% and decode to about 23 tok/s, still nearly 2× baseline but not 2.6×. Concurrency is not supported at this setting — four simultaneous requests kill the engine with a CUDA illegal memory access, which is fine for the single-user target this is tuned for but must be known before putting it behind anything that fans out. And it is a throughput claim only: this build's quality is unevaluated here, and the checkpoint asks for fp8 KV while shipping no k/v scaling factors, which is worth an eval before trusting very long answers.

• 31.75 tok/s single-stream at a 2048-token prompt, warm — 2.6× the 12.14 tok/s no-draft baseline on the same node
• The full 262,144-token context, served — measured at a 131,292-token prompt
• Speculative decoding from the checkpoint's own BF16 MTP head — no draft to download, no vLLM patch
• k=1→3 is a clean, tight climb (20.00 / 26.48 / 29.61 tok/s); past k=3 the curve flattens and k=4-6 are within noise of each other
• max-num-seqs 4, not 2: dropping it costs 9.6% of decode and saves no memory
• Every token reads 17.61 GB, so the roofline is ~15.5 tok/s and the no-draft baseline is 78% of it
• Working set ~44 GiB of 114 usable — memory is not the constraint on this model
• Decode is nearly flat in prompt length (12.19 at 512, 12.03 at 8192 with no draft)
• Single-user only: 4 concurrent requests crash the engine with a CUDA illegal memory access
• vLLM 0.26.0 serves it with no port, no custom backend flag and no quantization flag
• Runs on one Spark; the second node is free for something else

Software requirements

• 1 × DGX Spark (GB10, sm_121), NVIDIA driver ≥ 580 (CUDA-13 capable)
• vLLM 0.26.0 with torch 2.11.0+cu130 and flashinfer-python 0.6.14 — `qwen3_5_mtp` support (arch `Qwen3_5MTP`) landed natively, so no patch is needed for the draft head
• transformers 5.14.1 (the checkpoint declares transformers_version 5.7.0)
• The `hf` CLI to fetch the checkpoint (~21 GB)
• No patches, no out-of-tree model, no --moe-backend, no --quantization flag

Key vLLM parameters

API usage

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6-27b",
    "messages": [{"role": "user", "content": "Explain unified memory on GB10 in three sentences."}],
    "max_tokens": 400,
    "temperature": 0.7,
    "chat_template_kwargs": {"enable_thinking": false}
  }' | python3 -c 'import json,sys;print(json.load(sys.stdin)["choices"][0]["message"]["content"])'bash

# Count tokens from usage, never by counting SSE frames — speculative
# decoding bundles several tokens into one chunk, so frame-counting
# under-reports decode by roughly the acceptance rate.
curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3.6-27b","messages":[{"role":"user","content":"Write a red-black tree in Python with insert and delete."}],"max_tokens":512,"temperature":0,"stream":true,"stream_options":{"include_usage":true},"chat_template_kwargs":{"enable_thinking":false}}' \
  | grep -o '"completion_tokens":[0-9]*' | tail -1bash

curl -s http://127.0.0.1:8000/metrics | grep -E '^vllm:spec_decode' | grep -v '^#'

# accepted / draft_tokens                     = mean acceptance
# ..._per_pos_total{position="N"} / ..._num_drafts_total = per-position

# At k=5 on a greedy code workload, expect roughly:
#   mean ~0.70, per-position ~0.936 / 0.815 / 0.681 / 0.581 / 0.484bash

Troubleshooting

The engine dies during startup with NO traceback, right after `torch.compile took NN s`. Memory looks fine, there is no OOM in kern.log, and no core dump. It looks exactly like a segfaulting sm_121 kernel.

It is not a kernel. vLLM 0.26.0 ends its memory profiling with `assert init_snapshot.free_memory >= free_gpu_memory` — free GPU memory may only *decrease* while profiling. That holds for discrete VRAM and is false on GB10, where mem_get_info reports the whole system pool and rises whenever the OS reclaims page cache. Ours aborted having *gained* 2 GiB: `Error in memory profiling. Initial free memory 89.17 GiB, current free memory 91.27 GiB`. Fix: pass --kv-cache-memory-bytes, which is an early return before the assert (the log then says "reserved 14.0 GiB ... and skipped memory profiling"); profile_run still executes, so kernels and graphs still compile and you lose nothing but auto-sizing. Two corollaries. The AssertionError text often does not reach the log at all — the API server's 35 s startup timeout wins the race and truncates it — so grep for `Error in memory profiling` explicitly rather than concluding it is absent. And do not sit in a loop running `free -g` / `du` / `ps` over SSH while a serve is profiling; churning page cache can *cause* this.

Which num_speculative_tokens should I use?

Anything from 4 to 6; we ship 5. Pooled over 13 single-stream runs per setting at 2048/256 greedy, the medians and 95% intervals are: no draft 12.14 ±0.01, k=1 20.00 ±0.14, k=2 26.48 ±0.30, k=3 29.61 ±0.50, k=4 33.10 ±1.13, k=5 31.75 ±1.17, k=6 35.58 ±1.48. Two lessons. First, do not stop at the first k that improves — k=1 leaves more than half the available gain on the table. Second, do not believe a precise peak in the flat region: past k=3 the intervals overlap and the ordering is unstable, and we measured k=6 as both the best setting on one Spark (35.58) and clearly the worst on the other (32.78). Mean acceptance is the wrong thing to optimise on regardless — it falls monotonically the whole way (94.2% at k=1 to 60.7% at k=6) while throughput rises. Per-position acceptance is the more useful signal: at k=5 it runs about 0.936 / 0.815 / 0.681 / 0.581 / 0.484, and the marginal position is still near a coin flip, which is why the curve has flattened rather than turned down.

I ran the benchmark once and got a very different number from the one published.

Expected at high k, and it is the reason this recipe publishes a pooled median rather than a best run. Under speculation, throughput depends on how many drafted tokens survive verification, which depends on the text being generated — and the harness gives every request a unique cache-busting prefix, so each request generates different text. Measured per-request spread at 2048/256: 11.3% of the median at k=3, but 20-25% at k=4 through k=6 (stdev 2.08, 2.15 and 2.71 tok/s). Four separate passes of the identical k=5 config on the identical node produced 31.18, 33.87, 33.89 and 35.69 tok/s. By contrast the no-draft baseline has a stdev of 0.01 tok/s, because its work per token is fixed. So: run many requests, compare medians with intervals, and never let a single pass decide a tuning question.

I get nowhere near 30 tok/s on my own prompts.

Almost certainly workload and temperature, not configuration. Acceptance is strongly content-dependent and collapses as sampling temperature rises. The published figure is the site's spec-decoding convention: greedy (temperature 0) on a code-generation prompt. The same config on open-ended prose at temperature 0.7 shows per-position acceptance around 0.741 / 0.543 / 0.358 / 0.296 / 0.222 — mean roughly 37-43% — and generates at about 23 tok/s. That is still nearly 2× the no-draft baseline, so speculation is well worth keeping, but a chat user should expect the low twenties rather than the low thirties. Also confirm thinking is off; a reasoning preamble is real output tokens.

The engine dies with `CUDA error: an illegal memory access was encountered` as soon as more than one request is in flight.

Speculative decoding on this model does not survive concurrency on vLLM 0.26.0. Four simultaneous requests at k=5 killed the engine outright (`EngineDeadError`, every subsequent request 500s and the server shuts down); the fault surfaces in the FlashInfer metadata builder at `backend.py:510 seq_lens_cpu -> self.seq_lens.to('cpu')`, but CUDA errors are reported asynchronously so the real fault is an earlier kernel, and the dumped config points at the draft (`eagle_head`) path. It is specifically the draft: with the same server flags minus --speculative-config, four concurrent requests all complete normally (HTTP 200, nothing in the log). Concurrency 1 is completely solid — hours of sweeps, a dozen serves, hundreds of requests, no faults. So this recipe is safe for the single-user workstation case it is tuned for, but do not put it behind anything that fans out: either keep concurrency at 1, or drop --speculative-config and accept 12 tok/s. We did not establish whether a smaller k is safe under concurrency — do not assume lowering it is a workaround without testing it yourself. This is also why the published numbers use the spec-decoding profile, which has no concurrency row by design.

vLLM logs "Your GPU does not have native support for FP4 computation ... Weight-only FP4 compression will be used leveraging the Marlin kernel. This may degrade performance for compute-heavy workloads."

Expected on sm_121 and harmless here. The warning is about compute-bound work; single-stream decode is bandwidth-bound, and the measured 12.14 tok/s no-draft rate is 78% of this checkpoint's 15.5 tok/s bandwidth roofline, so the dequant path is not what is limiting you. Do not go looking for a faster NVFP4 backend to fix a decode number — check the per-token byte count first (this checkpoint reads 17.61 GB per token, 44% of it MLP).

"CUDAGraphMode.FULL_AND_PIECEWISE is not supported with spec-decode for attention backend FlashInferBackend ...; setting cudagraph_mode=PIECEWISE"

Expected, and worth understanding before you quote a speedup. The draft arm runs on PIECEWISE graphs while a no-draft baseline gets FULL, so the 2.9× published here is measured *net of* that handicap and is therefore conservative rather than flattering. Nothing to fix; do not add --enforce-eager trying to make the arms match, as that costs both of them.

Decode barely changes when I raise or lower --max-model-len.

Correct, and it is a property of the architecture rather than a bug. This is a dense model whose weights dominate per-token traffic so completely that KV reads are noise: with no draft it measures 12.19 tok/s at a 512-token prompt, 12.14 at 2048 and 12.03 at 8192. Prefill and TTFT are where prompt length shows up, not decode. Spend the flat decode curve on context — it is the cheap axis on this model.

The KV pool comes up far larger than max-model-len and reports several times the concurrency I need.

That is what an uncapped pool does on a hybrid-attention model, and it is waste rather than headroom for a single user — vLLM's "maximum concurrency" is just pool ÷ max-model-len, and every multiple above about 1.0x is memory you paid for and cannot use. Because only 16 of 64 layers cache anything, this model's window is already affordable many times over. Invert the usual procedure: set --max-model-len to the model's full 262144 first, then *shrink* the pool with --kv-cache-memory-bytes until concurrency is roughly 1.0-1.5x, and hand the rest back. 14 GiB gives 1.48x. Keep some margin rather than sitting at exactly 1.00x, because the pool varies a few percent between boots and the server refuses to start if it comes up smaller than max-model-len.

The KV pool shrank after I changed k.

Expected — the draft's state comes out of the same pool. At a fixed 14 GiB this config held 447,637 tokens with no draft, 414,553 at k=1, 386,698 at k=5 and 380,252 at k=6. So re-read `GPU KV cache size` after every change to k, and do not assume a context length that fit at one k still fits at a larger one.

My benchmark records draftAcceptanceRate as null even though speculation is clearly working.

The harness reads draft_n / draft_n_accepted out of the SSE `timings` object, which is a llama.cpp convention; vLLM does not put draft counts in the stream, so every vLLM spec-decode row comes back null. Scrape the Prometheus endpoint instead: `curl -s localhost:8000/metrics | grep '^vllm:spec_decode'`. Take the `_per_pos_total` series divided by `_num_drafts_total` — per-position is the number that actually tells you where k should stop, and the counters are cumulative so snapshot before and after.

A benchmark reports far fewer tokens than max_tokens, or decode looks impossibly high.

Two separate traps. If you are counting SSE frames, stop — speculative decoding bundles multiple accepted tokens into a single streamed chunk, so frame-counting under-reports decode by about the acceptance rate. Read usage.completion_tokens differentials instead. And if throughput looks too good, check the text: a repetition loop is trivially predictable, so it drives acceptance toward 100% and inflates tok/s. This build passes that gate — six open-ended 1500-token prose generations showed a maximum repeated 8-gram count of 1 and a mean unique-word ratio of 0.528 — but re-check it yourself after any change to the draft or the quantization.

Serving with a `--speculative-config` that names `qwen3_5_mtp` behaves oddly, or I am unsure the draft loaded at all.

Write "mtp": vLLM accepts `qwen3_5_mtp` but rewrites it internally, and the canonical spelling avoids the confusion. Then confirm from the log rather than assuming — you want `Resolved architecture: Qwen3_5MTP`, `Loading drafter model...` and `Detected MTP model. Sharing target model embedding weights with the draft model` (plus the same line for lm_head). If the architecture resolves to Qwen3_5ForConditionalGeneration instead, no draft was loaded and you are measuring the 12 tok/s baseline. Also verify the 15 `mtp.*` tensors are present in the checkpoint's index — the config declares mtp_num_hidden_layers: 1 whether or not the weights shipped.

A very long prompt takes a long time before the first token.

This is the honest cost of the long window, and it is worth seeing the curve before you promise anyone 262K. Prefill is compute-bound at roughly 750-1,100 tok/s, and decode barely moves, so length shows up almost entirely as time-to-first-token. Measured on the shipped config: 2,041-token prompt → 31.2 tok/s decode, TTFT 1.9 s; 8,206 → 28.7 tok/s, 7.5 s; 32,777 → 30.5 tok/s, 32.2 s; 131,292 → 29.5 tok/s, 175.6 s. So a 131K prompt is nearly three minutes before the first token appears, while generation itself stays around 30 tok/s. The context is real and it serves — but it is a throughput-friendly, latency-hostile window, and the recipe's headline decode number is measured at 2048 tokens, not at 262K.

The log says "Setting attention block size to 1568 tokens to ensure that attention page size is >= mamba page size" and pads the mamba page.

Normal for a hybrid model, not a misconfiguration. The 48 GDN linear-attention layers keep a recurrent state whose page must line up with the full-attention layers' KV pages, so vLLM picks an attention block size that makes them equal (and notes a 0.13% padding). Nothing to tune; it is why the effective KV cost per token is 32.8 KiB rather than a round number.

Graph capture reports a negative memory delta ("took -0.01 GiB").

A unified-memory accounting artifact, not a leak or an error. Free memory on GB10 is system-wide and moves for reasons unrelated to vLLM, so a short measurement window can straddle an OS reclaim and come out negative. Ignore the sign; if you need the real graph-pool cost, compare steady-state `free -g` with and without cudagraphs. It is the same underlying property that makes the memory-profiling assert above misfire.

Should I run this across both Sparks with TP2?

Not for capacity — 20.8 GiB sits on a 114 GiB node with the full context and ~70 GiB to spare, and TP2 adds cross-node latency to every token plus the RoCE/Gloo setup this recipe otherwise avoids entirely. The one genuinely interesting reason to try it is that TP2 would halve the bytes each node reads per token, which is the only way to move the 15.5 tok/s roofline that bounds a dense model. That is untested here. Also treat our two Sparks as possibly not interchangeable: the same k=3 config measured 29.20 tok/s on one and 30.57 on the other. That gap is suggestive rather than proven — it came from single passes, and we later found per-request spread at these settings is large enough to produce it by chance — but the discipline holds either way, because you cannot rule it out cheaply: never pair a baseline measured on one node with a tuned config measured on the other.