Qwen3-Coder-Next NVFP4

Qwen's agentic-coding Qwen3-Coder-Next (80B MoE, ~3B active) in NVFP4 on one DGX Spark, with a 1-layer EAGLE3 speculator — 72.0 tok/s single-stream, 1.18× the no-draft baseline, at the full 262,144-token context.

Contributors

@Marinara_Waif

Qwen3-Coder-Next is an ~80B sparse MoE built for agentic coding, and it is unusually well shaped for a DGX Spark. Two properties do the work. First, it is extremely sparse: 512 routed experts with top-10 routing and a moe_intermediate_size of only 512 means roughly 3B parameters are active per token, and the Spark is bandwidth-bound, so decode speed tracks active parameters rather than total size. Second, its attention is hybrid — full_attention_interval is 4, so 36 of its 48 layers are gated-delta linear attention and only 12 hold a KV cache, with 2 KV heads apiece. Served in NVFP4 the whole thing is 43 GB on disk and 43.22 GiB resident, which leaves most of a 128 GB node unused. That second property means the usual Spark fight — how much context can I afford — does not happen here. At --gpu-memory-utilization 0.85 with no cap, vLLM sizes the KV pool at 2,461,451 tokens, which is 9.4× concurrency at the model's full 262,144-token window. For a single user, every multiple above 1.0× is memory bought and not used, so this recipe inverts the procedure: serve the full native context, then shrink the pool with --kv-cache-memory-bytes until concurrency sits just above 1.0×. Eight GiB gets 318,547 tokens — 1.22× at full context — and gives roughly 49 GiB back. The tuning question that is left is speculative decoding, and here the checkpoint gives you nothing: Qwen3-Coder-Next ships no MTP head (its weight index holds 74,391 tensors and zero mtp/nextn entries), so the free in-checkpoint draft that makes several sibling Qwen recipes fast is simply absent. The draft used here is external — togethercomputer's Aurora EAGLE3 head, a single Llama layer with its own 32,000-token draft vocabulary. It was trained against the FP8 build of this model, not the NVFP4 one served here, so the honest expectation going in was that it might accept nothing. It does not: measured position-0 acceptance holds at 62-66% across every k tried, and at k=1 decode goes from 60.8 to 72.0 tok/s, a 1.18× gain on the same node at the same 2,048-token prompt. k=1 is where it should be left. Sweeping k=1 through k=5 (see keyParams) shows tokens-per-step saturating at about 2.3 by k=3 while the per-position acceptance falls off a cliff — the fifth drafted position is accepted 4.8% of the time — so past k=3 the drafting cost outruns the extra accepted tokens and throughput drops below the no-draft baseline. Everything here runs on stock vLLM 0.24.0: no source patches, no --moe-backend override, no quantization workarounds. One caveat to read before planning around the context number. The 262,144-token window is real and was exercised at a 250,223-token prompt, but on this model it is expensive at both ends. TTFT goes 0.44 s at 2K, 6.9 s at 32K, 45.3 s at 131K and 128 s at 250K — and unlike some hybrid models, decode does not stay flat behind it: it falls 72.0 → 37.9 → 14.5 → 8.8 tok/s across the same range. The 72.0 tok/s headline is a short-prompt number and should be read as one. This is a fast interactive coding model at ordinary context lengths and a slow batch one at the top of its window; the memory to serve that window is nearly free, the time is not.

• One DGX Spark, no tensor parallelism, no second node
• ~80B total / ~3B active MoE: 512 routed experts, top-10, moe_intermediate_size 512
• 43.22 GiB resident including the draft — 58.3 of 114 usable GiB total working set
• Full 262,144-token native context served, at 1.22× concurrency on an 8 GiB KV pool
• 72.0 tok/s single-stream decode at a 2,048-token prompt, warm (60.8 with no draft)
• Decode is strongly prompt-length dependent: 72.0 tok/s at 2K, 37.9 at 32K, 14.5 at 131K, 8.8 at 250K
• TTFT at length: 6.9 s at 32K, 45.3 s at 131K, 128 s at a 250,223-token prompt
• EAGLE3 draft at k=1: 65.7% acceptance, 1.657 tokens per step
• Stock vLLM 0.24.0, FLASHINFER_CUTLASS NVFP4 MoE backend auto-selected, zero patches

Software requirements

• vLLM 0.24.0 on torch 2.11.0+cu130 (CUDA 13 — the NVFP4 MoE kernels require it)
• NVIDIA driver 580.159.03 on DGX OS / Ubuntu 24.04, GB10 (sm_121)
• Python 3.12 in a dedicated venv; the `hf` CLI for the downloads
• No source patches, no out-of-tree plugin, no --moe-backend override
• Last verified 2026-07-28 on Spark-2 against three engine builds measured back to back in one session: vLLM 0.24.0 (74.5 tok/s decode at a 2,042-token prompt), 0.25.1 (72.4) and 0.26.0 (64.7). 0.24.0 and 0.25.1 are equivalent within run-to-run noise; **0.26.0 is a real 13% decode regression on this checkpoint** because it no longer offers FLASHINFER_CUTLASS for NVFP4 MoE on sm_121. Stay on 0.24.0 or 0.25.1 — see troubleshooting.

Key vLLM parameters

API usage

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen3-Coder-Next",
    "messages": [{"role": "user", "content": "Implement an LRU cache in Python with get and put, type hints and docstrings."}],
    "max_tokens": 512,
    "temperature": 0
  }' | python3 -c 'import sys,json; print(json.load(sys.stdin)["choices"][0]["message"]["content"])'bash

# Count tokens from usage.completion_tokens, never by counting SSE frames --
# speculative decoding bundles several tokens into one chunk, so frame
# counting under-reports by exactly the acceptance rate.
curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"Qwen/Qwen3-Coder-Next","messages":[{"role":"user","content":"Write a red-black tree in Python with insert and delete."}],"max_tokens":256,"temperature":0}' \
  -w '\ntotal: %{time_total}s\n' \
  | python3 -c 'import sys,json; d=json.loads(sys.stdin.read().split("\ntotal:")[0]); print(d["usage"])'bash

curl -s http://127.0.0.1:8000/metrics | grep -v '^#' \
  | grep -E 'spec_decode_num_(drafts_total|draft_tokens_total|accepted_tokens_total|accepted_tokens_per_pos_total)'

# acceptance rate   = accepted_tokens_total / draft_tokens_total
# tokens per step   = 1 + accepted_tokens_total / drafts_total
# per-position rate = accepted_tokens_per_pos_total{position="N"} / drafts_totalbash

Troubleshooting

After upgrading to vLLM 0.26.0 the recipe still serves, but decode drops from ~74 to ~65 tok/s and TTFT gets ~30% worse.

Expected on 0.26.0, and the cause is one log line. Measured on Spark-2 on 2026-07-28, all three builds back to back with identical flags, k=1, warm, greedy, KV pinned at 8 GiB: at a 2,042-token prompt decode is 74.5 tok/s on 0.24.0, 72.4 on 0.25.1 and 64.7 on 0.26.0 (-13.1% against 0.24.0), with prefill 4,746 -> 4,756 -> 3,625 tok/s and TTFT 430 -> 429 -> 563 ms. It is not the draft: EAGLE3 acceptance measured over the same benchmark pass is 63.3% on both 0.24.0 and 0.26.0, and residency is 43.22 GiB with a 318,547-token KV pool on all three. What changes is the NVFP4 MoE backend line at startup: 0.24.0 and 0.25.1 log "Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend", 0.26.0 logs "Using 'VLLM_CUTLASS'". 0.26.0 also draws MORE power for the slower result (42.0 W vs 36.6 W average at the same scenario). Stay on 0.24.0 or 0.25.1 for this recipe; grep the log for the backend line after any engine change.

Can I force FLASHINFER_CUTLASS back on vLLM 0.26.0 with --moe-backend flashinfer_cutlass?

No. It fails at startup, in about 90 seconds, with: "ValueError: NvFp4 MoE backend 'FLASHINFER_CUTLASS' does not support the deployment configuration since kernel does not support current device cuda." 0.26.0 tightened the device check and now declines the backend on sm_121 outright — this is the checkpoint-agnostic sm_121 rejection, distinct from the checkpoint-specific one on the NVIDIA Qwen3.6-35B-A3B NVFP4 recipe, where the same backend is refused for a missing activation-quantization scheme. There is no flag that recovers the 13%; the only fix is an older engine build.

--moe-backend humming on vLLM 0.26.0 dies with pynvml.NVMLError_NotSupported after five minutes of weight loading.

HUMMING is a new NVFP4 MoE backend in the 0.25.1/0.26.0 candidate list and it cannot initialize on GB10. It passes the device check, loads all 43 GiB of weights, and then dies in humming/utils/device.py:calculate_gpu_bandwidth(), which calls pynvml.nvmlDeviceGetMaxClockInfo(handle, NVML_CLOCK_MEM) to estimate memory bandwidth for its tuning heuristics. GB10 does not expose a memory clock over NVML — `nvidia-smi -q -d CLOCK` prints 'Memory : N/A' where it prints 'SM : 3003 MHz' — so the call raises NVMLError_NotSupported and the engine core never starts. Nothing to configure; do not spend a boot on it. Verified 2026-07-28 on Spark-2.

You expect the checkpoint's own MTP head to drive speculative decode, as on several sibling Qwen MoE models, and cannot find it.

It does not exist. Qwen3-Coder-Next's model.safetensors.index.json holds 74,391 tensors and zero mtp/nextn entries, and its config.json declares no num_nextn_predict_layers or mtp_num_hidden_layers. Passing --speculative-config '{"method":"mtp",...}' has nothing to bind to. An external draft is the only route, which is why this recipe carries one.

The only published draft is named -FP8 and this recipe serves an NVFP4 body. Is that a mismatch?

It is a mismatch on paper and it works anyway — but verify it rather than assuming, because the failure mode is silent. Drafts calibrated to a target's hidden states can accept nothing when paired with a different target precision, and verification rejects garbage safely, so the only symptom is decode sinking below the no-draft baseline. Measured here against the NVFP4 body: position-0 acceptance 65.7% at k=1 and 61.9-64.5% at every k up to 5, i.e. the draft transfers cleanly. Read spec_decode_num_accepted_tokens_total from /metrics after any draft change and require it to be well above zero. The likely reason it transfers is that this is EAGLE3 with its own 32,000-entry draft vocabulary and its own lm_head, rather than a DFlash-style head that shares the target's.

Your k-sweep comes back non-monotonic — k=2 measures below both k=1 and k=3 — and you cannot tell which value to ship.

That is a noise alarm, not a finding. Speculative throughput depends on how many drafted tokens survive verification, which is content-dependent, so the variance grows with k. Measured per-request stdev at a 2,048-token prompt on this config: 0.04 tok/s with no draft (the baseline is essentially noiseless), then 1.40 at k=1, 1.59 at k=2, 4.24 at k=3, 7.78 at k=4 and 8.57 at k=5 — at k=5 that is a per-request range of 49.5-71.0 tok/s. Do not read a dip between two better neighbours as structure. The honest shape here is a clear, real climb from the no-draft baseline into a k=1..3 plateau, then an unambiguous collapse at k=4 and k=5 that takes throughput below the baseline. Ship any value in the plateau; this recipe ships k=1 because it has the tightest spread and leaves the KV pool largest.

The server starts but the KV pool is far bigger than you asked for, or the box wedges when you push the pool up.

Do not push it up on this model — there is nothing to win. Uncapped at --gpu-memory-utilization 0.85 the pool comes up at 56.71 GiB = 2,461,451 tokens, which is 9.4x concurrency at 262,144 tokens per request. vLLM's reported maximum concurrency is just pool / max-model-len, and for a single user everything above 1.0x is memory you paid for and cannot use. Shrink it with --kv-cache-memory-bytes instead. Note also that --kv-cache-memory-bytes makes vLLM skip memory profiling entirely (it logs so), which is why no page-cache purge is needed before serving this config.

Decode reads lower than the numbers here, or a 512-token prompt benchmarks slower than a 2,048-token one.

The pass is cold. Expert planes fault in from NVMe on first touch and the speculative-decode Triton kernels JIT on first use, so the first pass after any restart is not comparable. Send a few hundred tokens of traffic, then run your benchmark twice and keep the second pass. Decode should fall gently as the prompt grows — measured warm at k=1: 75.9 tok/s at 512 tokens, 72.0 at 2,048, 64.7 at 8,192. If it rises across that range, you are still measuring the JIT.

You want to compare a draft arm against the no-draft baseline and are serving them on two different Sparks to save time.

Locate the peak that way if you like, but never let a speedup ratio straddle nodes. The 1.18x quoted here has baseline and winner on the same node at the same 2,048-token prompt. As a cross-check this recipe's sweep ran k=3 on both Sparks: 71.61 tok/s on Spark-1 and 70.98 on Spark-2, 0.9% apart, with acceptance matching to the third decimal — close enough to treat the two halves of the sweep as one curve, but not something to assume without measuring.

You serve the full 262,144-token window, send a very long prompt, and the model appears to have stopped responding.

It has not — it is prefilling, and on this model that takes minutes. Measured TTFT at k=1: 0.44 s at a 2,048-token prompt, 6.9 s at 32,781, 45.3 s at 131,292 and 128 s at 250,223. Decode falls across the same range too, from 72.0 to 37.9 to 14.5 to 8.8 tok/s, so a long-context request is slow to start *and* slow to generate. Budget for it, set generous client timeouts, and do not read the 72.0 tok/s headline as a long-context number — it is measured at a 2,048-token prompt.

`pkill -f "vllm serve"` over SSH kills your own session before it kills the server.

The remote command's argv contains the pattern, so pkill matches the shell running it. Use a character class: pkill -f "[v]llm serve". Same reason a long sweep started as a plain `ssh host 'bash sweep.sh'` dies with the connection — detach it with setsid nohup.