Qwen AgentWorld 35B-A3B — NVFP4

Qwen's agentic AgentWorld 35B-A3B (35B MoE, ~3B active) in NVFP4 on one DGX Spark, serving the full 262,144-token context at 83.1 tok/s single-stream with Qwen3.6's MTP head grafted in.

Qwen AgentWorld 35B-A3B is a 35B sparse MoE — 256 routed experts, top-8, about 3B active per token — on the Qwen3.5 hybrid attention stack: 30 of its 40 layers are gated-delta linear attention and 10 are full attention, at a 262,144-token native window. This recipe serves a third-party NVFP4 export of it (nvfp4-pack-quantized, group_size 16, static-minmax activation observer) on a single Spark, and the reason to bother is decode speed: 64.09 tok/s single-stream at a 2048-token prompt, against 30.4 for the same model, same node, same flags in BF16. The 2.11x is unusually large for a quant, and it is large for a legible reason worth checking on any vendor quant before you download it — the ignore list. This one has 191 entries and 189 of them are things that never run here: the 27-block vision tower and the tiny per-layer router gates. Only lm_head is a real carve-out. That means the two things that dominate bytes read per generated token — the routed experts AND the full-attention projections — are both 4-bit, which is exactly the case where a quant converts into decode rather than only into disk. Summed from the safetensors headers, a decode step reads 2.42 GB here against 5.89 GB in BF16, a byte ratio of 2.44 that lands within 13% of the measured 2.11x speed ratio. Nothing else needs explaining. Two practical notes before you copy the serve command. First, this checkpoint ships the vision tower — 333 visual.* tensors are present in the file — so it loads on stock vLLM with no patch and you must NOT pass --language-model-only, which would skip constructing a tower the weights then have nowhere to go. Second, the speedup shrinks as the prompt grows, because the quant compresses weights and not the KV cache: it is 2.12x at a 511-token prompt, 2.11x at 2,042 and 2.06x at 8,206, and by a 250,223-token prompt decode is down to 27.46 tok/s in absolute terms. If your workload is long-context repo prompts rather than chat, size your expectations off the tail of that curve, not its head.

• 1 × DGX Spark (GB10, sm_121) — no tensor parallelism, no second node
• Measured 83.08 tok/s single-stream decode at a 2048-token prompt, warm, greedy — 1.29x the same checkpoint with no draft (64.54, same node, same session) and 2.73x the model in BF16
• Speculative decoding on a model that ships no draft: Qwen3.6-35B-A3B's 19 mtp.* tensors graft in unmodified, 1.7 GB of file surgery, no training
• Loads on stock vLLM 0.24.0: no patch, no --language-model-only, no --moe-backend flag
• Full 262,144-token context served, proven with a real 261,081-token request (103.7 s to first token, 27.86 tok/s after)
• Weights 20.53 GiB; whole serving working set ~39 of 114 usable GiB, so two thirds of the Spark stays free
• A six-real-entry ignore list: experts, full attention and the linear-attention projections are all 4-bit — only lm_head, the router gates and an unused vision tower are not
• Decode step reads 2.42 GB against BF16's 5.89 GB — measured from the safetensors headers, and it explains the whole speedup
• No speculative draft available — the config advertises mtp_num_hidden_layers: 1 but the checkpoint ships zero MTP tensors
• Weight load 125 s from a single 20.4 GiB safetensors file; full startup ~4 min

Software requirements

• 1 × DGX Spark (GB10, sm_121), NVIDIA driver ≥ 580 (CUDA-13 capable — the FP4 MoE kernels require it)
• vLLM 0.24.0 with torch 2.11.0+cu130 (verified 2026-07-27). Stock, unpatched
• The `hf` CLI to fetch the checkpoint (~21.9 GB)
• ~25 GB of free disk on the serving node

Key vLLM parameters

API usage

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen-AgentWorld-35B-A3B-NVFP4",
    "messages": [{"role": "user", "content": "Explain MoE routing in three sentences."}],
    "max_tokens": 512
  }' | jq -r '.choices[0].message.content'bash

import json, time, urllib.request, uuid

body = {
    "model": "Qwen-AgentWorld-35B-A3B-NVFP4",
    "messages": [{"role": "user",
                  "content": f"[{uuid.uuid4().hex[:8]}] Write an LRU cache in Python."}],
    "max_tokens": 256, "temperature": 0, "stream": True,
    "stream_options": {"include_usage": True}, "ignore_eos": True,
}
req = urllib.request.Request("http://127.0.0.1:8000/v1/chat/completions",
                             data=json.dumps(body).encode(),
                             headers={"Content-Type": "application/json"})
t0 = time.perf_counter(); t_first = None; usage = None
with urllib.request.urlopen(req) as res:
    for raw in res:
        line = raw.decode().strip()
        if not line.startswith("data: "):
            continue
        if line[6:] == "[DONE]":
            break
        ch = json.loads(line[6:])
        usage = ch.get("usage") or usage
        if ch.get("choices") and ch["choices"][0]["delta"].get("content") and not t_first:
            t_first = time.perf_counter()
t_end = time.perf_counter()
print(f"ttft {1000*(t_first-t0):.0f} ms")
print(f"decode {(usage['completion_tokens']-1)/(t_end-t_first):.1f} tok/s")python

import json, struct, collections

PATH = "/home/you/models/hf/Qwen-AgentWorld-35B-A3B-NVFP4/model.safetensors"
TOPK, NUM_EXPERTS, BANDWIDTH = 8, 256, 273e9

with open(PATH, "rb") as f:
    hdr = json.loads(f.read(struct.unpack("<Q", f.read(8))[0]))
hdr.pop("__metadata__", None)

cat = collections.Counter()
for name, meta in hdr.items():
    lo, hi = meta["data_offsets"]
    if "visual" in name:                              c = "visual"
    elif "experts" in name and "shared" not in name:  c = "routed_experts"
    elif "embed_tokens" in name:                      c = "embed"
    else:                                             c = "per_token_dense"
    cat[c] += hi - lo

# a decode step reads everything except the vision tower and the embedding gather,
# and only TOPK of NUM_EXPERTS routed experts
per_token = cat["per_token_dense"] + cat["routed_experts"] * TOPK / NUM_EXPERTS
print(f"{per_token/1e9:.2f} GB/token -> ceiling {BANDWIDTH/per_token:.0f} tok/s")python

grep -E 'GPU KV cache size|Maximum concurrency' /tmp/vllm.log
# GPU KV cache size: 412,980 tokens
# Maximum concurrency for 262,144 tokens per request: 1.58xbash

Troubleshooting

'ValueError: There is no module or parameter named visual in Qwen3_5MoeForConditionalGeneration' at weight loading.

You passed --language-model-only on a vLLM whose qwen3_5.py has been patched with a tower-skipping guard (a patch some language-only checkpoints on this architecture require). This checkpoint is not one of them: it ships all 333 visual.* tensors, so the tower must be built. Drop the flag. The general rule for this architecture family is that the tower decision is a property of the individual checkpoint, not of the architecture — count the visual tensors in the checkpoint before writing the serve line. The mirror-image failure, on a checkpoint that lacks the tower, is 'Following weights were not initialized from checkpoint: {visual.…}'.

How do I count the visual tensors when there is no model.safetensors.index.json?

Read the safetensors header directly. The first 8 bytes of the file are a little-endian u64 giving the header length; the next that-many bytes are JSON keyed by tensor name. You can do it over HTTP with two range requests without downloading the weights: `curl -r 0-7` for the length, then `curl -r 8-<n+7>` for the header. This checkpoint's header is 17,583,240 bytes and names 124,756 tensors, 333 of them under model.visual.

Decode is around 64 tok/s. Should a 4-bit 35B-A3B be faster than that?

No — that is close to what the memory bus allows. Decode on the Spark is bandwidth-bound. Summing the safetensors header and weighting the routed experts by top-8-of-256, a decode step here reads 2.42 GB, which puts the arithmetic ceiling at roughly 113 tok/s against 273 GB/s; 64.09 is 57% of it, a normal fraction once NVFP4 dequantization and the unquantized lm_head are accounted for. The backends chosen (FLASHINFER_CUTLASS NVFP4 MoE, FLASH_ATTN) are the right ones on sm_121.

The quant is 3.2x smaller on disk but only 2.11x faster. Where did the rest go?

Nowhere — the two ratios answer different questions and should not match. File size is dominated by all 256 experts; per-token bytes read count only the 8 that route. The measured per-token read goes 5.89 GB (BF16) to 2.42 GB (NVFP4), a 2.44x byte ratio, which lands within 13% of the 2.11x measured speed ratio. That residual is dequantization work and the unquantized lm_head, which is 0.95 GiB read on every single token and is not compressed at all. Always compute the byte ratio, not the file ratio, before predicting a quant's speedup.

The speedup I measure is smaller than 2.11x.

Check your prompt length before anything else. This quant compresses weights and not the KV cache — its quantization_config carries no kv_cache_scheme — so as the prefix grows the share of per-token bytes that are cache reads rises and the weight saving becomes a smaller fraction of the work. Measured against BF16 at matched prompt lengths on the same node: 2.12x at 511 tokens, 2.11x at 2,042, 2.06x at 8,206. In absolute terms decode falls further out: 54.89 tok/s at 32,776, 37.35 at 131,291, 27.46 at 250,223. A quant does less for a long-context workload than for a chat one, which is the opposite of the usual intuition.

Can I turn on speculative decoding? The config says mtp_num_hidden_layers: 1.

No draft ships with this checkpoint. The config advertises an MTP layer but the file holds 124,756 tensors and not one matches 'mtp' — the language-model layer indices run 0-39 with nothing at index 40, where a native MTP layer would sit. Passing '{"method":"mtp"}' will not find weights to load. Always grep the checkpoint, not just the config; the config is a declaration of intent and the tensor names are the fact.

The log says 'Setting attention block size to 1056 tokens to ensure that attention page size is >= mamba page size'.

Expected on this architecture, not a warning to act on. Thirty of the forty layers are gated-delta linear attention carrying a per-sequence recurrent state, whose page has to fit alongside the full-attention pages — which forces the unusually large block size. It is part of why per-token KV lands where it does.

Should I serve this on two Sparks with TP2 instead?

There is no reason to. The whole working set is ~39 of 114 usable GiB on one node, so tensor parallelism would buy memory that is not remotely scarce while adding the RoCE fabric setup, the per-node NCCL_IB_GID_INDEX resolution, the Gloo interface binding and the cross-node cudagraph deadlock that forces --enforce-eager. Keep it single-node — and if you have two Sparks, run two copies rather than one split one.

Is this quant's output actually good? It is a third-party export.

This recipe measures speed, not quality, and does not publish an accuracy claim for it. What can be said from serving it: generation is coherent and on-task across the benchmark prompts and hand checks, and the quantization scheme is a conservative one — nvfp4-pack-quantized at group_size 16 with a static-minmax activation observer, leaving lm_head in BF16. Before trusting it on work that matters, run an eval you care about against both this and the BF16 release through an identical pipeline and compare the delta; a quant-vs-base quality claim only means anything when both sides go through the same extraction.