Qwen AgentWorld 35B-A3B — BF16

Qwen's agentic AgentWorld 35B-A3B (35B MoE, ~3B active) unquantized in BF16 on one DGX Spark, serving the full 262,144-token context at 30.4 tok/s single-stream.

Contributors

@kvncrw

Qwen AgentWorld 35B-A3B is a 35B sparse MoE — 256 routed experts, top-8, about 3B active per token — on the Qwen3.5 hybrid attention stack: 30 of its 40 layers are gated-delta linear attention and 10 are full attention, at a 262,144-token native window. It is a language-only checkpoint despite carrying a vision_config, and at 69.3 GB of BF16 it lands on a single Spark with room to spare, which is the reason to run it unquantized: you get the model exactly as Qwen shipped it, with no quantization artifact to audit and no calibration set to defend. The price is decode speed. Every generated token re-reads roughly 3B active parameters at two bytes each, and decode on the Spark is bandwidth-bound, so 30.4 tok/s single-stream is what the memory bus allows — a 4-bit build of the same model is the lever if you want it faster, and this recipe is the honest BF16 reference it would be measured against. Two things need saying before you copy the serve command. First, stock vLLM 0.24.0 and 0.26.0 both refuse to load this checkpoint: the registered architecture is Qwen3_5MoeForConditionalGeneration, whose __init__ builds a 27-block vision tower unconditionally, and the checkpoint contains zero visual.* tensors, so weight loading dies on 113 uninitialized parameters. Passing --language-model-only is not sufficient on its own — that flag zeroes the multimodal input limits but does not skip constructing the tower — so a three-line guard in vllm/model_executor/models/qwen3_5.py is required, and it is written out in full in the steps below. Second, memory is not the binding constraint here but it is not free either: the weights alone take 64.69 GiB of a 114 GiB budget, so unlike the quantized siblings of this architecture you do not get to be careless with the KV pool. Pinning it at 8 GiB gives 1.58x concurrency at the full context; pinning it at 4 GiB, which is plenty for a 4-bit build of the same architecture, will not start.

• 1 × DGX Spark (GB10, sm_121) — no tensor parallelism, no second node
• Native BF16, exactly as Qwen published it — no quantization, no calibration, nothing to audit
• Measured 30.4 tok/s single-stream decode at a 2048-token prompt, warm, greedy
• Full 262,144-token context served, at 1.58x concurrency on an 8 GiB pinned KV pool
• Hybrid attention: 30 of 40 layers are gated-delta linear, 10 are full attention — which is why 262K context costs only 5.08 GiB of KV
• Requires a three-line vLLM patch: the vision tower is built unconditionally and this checkpoint has no visual weights
• No speculative draft available — the config advertises mtp_num_hidden_layers: 1 but the checkpoint ships zero MTP tensors
• Working set ~80 of 114 usable GiB; weights alone are 64.69 GiB
• Weight load ~140 s, full startup ~4 min from a purged page cache

Software requirements

• 1 × DGX Spark (GB10, sm_121), NVIDIA driver ≥ 580 (CUDA-13 capable)
• vLLM 0.24.0 with torch 2.11.0+cu130 (verified 2026-07-27). vLLM 0.26.0 has the identical vision-tower defect and needs the same patch
• A three-line patch to vllm/model_executor/models/qwen3_5.py — see the steps
• The `hf` CLI to fetch the checkpoint (~69.3 GB)
• ~70 GB of free disk on the serving node

Key vLLM parameters

API usage

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen/Qwen-AgentWorld-35B-A3B",
    "messages": [{"role": "user", "content": "Explain MoE routing in three sentences."}],
    "max_tokens": 512
  }' | jq -r '.choices[0].message.content'bash

import json, time, urllib.request, uuid

body = {
    "model": "Qwen/Qwen-AgentWorld-35B-A3B",
    "messages": [{"role": "user",
                  "content": f"[{uuid.uuid4().hex[:8]}] Write an LRU cache in Python."}],
    "max_tokens": 256, "temperature": 0, "stream": True,
    "stream_options": {"include_usage": True}, "ignore_eos": True,
}
req = urllib.request.Request("http://127.0.0.1:8000/v1/chat/completions",
                             data=json.dumps(body).encode(),
                             headers={"Content-Type": "application/json"})
t0 = time.perf_counter(); t_first = None; usage = None
with urllib.request.urlopen(req) as res:
    for raw in res:
        line = raw.decode().strip()
        if not line.startswith("data: "):
            continue
        if line[6:] == "[DONE]":
            break
        ch = json.loads(line[6:])
        usage = ch.get("usage") or usage
        if ch.get("choices") and ch["choices"][0]["delta"].get("content") and not t_first:
            t_first = time.perf_counter()
t_end = time.perf_counter()
print(f"ttft {1000*(t_first-t0):.0f} ms")
print(f"decode {(usage['completion_tokens']-1)/(t_end-t_first):.1f} tok/s")python

grep -E 'GPU KV cache size|Maximum concurrency' /tmp/vllm.log
# GPU KV cache size: 412,980 tokens
# Maximum concurrency for 262,144 tokens per request: 1.58xbash

Troubleshooting

Engine core initialization failed — 'ValueError: Following weights were not initialized from checkpoint: {visual.blocks.25.norm2.weight, visual.patch_embed.proj.weight, ...}' (113 parameters).

This is the defining failure of this checkpoint and it happens on stock vLLM 0.24.0 and 0.26.0 alike. The registered architecture is Qwen3_5MoeForConditionalGeneration, whose __init__ builds a 27-block Qwen3_VisionTransformer unconditionally, and this is a language-only release with zero visual.* tensors in model.safetensors.index.json. Passing --language-model-only alone does NOT fix it: that flag sets MultiModalConfig.language_model_only, which zeroes the per-modality input limits but is never consulted at construction time in qwen3_5.py. Apply the three-line guard from the steps (skip the tower and set self.visual = None when language_model_only is set) AND pass the flag. vLLM already does exactly this in qwen3_next.py, so the shape of the fix is upstream's own.

'To serve at least one request with the model's max seq len (262144), 5.08 GiB KV cache is needed, which is larger than the available KV cache memory (4.0 GiB).'

Your --kv-cache-memory-bytes is too small for BF16. A 4 GiB pool is enough for a 4-bit build of this architecture but not for this one — the same 262,144-token window costs 5.08 GiB here. Use 8589934592 (8 GiB), which yields 412,980 tokens and 1.58x concurrency. Do not solve it by lowering --max-model-len; the message helpfully suggests 205,920 tokens, but memory is not scarce enough on this node to give up a quarter of the context window for it.

Decode is only ~30 tok/s. Is a kernel misconfigured?

No — that is what BF16 costs on this hardware. Decode on the Spark is bandwidth-bound, and each generated token re-reads roughly 3B active parameters at two bytes each. The measured 30.4 tok/s at a 2048-token prompt is a normal fraction of the resulting roofline, and the backends chosen (FlashInfer CUTLASS unquantized MoE, FLASH_ATTN) are the right ones. If you want this model faster, the lever is a 4-bit expert quantization, not a flag — and be aware that on a high-sparsity MoE a quant which leaves attention unquantized buys much less decode than its file-size reduction suggests, because attention can dominate per-token bytes read even when the experts dominate the file.

Can I turn on speculative decoding? The config says mtp_num_hidden_layers: 1.

No draft ships with this checkpoint. The config advertises an MTP layer but model.safetensors.index.json holds 693 tensors and not one of them matches 'mtp' — the layer indices run 0-39 with nothing at index 40, where a native MTP layer would sit. Passing '{"method":"mtp"}' will not find weights to load. Always grep the index, not just the config; the config is a declaration of intent and the index is the fact.

The log says 'Setting attention block size to 1056 tokens to ensure that attention page size is >= mamba page size'.

Expected on this architecture, not a warning to act on. Thirty of the forty layers are gated-delta linear attention carrying a per-sequence recurrent state, whose page has to fit alongside the full-attention pages — which forces the unusually large block size. It is part of why per-token KV lands where it does.

'free memory less than desired GPU memory utilization' at startup.

Page cache from the 69 GB download is squatting in the unified pool; `free -g` showing a large buff/cache figure with little free is the signature. Run `python3 ~/Dev/vLLM-Moet/spark/purge-cache.py ~/models/hf`. Note this only applies on a boot where vLLM is auto-sizing the pool — --kv-cache-memory-bytes skips that check entirely, so once you pin KV the purge buys nothing and costs a full re-read of the 69 GB tree from NVMe.

Should I serve this on two Sparks with TP2 instead?

There is no reason to. The whole working set is ~80 of 114 usable GiB on one node, so tensor parallelism would buy memory that is not scarce while adding the RoCE fabric setup, the per-node NCCL_IB_GID_INDEX resolution, the Gloo interface binding and the cross-node cudagraph deadlock that forces --enforce-eager. Keep it single-node.

Serving a genuinely multimodal Qwen3.5 checkpoint from the same venv after applying the patch.

It still works. The guard is conditional on MultiModalConfig.language_model_only, which defaults to False and is only set by the --language-model-only CLI flag — so a serve that does not pass that flag takes the original code path and builds the tower exactly as before. Keep the .orig backup anyway; every node has its own venv, so a patch applied on one node does not exist on the other.