KAT-Coder V2.5-Dev NVFP4

Kwaipilot's agentic-coding KAT-Coder V2.5-Dev (35B MoE, ~3B active) in full NVFP4 on one DGX Spark, serving the full 262,144-token context at 82 tok/s single-stream decode — a 27% gain over the same config with no draft, from grafting Qwen3.6-35B-A3B's multi-token-prediction head into a checkpoint that shipped without one. Speculative decoding is verified, so the draft costs nothing in output quality, and the whole working set is 38 of 114 usable GiB.

Contributors

@joemuller

KAT-Coder V2.5-Dev is Kwaipilot's (Kuaishou) agentic-coding model — a Qwen3.6-35B-A3B fine-tune (SWE-bench Verified 69.4%) that keeps Qwen3.6's shape: a 35B sparse MoE with ~3B active per token, a hybrid attention stack (three linear-attention layers to every full-attention one), and a vision tower. Quantized to NVFP4 it is a single 21.9 GB safetensors file that fits one DGX Spark with room to spare. The headline of this recipe is what Kwaipilot left out: they fine-tuned off Qwen3.6 and dropped its multi-token-prediction head, so the checkpoint declares mtp_num_hidden_layers 0 and carries zero mtp.* tensors among its 31,333. Qwen3.6's head is still published, the two models are architecturally identical in every shape-defining field, and an MTP head reads the target's hidden states while sharing its embeddings and lm_head — so it grafts in unmodified. It does, and it works: 84% acceptance and 82 tok/s against 64.7 with no draft, for 19 copied tensors and no training. Because speculative decoding is verified, none of that can change what the model outputs. Two things to know before you plan around it. First, it is a thinking model — it opens a reasoning block by default, so pass chat_template_kwargs.enable_thinking=false for instruct-style or benchmark use. Second, vLLM warns at load that the NVFP4 attention uses a different weight global scale per parallel projection (q/k/v), which it fuses — 'likely reduced accuracy'; it serves fine and coding output looks correct, but verify quality on your own tasks before trusting it for anything exacting.

• 1 × DGX Spark (GB10, sm_121) — no tensor parallelism, no second node
• Full 262,144-token context, actually served — confirmed at a 258,048-token prompt
• Measured 81.9 tok/s single-stream decode at k=2, against 64.7 tok/s with no draft — +27%
• The draft is Qwen3.6-35B-A3B's MTP head grafted in: KAT-Coder ships none (mtp_num_hidden_layers 0, zero mtp.* tensors)
• 84% acceptance, 2.7 tokens/step — within ~1.5 pts of what the same head scores on Qwen3.6 itself, so the fine-tune barely moved the hidden states the draft reads
• Speculative decoding is verified, so the graft cannot change output quality — only speed
• k=2 and k=3 are a plateau (81.9 / 83.3); k=1 and k=4 are clearly worse, so the peak is bracketed
• Full NVFP4 including attention (21.9 GB) — decode is near-roofline because attention isn't left BF16
• Thinking model: opens a reasoning block by default — disable per-request for instruct/benchmark use
• Long prompts are prefill-bound: TTFT is ~1.1 s at 8K and ~102 s at 258K — see the curve below
• vLLM auto-selects the FLASHINFER_CUTLASS NVFP4 MoE backend on sm_121 — do not force marlin
• Whole working set is ~38 of 114 usable GiB; ~76 GiB stays free
• Accuracy caveat: vLLM warns the NVFP4 attention uses per-projection global scales — verify quality on your tasks

Software requirements

• 1 × DGX Spark (GB10, sm_121), NVIDIA driver ≥ 580 (CUDA-13 capable)
• vLLM 0.24.0 with torch 2.11.0+cu130 — CUDA 13 is required for the FP4 MoE kernels; a cu129 build fails outright
• The `hf` CLI to fetch the checkpoint (~22 GB, one file)
• No patches, no custom backend flags, no sitecustomize workarounds

Quick start

1. 1

   Free the node first

   Anything still holding unified memory — a previous vLLM, a leftover Ray cluster — comes out of the same pool. Check before you start, and drop the page cache the checkpoint download left behind, or vLLM reports 'free memory less than desired GPU memory utilization' at startup.

   pgrep -af 'vllm serve|ray::' || echo 'node is clear'
   ray stop --force 2>/dev/null || true
   python3 ~/Dev/vLLM-Moet/spark/purge-cache.py ~/models/hf
   free -gbash

2. 2

   Download the checkpoint

   One safetensors file, ~22 GB. Only this node needs it — there is no second rank to mirror to.

   hf download sakamakismile/KAT-Coder-V2.5-Dev-NVFP4 \
     --local-dir ~/models/hf/KAT-Coder-V2.5-Dev-NVFP4bash

   Verify model.safetensors is 21,901,607,200 bytes and there are zero *.incomplete files before serving; a partial file fails deep into load.

3. 3

   Graft in Qwen3.6's MTP head

   This is the step that makes the recipe fast, and it is pure file surgery — no training, no GPU, about a minute. KAT-Coder is a Qwen3.6-35B-A3B fine-tune whose every shape-defining field matches (hidden 2048, vocab 248320, 256 experts top-8, head_dim 256, 40 layers, full_attention_interval 4); the only difference is that Kwaipilot dropped the MTP head. Copy Qwen3.6's 19 mtp.* tensors in, hardlink the body so you don't duplicate 21.9 GB, and flip two config fields. Credit: the same graft was first published for llama.cpp by gbuzhf (KAT-Coder-V2.5-Dev-APEX-MTP-GGUF).

   # needs any Qwen3.6-35B-A3B checkpoint that kept its head (the Unsloth NVFP4-Fast
   # build does; its quant config excludes the head with `re:^mtp.*`, so it is BF16)
   python3 - <<'PY'
   import json, struct, os, shutil
   SRC_MTP = os.path.expanduser('~/models/hf/Qwen3.6-35B-A3B-NVFP4-Fast')
   SRC_KAT = os.path.expanduser('~/models/hf/KAT-Coder-V2.5-Dev-NVFP4')
   OUT     = os.path.expanduser('~/models/hf/KAT-Coder-V2.5-Dev-NVFP4-MTP')
   def hdr(p):
       with open(p,'rb') as f:
           n = struct.unpack('<Q', f.read(8))[0]
           return json.loads(f.read(n)), 8 + n
   os.makedirs(OUT, exist_ok=True)
   shard = os.path.join(SRC_MTP, 'model-00005-of-00005.safetensors')
   h, base = hdr(shard)
   mtp = {k: v for k, v in h.items() if k.startswith('mtp.')}
   assert len(mtp) == 19, len(mtp)
   order = sorted(mtp, key=lambda k: mtp[k]['data_offsets'][0])
   nh, cur = {}, 0
   for k in order:
       s, e = mtp[k]['data_offsets']
       nh[k] = {'dtype': mtp[k]['dtype'], 'shape': mtp[k]['shape'],
                'data_offsets': [cur, cur + (e - s)]}
       cur += e - s
   blob = json.dumps(nh).encode(); blob += b' ' * ((8 - len(blob) % 8) % 8)
   with open(shard,'rb') as src, open(os.path.join(OUT,'mtp.safetensors'),'wb') as dst:
       dst.write(struct.pack('<Q', len(blob))); dst.write(blob)
       for k in order:
           s, e = mtp[k]['data_offsets']; src.seek(base + s); rem = e - s
           while rem:
               b = src.read(min(rem, 1 << 24)); dst.write(b); rem -= len(b)
   kat = os.path.join(SRC_KAT, 'model.safetensors')
   link = os.path.join(OUT, 'model.safetensors')
   if not os.path.exists(link): os.link(kat, link)   # same fs = free
   body, _ = hdr(kat)
   wm = {k: 'model.safetensors' for k in body if k != '__metadata__'}
   wm.update({k: 'mtp.safetensors' for k in nh})
   json.dump({'metadata': {'total_size': os.path.getsize(kat) + os.path.getsize(os.path.join(OUT,'mtp.safetensors'))},
              'weight_map': wm}, open(os.path.join(OUT,'model.safetensors.index.json'),'w'))
   cfg = json.load(open(os.path.join(SRC_KAT,'config.json')))
   cfg['text_config']['mtp_num_hidden_layers'] = 1
   q = cfg.get('quantization_config') or cfg['text_config'].get('quantization_config')
   q['ignore'].append('re:^mtp.*')      # REGEX, not a literal name — see troubleshooting
   json.dump(cfg, open(os.path.join(OUT,'config.json'),'w'), indent=2)
   for f in os.listdir(SRC_KAT):
       if f.endswith('.safetensors') or f.endswith('.index.json') or f in ('config.json','.cache'): continue
       s = os.path.join(SRC_KAT, f)
       if os.path.isfile(s): shutil.copy2(s, os.path.join(OUT, f))
   print('grafted ->', OUT)
   PYbash

   Costs 1.7 GB of disk, not 23.6 — the body is hardlinked and the original checkpoint is left untouched, so you can A/B the two by pointing --model at either directory.

4. 4

   Serve on one node, with the grafted draft

   This is the shipped configuration. Note what is NOT here: no --moe-backend, no --tensor-parallel-size, no --enforce-eager, no quantization flag. vLLM reads the compressed-tensors config out of the checkpoint and picks the backend itself; the only addition over the no-draft config is --speculative-config. Run it as a lingering systemd user service so it survives your SSH session (loginctl enable-linger $USER once, if you haven't).

   systemd-run --user --unit=kat-serve --collect \
     -p MemoryMax=112G -p MemorySwapMax=0 -p LimitMEMLOCK=infinity \
     --setenv=PATH="$HOME/venvs/vllm/bin:/usr/local/cuda/bin:$HOME/.local/bin:/usr/bin:/bin" \
     --setenv=VLLM_USE_DEEP_GEMM=0 \
     bash -lc 'exec vllm serve $HOME/models/hf/KAT-Coder-V2.5-Dev-NVFP4-MTP \
       --served-model-name kat-coder \
       --max-model-len 262144 \
       --kv-cache-memory-bytes 6442450944 \
       --gpu-memory-utilization 0.85 \
       --max-num-seqs 4 \
       --max-num-batched-tokens 8192 \
       --speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
       --host 0.0.0.0 --port 8000 > /tmp/vllm.log 2>&1'bash

   MemorySwapMax=0 turns any overcommit into a clean kill instead of wedging the box. This matters here: uncapped, the KV pool alone is 77.95 GiB, and a long prefill on top can spike past the cap — see troubleshooting.

5. 5

   Confirm what it actually chose

   A few lines in the log tell you the run is healthy. If the MoE backend line says EMULATION, or the KV pool is far off 274,528 tokens, stop and read the troubleshooting section rather than benchmarking.

   grep -E "num_spec_tokens|NvFp4 MoE backend|attention backend|Model loading took|GPU KV cache size|Maximum concurrency" /tmp/vllm.logbash

   Expect FLASHINFER_CUTLASS, FLASH_ATTN, 22.11 GiB (20.53 base + 1.69 draft), 274,528 KV tokens, 1.05x, and num_spec_tokens=2.

6. 6

   Warm it up before you trust any number

   The first requests after a start pay torch.compile and kernel JIT. Send a few hundred tokens of coding traffic before recording anything, and if you benchmark, run the harness twice and keep the second pass.

   for i in 1 2 3 4 5; do
     curl -s http://127.0.0.1:8000/v1/chat/completions \
       -H 'Content-Type: application/json' \
       -d '{"model":"kat-coder",
            "messages":[{"role":"user","content":"Write an LRU cache in Python."}],
            "max_tokens":256,"temperature":0,
            "chat_template_kwargs":{"enable_thinking":false}}' > /dev/null
   donebash

7. 7

   Check the memory you actually used

   Read this with the API up and idle, not during weight loading — mid-load the KV pool does not exist yet and the figure flatters you by exactly the amount you are about to allocate.

   free -g
   # expect ~42 GiB used of 121 — weights 20.53 + draft 1.69 + KV 6 + overhead ~9.5 + OS ~4bash

8. 8

   Check the draft is actually being accepted

   Decode tok/s alone won't tell you a draft is mispaired — a bad draft is silently correct and merely slow. Read acceptance from the metrics endpoint after some real traffic. Expect ~84% at k=2 with ~2.7 tokens/step; if you see single digits, the head did not load or does not match the target.

   curl -s http://127.0.0.1:8000/metrics | grep -E 'spec_decode_num_(accepted|draft)_tokens_total'
   # accepted / drafted ~= 0.84bash

Key vLLM parameters

API usage

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "kat-coder",
    "messages": [{"role": "user", "content": "Write a Python function to merge two sorted lists."}],
    "max_tokens": 512,
    "chat_template_kwargs": {"enable_thinking": false}
  }' | jq -r '.choices[0].message.content'bash

import json, time, urllib.request, uuid

body = {
    "model": "kat-coder",
    "messages": [{"role": "user",
                  "content": f"[{uuid.uuid4().hex[:8]}] Write an LRU cache in Python."}],
    "max_tokens": 256, "temperature": 0, "stream": True,
    "stream_options": {"include_usage": True}, "ignore_eos": True,
    "chat_template_kwargs": {"enable_thinking": False},
}
req = urllib.request.Request("http://127.0.0.1:8000/v1/chat/completions",
                             data=json.dumps(body).encode(),
                             headers={"Content-Type": "application/json"})
t0 = time.perf_counter(); t_first = None; usage = None
with urllib.request.urlopen(req) as res:
    for raw in res:
        line = raw.decode().strip()
        if not line.startswith("data: "):
            continue
        if line[6:] == "[DONE]":
            break
        ch = json.loads(line[6:])
        usage = ch.get("usage") or usage
        if ch.get("choices") and ch["choices"][0]["delta"].get("content") and not t_first:
            t_first = time.perf_counter()
t_end = time.perf_counter()
print(f"ttft {1000*(t_first-t0):.0f} ms")
print(f"decode {(usage['completion_tokens']-1)/(t_end-t_first):.1f} tok/s")python

--speculative-config '{"method":"ngram","num_speculative_tokens":3,"prompt_lookup_max":4,"prompt_lookup_min":2}'
# measured warm at 2048/256: 47 tok/s open-ended (down from 64.1),
# but ~167 tok/s on a code-echo prompt (up from 64.1). Check acceptance:
# curl -s http://127.0.0.1:8000/metrics | grep spec_decode_num_accepted_tokens_totalbash

Troubleshooting

At load, vLLM warns: 'In NVFP4 linear, the weight global scale is different for parallel layers (e.g. q_proj, k_proj, v_proj). This will likely result in reduced accuracy.'

Expected with this checkpoint, and worth understanding rather than ignoring. This build quantizes attention to NVFP4 (unlike weight-only NVFP4 quants that leave attention FP8), and it stored a separate FP4 global scale per projection. vLLM fuses q/k/v into one QKV kernel, which wants a single shared scale, so it warns. It serves and coding output looks correct in practice, but this is a real accuracy risk — verify quality on your own tasks before trusting it for anything exacting, and prefer a shared-global-scale NVFP4 build if one exists. There is no flag to silence it; it does not stop the run.

Should I use ngram/prompt-lookup instead of the grafted MTP draft?

No, not for general use — the graft supersedes it. Before the MTP head was grafted in, ngram was the only draft-free option: measured warm at 2048/256 it gave ~47 tok/s on the open-ended harness (a 27% REGRESSION from 64.1 no-draft, because prompt-lookup finds nothing to draft) yet ~167 tok/s on a verbatim code-echo (2.6x), because almost every drafted token is accepted. The MTP draft gives 81.9 tok/s across the board with no workload sensitivity, so it is the better default. ngram remains interesting only if your traffic is overwhelmingly near-verbatim edits, where its 2.6x still beats the MTP head's 1.27x — the two cannot be combined.

The model emits reasoning prose before the answer, or a benchmark measures far fewer 'answer' tokens than expected.

It is a thinking model and opens a reasoning block by default. On vLLM 0.24.0 with no reasoning parser configured, that reasoning is emitted inside the normal content stream (it shows up as text ending in '</think>' followed by the real answer), not as separate reasoning_content. Pass "chat_template_kwargs": {"enable_thinking": false} to get an instruct-style answer with no thinking — the template then emits an empty '<think>\n\n</think>' and goes straight to the response. Do this whenever you benchmark.

The server dies mid-run with no CUDA traceback, memory frees cleanly, and the port starts refusing connections — often when a long (8K+) prompt arrives.

That is a cgroup OOM-kill, not a CUDA error (a real GPU OOM leaves a traceback; a systemd MemoryMax kill with MemorySwapMax=0 just terminates the process and releases everything). It happens if you serve with an UNCAPPED KV pool: at util 0.85 the pool alone is 77.95 GiB, plus 20.53 GiB weights, and a long prefill's activations on top can spike past a ~110 GiB cap. The fix is the shipped config: pin the pool with --kv-cache-memory-bytes 6442450944, which drops resident to ~27 GiB and leaves ~85 GiB of headroom for prefill transients. Keep MemorySwapMax=0 so any future miss is a clean kill rather than a box-wide swap-wedge.

The KV pool comes up around 4 million tokens and 15x concurrency. Is that a misconfiguration?

No — that is what this model does when you let it. Only 10 of 40 layers are full attention (full_attention_interval 4) with 2 KV heads at head_dim 256, so KV is cheap and the pool is genuinely that big; it is just useless to one user. Pin it with --kv-cache-memory-bytes 6442450944 to get 274,528 tokens = 1.05x at the full 262,144-token context (308,955 = 1.18x without the grafted draft head), and hand the rest back. The usual Spark instinct — raise --max-model-len until concurrency approaches 1.0x — is already satisfied here at the model's full native context.

The log says 'Setting attention block size to 1056 tokens to ensure that attention page size is >= mamba page size.'

Expected on this architecture, not a warning to act on. The linear-attention (mamba-style) layers carry a per-sequence state whose page must fit alongside the attention pages, which forces the unusually large block size. It is a property of the hybrid attention stack, not a problem.

The context is 262K but a long prompt takes a long time before the first token.

Expected — this config is prefill-bound at length, and the KV headroom that makes 262K affordable does nothing for prefill compute. Measured single-stream, thinking off: an 8,192-token prompt is ~1.1 s TTFT (7,742 tok/s prefill); 32,768 → 4.9 s at 6,654 tok/s; 131,072 → 33.7 s at 3,891 tok/s; 258,048 → 102.4 s at 2,523 tok/s. Decode holds up far better than TTFT over the same range: 61.8 tok/s at 8K, 55.3 at 32K, 39.2 at 131K, 28.8 at 258K. If you need interactive latency keep prompts short; the long window is for a large file or repo you load once and then ask several questions about.

Should I force --moe-backend marlin, as other DGX Spark NVFP4 write-ups say?

No. On vLLM 0.24.0 + torch 2.11.0+cu130 this checkpoint auto-selects FLASHINFER_CUTLASS out of the candidate backends and serves correctly. The TRT-LLM fused-MoE autotuner may log 'Skipped N unsupported tactic(s)' — that is it pruning tactics with no sm_121 kernel and keeping the ones that work, not an error. The marlin-only advice belongs to a different checkpoint and build.

hf download leaves one huge file and no model.safetensors.index.json.

Correct for the base checkpoint — it is a single 21.9 GB model.safetensors with no shard index, so tools that expect an index need adjusting. Verify the one file is exactly 21,901,607,200 bytes and there are no *.incomplete files; that is the whole integrity check. Note the grafted directory built above DOES have an index, because it has two shards (the hardlinked body and mtp.safetensors).

After grafting, vLLM dies during load with KeyError on 'weight_packed' for an mtp tensor.

The MTP head is BF16 and must be excluded from the compressed-tensors config, and the exclusion has to be a REGEX. KAT-Coder's config carries 10,491 literal ignore entries and zero regex ones, because it never had a head to exclude — so you must append `re:^mtp.*` yourself (the build step does). vLLM matches ignore patterns against the prefixed module name, so a literal like `mtp.layers.0.self_attn.q_proj` misses and the layer gets treated as quantized. Qwen3.6's own NVFP4 builds carry exactly this `re:^mtp.*` entry — copy the convention.

Can I quantize the grafted draft head to W4A16 to make it cheaper?

Not on vLLM 0.24.0, and it would not be worth much anyway. Quantizing the head's dense-active tensors (fc, attention q/k/v/o, shared expert) with group-128 RTN produces a clean checkpoint, but the load fails with an AssertionError in load_merged_column_weight: qwen3_5_mtp.py stacks q/k/v into one qkv_proj and gate/up into gate_up_proj, and the packed layout does not satisfy the merged loader's shape check. Backing off to only the unmerged tensors does load, but covers 36 of the 129 MB read per draft forward — a ~1.5% ceiling, below run-to-run noise. Size the opportunity by ACTIVE read, not file size: the head is 1.69 GB but 95% of that is routed-expert planes that are only touched 8-of-256 at a time. Leave it BF16.

Greedy output at temperature 0 is not reproducible run to run — is the draft breaking determinism?

No. This model is nondeterministic at temperature 0 with or without the draft — the no-draft baseline also fails to reproduce its own greedy output on the same server, which is expected from MoE routing under batching plus the NVFP4 attention scale fusion flagged above. The practical consequence is that the usual free correctness check for speculative decoding (confirm greedy spec output is token-identical to greedy no-draft) is UNAVAILABLE here: a hash mismatch tells you nothing. What still holds is the guarantee — the accept/reject step is distribution-preserving, so the draft cannot change what the target would have produced. Verify the draft with acceptance rate instead.