T-Search 35B-A3B NVFP4

T-Bank's agentic-search T-Search (36B MoE, ~3B active) in NVFP4 on one DGX Spark, with the checkpoint's own MTP head speculating 3 tokens ahead — 62.0 tok/s single-stream, 1.68x the no-draft baseline, at the full 262,144-token context. Re-verified unchanged on vLLM 0.26.0.

T-Search is an agentic retriever: T-Bank post-trained Qwen3.6-35B-A3B on synthetic multi-step search tasks in English and Russian, and it is built to drive a ReAct loop that issues corpus queries, reads snippets, keeps a coverage state across rounds and returns a ranked evidence set. Its publisher is explicit that the checkpoint is not a general-purpose upgrade over its base — what it is good at is search. That makes it a natural single-node Spark workload, because agentic retrieval means long prompts, many turns, and one user waiting on each one. The configuration below is shaped by three properties of the checkpoint, and none of them required a decision on our part. First, t-tech ships its own NVIDIA ModelOpt NVFP4 export, 23.75 GB against 69.7 GB of BF16, and vLLM recognises it as modelopt_fp4 with no flags. That 4-bit cut is routed-experts-only — the 869-entry exclude_modules list keeps every attention block, every linear-attention block, the shared experts and the embeddings in BF16 — so it should be read as a memory win first and a speed win second. Second, the same repo ships a real MTP head: 785 mtp.* tensors in a separate model-mtp.safetensors that the checkpoint's own weight index references, so a single --speculative-config line pointing back at the model directory is the whole of speculative decoding here. It is worth 1.63x. Third, only 10 of the 40 layers run full attention, those layers carry 2 KV heads, and the checkpoint asks for an FP8 KV cache — so the full 262,144-token context is not something you fight for on this model, it is something you are handed. An 8 GiB pool holds two and a half of them, and a 250,222-token request was actually sent to confirm it. The net shape is a model that occupies about 42 GiB of a 114 GiB node while serving its whole native context at 61.7 tok/s. There is no tensor parallelism, no patched engine, and no second Spark.

• 1 × DGX Spark (GB10, sm_121) — no tensor parallelism, no second node, no engine patch, on vLLM 0.24.0 and 0.26.0 alike
• Measured 62.0 tok/s single-stream decode at a 2048-token prompt, warm, greedy (vLLM 0.26.0, 2026-07-28; 61.7 on 0.24.0 the day before)
• 1.68x over the same node's 36.8 tok/s no-draft baseline, measured back to back in one session, from the MTP head that ships inside the checkpoint
• Engine-version safe: 0.24.0 → 0.26.0 moved decode 61.7 → 62.0 and the baseline 37.8 → 36.8 — both inside this machine's run-to-run spread, with no flag changes needed
• Draft acceptance actually improved on 0.26.0 — 0.652 → 0.700 overall, per-position 0.86 / 0.70 / 0.55 — without moving decode, so the extra accepted tokens are paid back in slower steps
• k=3 is the peak: k=4 measured 52.94 tok/s on an identical node, a 16% regression
• Swept k=0…5: 37.78 / 55.47 / 58.15 / 61.72 / 52.94 / 51.26 tok/s — a clean peak at 3, published with the losing values
• Full 262,144-token context served, at 2.63x concurrency on an 8 GiB pinned KV pool
• Hybrid attention — 30 of 40 layers are gated-delta linear, 10 are full attention with 2 KV heads — plus an FP8 KV cache the checkpoint asks for itself
• ~42 GiB of working set on a 114 GiB node: 23.81 GiB of weights + draft, 8 GiB of KV, the rest transients
• The 4-bit cut is routed-experts-only; attention, the linear-attention blocks, the shared experts, the embeddings and the MTP head are all BF16
• Measured at length: 60.94 tok/s at a 32K prompt, 54.48 at 131K, 47.45 at a 250,222-token prompt — the full window was actually exercised

Software requirements

• 1 × DGX Spark (GB10, sm_121), NVIDIA driver ≥ 580 (CUDA-13 capable)
• vLLM 0.26.0 with torch 2.11.0+cu130 — last verified 2026-07-28 on this version; originally built and measured on 0.24.0, which serves it identically. CUDA 13 is required for the NVFP4 MoE kernels; a cu129 build will not run them
• No vLLM patch is needed for this checkpoint, on either 0.24.0 or 0.26.0
• The `hf` CLI to fetch the checkpoint (~25.4 GB)
• ~26 GB of free disk on the serving node

Key vLLM parameters

API usage

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "t-search",
    "messages": [{"role": "user", "content": "You have a corpus search tool. Plan the first three queries you would run to answer: which EU directives govern battery recycling?"}],
    "max_tokens": 512
  }' | jq -r '.choices[0].message.content'bash

# Measure from usage.completion_tokens, never by counting SSE frames:
# speculative decoding bundles several tokens into one chunk, so frame
# counting understates decode by roughly the acceptance rate.
curl -s -o /tmp/r.json -w '%{time_total}\n' \
  http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"t-search","messages":[{"role":"user","content":"Explain MoE routing in detail."}],"max_tokens":256,"temperature":0}'
jq '.usage' /tmp/r.jsonbash

Troubleshooting

You are on vLLM 0.26.0 and the log says "Using 'VLLM_CUTLASS' NvFp4 MoE backend", not FLASHINFER_CUTLASS as this page used to claim.

Expected on 0.26.0 for this checkpoint, and harmless. Auto-selection's menu of working NVFP4 MoE backends on GB10 depends on both the engine version and the checkpoint's quant layout — on 0.26.0 the same t-tech/T-Search-NVFP4 tree picks VLLM_CUTLASS where 0.24.0 picked FLASHINFER_CUTLASS, while other NVFP4 checkpoints on the same build still get FLASHINFER_CUTLASS. Measured back to back on 2026-07-28: decode 61.72 (0.24.0) vs 62.02 tok/s (0.26.0) at a 2,048-token prompt, so nothing was lost. Do not chase it with an explicit --moe-backend; only EMULATION in that line means something is actually wrong.

Engine core initialization failed — "ValueError: There is no module or parameter named 'visual' in Qwen3_5MoeForConditionalGeneration", followed by a dump of every language_model.* parameter.

You passed --language-model-only to an engine whose qwen3_5.py has been patched to skip building the vision tower when that flag is set — but this checkpoint ships the tower. model.safetensors.index.json contains 333 model.visual.* tensors, so the loader has 333 weights and nowhere to put them. Drop the flag. Stock vLLM 0.24.0 serves this checkpoint with no patch at all. The confusing part is that the opposite failure exists on sibling checkpoints of the same architecture — a language-only release with a leftover vision_config fails with 113 UNINITIALIZED visual.* parameters and needs exactly that flag plus that guard. The two error messages point in opposite directions; the weight index settles it in one command.

Decode with the draft enabled is no faster than without it, or slower.

Read /metrics before touching any flag. A draft whose logits do not match the target is harmless to correctness — verification rejects every token — so the only symptom is throughput sinking to the baseline. spec_decode_num_accepted_tokens_total must be well above zero; ours runs at 0.652 of drafted tokens. If it is zero, the draft is not the one this target was built for. If it is healthy and throughput is still flat, you are probably measuring cold: warm the server with a few hundred tokens and run again.

Decode is FASTER at an 8192-token prompt than at 512 tokens.

That ordering is backwards and means the run is cold — the first scenario paid for the NVFP4 expert planes faulting in from NVMe and for the speculative-decode Triton kernels JIT-compiling. Throw the pass away, send warmup traffic, and measure again. A warm run on this config falls monotonically: 64.05 → 61.72 → 59.62 tok/s across 512 / 2048 / 8192-token prompts.

The engine logs 'CUDAGraphMode.FULL_AND_PIECEWISE is not supported with spec-decode ... setting cudagraph_mode=PIECEWISE'.

Expected and harmless with the FlashInfer attention backend. It does mean the speculative arm runs on piecewise graphs while a no-draft baseline gets full ones, so the 1.63x published here is net of that handicap — the real speculative win is, if anything, slightly larger. Still true on vLLM 0.26.0 (2026-07-28): the warning names FlashInferBackend's AttentionCGSupport.UNIFORM_SINGLE_TOKEN_DECODE, the speculative arm captures 7 PIECEWISE graphs, and the no-draft baseline captures 3 FULL decode graphs on top of its 4 piecewise ones.

You want more context than 262,144 tokens.

There isn't any — max_position_embeddings is 262,144 and that is what this recipe serves. The unusual thing about this model is the opposite problem: the KV pool is far larger than one user can use (2.63x concurrency at full length), so the tuning move is to shrink --kv-cache-memory-bytes toward ~3.5 GiB and hand the memory back, not to grow it. Re-read the pool size from the log after any change to num_speculative_tokens, since the draft takes its state out of the same pool: 812,849 tokens at k=0, 689,341 at k=3, 675,451 at k=4.

A 250,000-token request seems to hang.

It is prefilling. Decode on this model degrades gently with context — 61.72 tok/s at 2048 tokens, 60.94 at 32,778, 54.48 at 131,290, 47.45 at 250,222 — but time-to-first-token does not: 0.40 s, 6.4 s, 48.2 s and 141.5 s at those same four prompt lengths, as prefill throughput falls from ~5,100 to ~1,770 tok/s. A quarter-million-token prompt therefore costs well over two minutes before the first token appears. That is the price of the context, and it is why every decode figure on this page is quoted with the prompt length it was taken at. (The long-prompt rows were measured on the second, identical Spark running the same config.)

Ruled out, so you don't have to: things that were NOT needed.

No vLLM patch (the checkpoint loads on stock 0.24.0 and stock 0.26.0). No --moe-backend or --attention-backend override (auto-selection is correct on sm_121 — though it picks a different NVFP4 MoE backend per engine version: FLASHINFER_CUTLASS on 0.24.0, VLLM_CUTLASS on 0.26.0, with no measured decode difference). No --enforce-eager (that is a cross-node TP2 workaround; this is a single node and cudagraphs are fine). No page-cache purge (--kv-cache-memory-bytes skips the startup free-memory check that the purge exists to satisfy). No --trust-remote-code. And no lowering of --max-num-seqs — that would shrink the speculative CUDA-graph capture sizes and cost decode.