Hy3 NVFP4

2 × Spark160K ctx· memory-bound19.5 tok/svLLMthinking offvanilla

Serve the full Hunyuan Hy3 (295B MoE, 21B active) across two DGX Sparks with NVFP4 expert planes, BF16 attention, and the checkpoint's native MTP head driving speculative decode — 19.5 tok/s single-stream at 160K context. Needs the marlin MoE backend on sm_121 and a one-line vLLM patch for the MTP head; both are covered below.

Serve the full 295B Hunyuan Hy3 (21B active per token) across two DGX Sparks. This build quantizes only the routed-expert FFNs to NVFP4 and leaves attention, the shared expert, the dense MLP, the router and embeddings in BF16, so the checkpoint lands at ~169 GB — too large for one 121 GiB node, hence TP2. The whole difficulty is kernel support: GB10 is sm_121, and the NVFP4-MoE kernel situation there is a minefield. The working combination is CUDA 13 + --moe-backend marlin (dequant FP4→FP16) + --enforce-eager + a memory budget that sums correctly across cgroups. Tuned for a single user: 13.6 tok/s single-stream at 160K context. Read the troubleshooting section before you start — it lists the kernel backends that do NOT work on this hardware, each of which costs a ~11-minute boot to rediscover.

• 2 × DGX Spark (GB10, sm_121), tensor-parallel (TP2) — the full 295B model, no prune
• NVFP4 weight-only on routed-expert FFNs; attention/shared/dense/router/embeddings stay BF16
• --moe-backend marlin is the ONLY NVFP4-MoE backend that runs on sm_121 (dequant FP4→FP16)
• CUDA 13 is mandatory for the FP4 MoE kernels — a cu129 build fails outright
• MTP speculative decode (k=1) off the checkpoint's own draft layer: 61.3% acceptance, 1.61 tokens/step
• Measured: 19.5 tok/s single-stream decode (13.6 without MTP, so +43%), 252 ms TTFT
• 160K context from a 176,784-token FP8 KV pool (1.08x) — tuned for one user, so the pool is divided, not idled
• Needs a one-line vLLM patch so the MTP head isn't wrongly quantized — see troubleshooting
• First start takes ~7-11 min (99 shards + marlin repack + drafter load + profiling)

Software requirements

• 2 × DGX Spark (GB10, sm_121), NVIDIA driver ≥ 580 (CUDA-13 capable)
• vLLM with HYV3ForCausalLM support built against CUDA 13 (we run 0.24.0 + torch 2.11.0+cu130 in a venv; upstream's proven path is the vllm/vllm-openai:hy3-arm64-cu130 image)
• A vLLM patched so the MTP shared head isn't quantized (one line — see the MTP step); without it the drafter fails to load at TP>1
• Direct RoCE link between nodes with static point-to-point IPs. Resolve each node's RoCEv2 GID index at start time — they differ per node and drift across reboots (on our pair: .1 = 3, .2 = 6)
• ~169 GB free disk per node — both nodes need the full checkpoint at the same path
• Passwordless SSH between nodes

Quick start

1. 1

   Stage the checkpoint on both nodes (same path)

   Each TP rank reads all shards to extract its slice, so both nodes need the complete ~169 GB tree at an identical path. Download once, then mirror over the direct RoCE link — we measured 169 GB in 3.7 min (~760 MB/s) that way, versus pulling twice over the internet.

   # node 1
   hf download LibertAIDAI/Hy3-NVFP4 --local-dir ~/models/hf/Hy3-NVFP4
   
   # mirror to node 2 over the direct RoCE link
   rsync -a --partial --inplace \
     -e 'ssh -c aes128-gcm@openssh.com -o Compression=no' \
     ~/models/hf/Hy3-NVFP4/ 192.168.100.2:~/models/hf/Hy3-NVFP4/bash

   Verify both sides end up with 99 *.safetensors shards and zero *.incomplete files before serving — a partial tree fails deep into load.

2. 2

   Start the Ray cluster (head first, then worker)

   TP2 is placed by Ray, so the cluster comes up before vLLM. Three things are load-bearing here. (1) Start each raylet through a self-SSH LOGIN shell: actors inherit the raylet's PATH, and a fresh session picks up the raised memlock limit that RDMA registration needs. (2) The fabric env must be on the RAYLET — Ray actors inherit the raylet's environment, not the driver's, so exporting it in the serve script alone leaves the workers unconfigured. (3) Resolve each node's RoCEv2 GID index at start time; they differ per node and drift (ours were .1=3, .2=6 simultaneously), and a wrong one kills NCCL at init with 'unhandled system error'.

   VENV=$HOME/venvs/vllm
   COMMON="RAY_memory_monitor_refresh_ms=0 NCCL_IB_DISABLE=0 NCCL_IB_HCA=rocep1s0f1 \
   NCCL_SOCKET_IFNAME=enp1s0f1np1 GLOO_SOCKET_IFNAME=enp1s0f1np1 \
   MALLOC_MMAP_THRESHOLD_=65536"
   
   # resolve each node's RoCEv2 GID index for its fabric IP
   gid_index() {  # $1 = node ip
     ssh "$1" "for g in /sys/class/infiniband/rocep1s0f1/ports/1/gids/*; do \
       i=\${g##*/}; \
       [ \"\$(cat /sys/class/infiniband/rocep1s0f1/ports/1/gid_attrs/types/\$i 2>/dev/null)\" = 'RoCE v2' ] || continue; \
       case \$(cat \$g) in *ffff:$(printf '%02x%02x:%02x%02x' $(echo $1 | tr . ' '))) echo \$i; break;; esac; done"
   }
   GID1=$(gid_index 192.168.100.1); GID2=$(gid_index 192.168.100.2)
   echo "GID indexes: .1=$GID1 .2=$GID2"
   
   # head (.1) — via self-SSH so it's a fresh login shell
   ssh 192.168.100.1 "bash -lc \"env $COMMON VLLM_HOST_IP=192.168.100.1 NCCL_IB_GID_INDEX=$GID1 \
     systemd-run --user --scope --collect -p MemoryMax=104G -p MemorySwapMax=0 \
     $VENV/bin/ray start --head --node-ip-address=192.168.100.1 --port=6379 --object-store-memory=2000000000\""
   sleep 6
   
   # worker (.2)
   ssh 192.168.100.2 "bash -lc \"env $COMMON VLLM_HOST_IP=192.168.100.2 NCCL_IB_GID_INDEX=$GID2 \
     systemd-run --user --scope --collect -p MemoryMax=110G -p MemorySwapMax=0 \
     $VENV/bin/ray start --address=192.168.100.1:6379 --object-store-memory=2000000000 \
     --node-ip-address=192.168.100.2\""
   
   $VENV/bin/ray status   # expect 2 nodes, 2.0 GPUbash

   Budget the MemoryMax caps as a SUM against the 121 GiB box, not individually: head raylet 104G + serve driver 8G + OS ~4G. Per-scope caps do not bound the total — 110G + 8G looks safe per-scope and still swap-wedges the machine. MemorySwapMax=0 is what makes a miss a clean kill rather than a thrash.

3. 3

   Serve TP2 with the marlin MoE backend

   --moe-backend marlin is the load-bearing flag: it is the only NVFP4-MoE backend that runs on sm_121. --enforce-eager skips cudagraph capture, the biggest transient allocation. Purge the page cache on both nodes first — vLLM's startup check wants free >= util*total and cudaMalloc can't reclaim page cache on GB10. Context length is chosen by dividing the KV pool, not by growing it: we optimize for a single user, so --max-model-len is pushed until concurrency approaches 1.0x and --max-num-seqs drops to 2.

   VENV=$HOME/venvs/vllm
   export VLLM_HOST_IP=192.168.100.1
   export NCCL_SOCKET_IFNAME=enp1s0f1np1 GLOO_SOCKET_IFNAME=enp1s0f1np1
   export NCCL_IB_DISABLE=0 NCCL_IB_HCA=rocep1s0f1 NCCL_IB_GID_INDEX=$GID1
   export RAY_memory_monitor_refresh_ms=0 MALLOC_MMAP_THRESHOLD_=65536
   
   # purge page cache on BOTH nodes
   python3 ~/Dev/vLLM-Moet/spark/purge-cache.py ~/models/hf
   ssh 192.168.100.2 "python3 ~/Dev/vLLM-Moet/spark/purge-cache.py ~/models/hf"
   
   systemd-run --user --scope --collect -p MemoryMax=8G -p MemorySwapMax=0 \
     $VENV/bin/vllm serve ~/models/hf/Hy3-NVFP4 \
     --served-model-name hy3-nvfp4 --trust-remote-code \
     --distributed-executor-backend ray \
     --tensor-parallel-size 2 --pipeline-parallel-size 1 \
     --moe-backend marlin \
     --enforce-eager \
     --kv-cache-dtype fp8 \
     --max-model-len 163840 \
     --max-num-seqs 2 \
     --max-num-batched-tokens 8192 \
     --gpu-memory-utilization 0.85 \
     --speculative-config '{"method":"mtp","num_speculative_tokens":1}' \
     --host 0.0.0.0 --port 8000bash

   First start takes ~7-11 min (99 shards + marlin incremental repack + profiling). The driver only holds the API server and engine core — the weights live in the Ray actors — so an 8G cap is ample. Expect the log to confirm quantization=modelopt_fp4 and moe_backend='marlin'. The --speculative-config line turns on MTP using the checkpoint's own draft layer; it requires the patch in the next step.

4. 4

   Patch vLLM so the MTP head isn't quantized (both nodes)

   vLLM builds the MTP shared head with no `prefix`, so the quant config's exclusion check never matches "lm_head". On this checkpoint — which deliberately keeps lm_head in BF16 — the head is therefore built as a packed NVFP4 parameter at half width, and then load_weights feeds it the target model's full-width BF16 lm_head weight. The drafter dies at load with a 2048-vs-4096 shape error. Naming the head "lm_head" makes the exclusion fire, exactly as the target model's own head does. Apply on BOTH nodes — each has its own site-packages.

   # vllm/model_executor/models/hy_v3_mtp.py — HYV3SharedHead.__init__
   -    def __init__(self, config, quant_config=None) -> None:
   +    def __init__(self, config, quant_config=None, prefix: str = "") -> None:
            super().__init__()
            self.head = ParallelLMHead(
   -            config.vocab_size, config.hidden_size, quant_config=quant_config
   +            config.vocab_size,
   +            config.hidden_size,
   +            quant_config=quant_config,
   +            prefix=maybe_prefix(prefix, "lm_head"),
            )
   
   # ...and at the call site in HYV3MultiTokenPredictorLayer.__init__:
   -        self.shared_head = HYV3SharedHead(config=config, quant_config=quant_config)
   +        self.shared_head = HYV3SharedHead(
   +            config=config, quant_config=quant_config, prefix=prefix
   +        )python

   The module attribute stays `shared_head.head`, so the weight mapping in load_weights still resolves — only the quant-config matching changes. Upstream vLLM main has the same defect.

5. 5

   Smoke test, then check the KV pool you actually got

   Confirm coherence, then read the KV numbers out of the startup log — that is the figure to record, along with the STEADY-STATE memory once the API is up.

   curl -s localhost:8000/v1/chat/completions \
     -H 'Content-Type: application/json' \
     -d '{"model":"hy3-nvfp4","messages":[{"role":"user","content":"capital of France? one word"}],"max_tokens":8,"temperature":0}'
   # -> "Paris."
   
   grep -E 'GPU KV cache size|Maximum concurrency' ~/logs/hy3-nvfp4-mtp.log
   # GPU KV cache size: 176,784 tokens
   # Maximum concurrency for 163,840 tokens per request: 1.08x
   
   # steady-state memory — read this AFTER the API answers, not during load
   free -g | sed -n '2p'
   ssh 192.168.100.2 "free -g | sed -n '2p'"bash

   Never benchmark the first requests after start — expert planes fault in from NVMe and the marlin repack is still settling. Warm with a couple of throwaway generations first. With MTP on, also grep for 'Draft acceptance rate' — ours is 61.3% (1.61 tokens/step), which is what the +43% decode gain comes from.

Key vLLM parameters

API usage

# 2 warmup generations, then time first-token vs last-token over 256 tokens
curl -s localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"hy3-nvfp4",
       "messages":[{"role":"user","content":"Explain paged attention in detail."}],
       "max_tokens":256, "temperature":0, "stream":true,
       "stream_options":{"include_usage":true}}'

# decode_tps = (completion_tokens - 1) / (t_last_token - t_first_token)
# measured: 13.6 tok/s single-stream, 176 ms TTFTbash

Troubleshooting

cudaErrorNoKernelImageForDevice with the default (flashinfer-cutlass) MoE backend

flashinfer ships zero sm_120/121 cubins and the CUTLASS grouped FP4 GEMM is broken on sm120/121. Use --moe-backend marlin.

--moe-backend flashinfer_cutedsl is rejected

That kernel is gated to family(100) — datacenter sm_100, not sm_121. Dead end on GB10.

'b12x fused MoE requires CUDA 13 or later' — or b12x livelocks on CUDA 13

On cu129 b12x refuses to run; on a cu130 build with cutlass-dsl 4.5.2 it selects and loads then livelocks in the CuTe-DSL JIT (4.5.x breaks sm121 PTX codegen). Downgrading to cutlass-dsl 4.4.2 breaks the flashinfer import. Dead end — use marlin.

cudaErrorUnsupportedPtxVersion on any FP4 kernel JIT

The CUDA toolkit is newer than the driver (e.g. a cu13.2 image on a 580 driver). Match the toolkit to the driver — the known-good pairing is CUDA 13.0.

Box becomes pingable but SSH-dead during startup (swap-wedge)

The unified pool was overcommitted and the machine is thrashing — the kernel network stack still answers ping but userspace (including sshd and tailscaled) is starved. We hit this on 2026-07-19 trying to pin 23 GiB/rank of KV at --max-model-len 262144. Recovery did NOT need a hard reboot: the wedged node came back on its own once we killed Ray on the *surviving* peer (`ray stop --force` + `pkill -9 -f 'ray::'`), which released the cross-node pressure. Try that before power-cycling. Reach the wedged node's peer over the direct fabric IP, since Tailscale will be down.

'RuntimeError: cancelled' in shm_broadcast, then 'Engine core initialization failed'

That is the head losing a TP worker, not a comms bug — check the PEER node's memory. In our case the rank-1 worker had climbed to 120 GB used / 1 GB free and stopped responding, so the head's engine core cancelled the broadcast. Lower the KV budget or max-model-len and retry.

Per-cgroup MemoryMax set but the box still wedged

Per-scope caps don't bound the SUM. On the head, raylet + the serve driver + the OS are separate scopes, so caps of 110G and 8G on a 121 GiB box still permit a global overcommit. Budget them together: we use MemoryMax=104G (head raylet) + 8G (driver), and 110G on the worker raylet, all with MemorySwapMax=0 so a miss is a clean kill rather than a thrash.

'Gloo connectFullMesh Connection refused', remote=[127.0.0.1]

Set GLOO_SOCKET_IFNAME to the fabric interface (same as NCCL_SOCKET_IFNAME) and a per-node VLLM_HOST_IP on both nodes.

Minutes-long cold TTFT / slow first requests

Expert planes fault in from NVMe on first touch and marlin is still repacking. Warm the model before benchmarking; don't mistake a cold prefill for a hang.

Drafter fails to load: 'The size of tensor a (2048) must match the size of tensor b (4096) at non-singleton dimension 1'

The MTP shared head was quantized when it shouldn't be. 2048 is not a tensor-parallel shard — it is NVFP4 4-bit packing of a 4096-wide hidden dim. vLLM builds HYV3SharedHead's ParallelLMHead with no `prefix`, so the quant config's ignore-matching never sees "lm_head", and a checkpoint that keeps lm_head in BF16 still gets a packed head; load_weights then feeds it the full-width BF16 weight. Fix by passing prefix=maybe_prefix(prefix, "lm_head") when constructing it (see the patch step). Apply on every node — each has its own site-packages, and the failure surfaces on whichever rank loads the drafter.