GLM-5.2 — 2-bit MoE

2 × Spark96K ctx21.6 tok/s@ 2KvLLMthinking onfast

Run GLM-5.2 (753B MoE) across two DGX Sparks with vLLM-Moet's 2-bit expert kernels, expert-pruned planes, and NCCL-over-RoCE — ~15 tok/s single-stream, ~22 at 96K context with the fast build and its 4-bit speculator. Every number measured on real GB10 hardware.

Run GLM-5.2 (753B-parameter MoE) across two DGX Sparks with vLLM-Moet's 2-bit expert kernels. Everything here is measured on real hardware (GB10, sm_121, 121 GiB unified LPDDR5x per node, CUDA 13, driver 580.159.03). No Docker — vLLM 0.24.0 ships aarch64 wheels. This is Sapid Labs' own runbook; the helper scripts encode every GB10-specific memory workaround, so you can run it without understanding all six.

• 2 × DGX Spark (GB10, sm_121), tensor-parallel (TP2)
• vLLM-Moet 2-bit expert kernels (2-bit MoE W2 planes)
• Expert-pruned planes: 256→208 per layer (REAP-style) so they stay page-cache-resident
• NCCL-over-RoCE (200G ConnectX), FULL cudagraphs (needs NCCL ≥ 2.30.7)
• MTP speculative decode (k=1); fast build adds big-3 NVFP4 + REAP + top-k4 + a W4A16 dspark speculator at K=2
• ~15 tok/s single-stream base; ~22 with the fast build at 96K context (current best)
• The 4-bit speculator is worth +8.7% decode over the BF16 one (2.8σ, 15 repeats/arm) at no measurable acceptance cost — sapidlabs/Sparkulator-GLM-5.2
• Absolute throughput on this box drifts >10% day to day — only same-session paired comparisons mean anything, and 3 repeats can't resolve a 9% effect
• 96K context measured free: KV was under-provisioned at 2G/8K, and 10G buys 113,088 KV tokens at 0.9 sigma (128K is available for ~6% decode)
• Quality: base GSM8K 91% strict (300-ex); fast build GSM8K-50 92% — both measured on-device

Software requirements

• 1–2 DGX Sparks with a working driver (nvidia-smi shows GB10)
• 200G ConnectX link up between nodes, static IPs, passwordless SSH both ways
• ~900 GiB disk per node for GLM-5.2 (planes + checkpoint)
• Python 3.12; vLLM 0.24.0 (aarch64 wheels); Ray 2.56.0
• NCCL ≥ 2.30.7 staged at ~/nccl-2.30.7/libnccl.so.2 on both nodes (from the nvidia-nccl-cu13 wheel)
• Root access once, for the memlock limit fix (full-speed RDMA)

Quick start

1. 1

   Install vLLM-Moet (each node)

   vLLM 0.24.0 has aarch64 wheels — no Docker, no source build. Create the venv, clone the spark-gb10 branch, apply the three patches, pin deps, and build DeepGEMM. Do this on both nodes, or install once and rsync the venv + repo to the peer (identical OS/user).

   python3.12 -m venv ~/venvs/vllm-moet
   ~/venvs/vllm-moet/bin/pip install vllm==0.24.0 "ray[default]==2.56.0"
   
   git clone -b spark-gb10 https://github.com/Sapid-Labs/vLLM-Moet ~/Dev/vLLM-Moet
   cd ~/venvs/vllm-moet/lib/python3.12/site-packages
   git apply ~/Dev/vLLM-Moet/patch/vllm-moet-v0.24.0.patch
   git apply ~/Dev/vLLM-Moet/spark/spark-unified-memory.patch
   git apply ~/Dev/vLLM-Moet/spark/sync-pp-spec-sched-fix.patch
   
   # dep pins (see docs/v024-port.md for why)
   ~/venvs/vllm-moet/bin/pip uninstall -y flashinfer-cubin
   ~/venvs/vllm-moet/bin/pip install flashinfer-python==0.6.14
   git clone https://github.com/deepseek-ai/DeepGEMM ~/Dev/DeepGEMM
   cd ~/Dev/DeepGEMM && git checkout a6b593d2826719dcf4892609af7b84ee23aaf32a
   git submodule update --init --depth 1 third-party/cutlass third-party/fmt
   ~/venvs/vllm-moet/bin/python setup.py bdist_wheel
   ~/venvs/vllm-moet/bin/pip install --no-deps dist/deep_gemm-*.whlbash

   Second node: rsync -a ~/venvs/vllm-moet peer:~/venvs/ and rsync -a ~/Dev/vLLM-Moet peer:~/Dev/ is sufficient.

2. 2

   Lift the memlock limit (one root one-liner, per node)

   DGX OS ships an 8 MB memlock limit that breaks NCCL-over-RDMA (ibv_reg_mr: Cannot allocate memory). Without this the scripts still run over TCP, ~15% slower.

   sudo tee /etc/security/limits.d/99-rdma-memlock.conf <<< $'youruser soft memlock unlimited\nyouruser hard memlock unlimited'bash

   Applies to new login sessions; the cluster script self-SSHes for this reason.

3. 3

   Get the pruned TP2 planes (each node downloads only its shard)

   Never let the server convert weights at load time — on unified memory that OOMs the box. Download the exact expert planes we serve: 2-bit, expert-pruned (208/layer), TP2-sharded, ~79 GB per rank. Both nodes also need the base GLM-5.2-FP8 checkpoint (config/tokenizer/non-expert weights).

   hf download zai-org/GLM-5.2-FP8 --local-dir ~/models/hf/GLM-5.2-FP8
   
   # on .1 (rank 0):
   hf download sapidlabs/GLM-5.2-2bit-MoE-planes-pruned208-tp2 --include 'rank0/*' \
     --local-dir /tmp/planes && mv /tmp/planes/rank0 ~/models/hf/GLM-5.2-FP8/moe_w2_planes_tp2_p208
   # on .2 (rank 1): same command with 'rank1/*'bash

   Or produce them yourself: spark/prepack_planes.py (TP2 variant) then spark/routing/prune_planes.py with spark/routing/keep208.json (row-select, no requant, ~15 min).

4. 4

   Start the Ray cluster and serve GLM-5.2 (from the head node .1)

   Needs NCCL ≥ 2.30.7 staged on both nodes — Torch's bundled 2.28.9 deadlocks TP2 FULL cudagraph replay on GB10+CX7. start-ray-cluster.sh brings up raylets on both nodes, delivers the NCCL pin via VLLM_NCCL_SO_PATH, and resolves each node's RoCEv2 GID index (they drift across reboots — check its '== RoCEv2 GID indexes ==' banner if NCCL dies at init).

   ~/Dev/vLLM-Moet/spark/start-ray-cluster.sh    # raylets on both nodes, memory-capped
   VLLM_MOE_W2_PREPACKED_DIR=$HOME/models/hf/GLM-5.2-FP8/moe_w2_planes_tp2_p208 \
     MTP_K=1 ~/Dev/vLLM-Moet/spark/serve-glm52-tp2-mtp.sh
   ~/Dev/vLLM-Moet/spark/warm-planes.sh ~/models/hf/GLM-5.2-FP8/moe_w2_planes_tp2_p208 --peerbash

   Boot log must show POOL-PRUNED 256->208 (once per MoE layer per rank) or the pruned planes weren't picked up. Expect ~15 tok/s single-stream after a few hundred tokens settle. Re-run warm-planes.sh after any cache purge.

5. 5

   (Optional) Fast build — ~22 tok/s single-stream (current best)

   Stack four levers on the base config: NVFP4 on just the big-3 attention linears (o_proj, q_b_proj, kv_b_proj — the full attention+shared cut degrades quality, so builds ship big-3 only; all other dense stays FP8, experts stay 2-bit), REAP-calibrated planes, top-k=4 routing (num_experts_per_tok: 4, the s12 win), and the W4A16 dspark speculator at K=2. Needs the dspark venv port applied on both nodes (dspark-port/apply.sh) and the speculator at the SAME path on both nodes. The 4-bit speculator is a drop-in for the BF16 one — same weights, same acceptance, 4.57 GiB instead of 7.09 — but it only works with the quantized-drafter fixes in the port, which is why the port has to be re-applied.

   hf download sapidlabs/Sparkulator-GLM-5.2 --local-dir ~/models/hf/Sparkulator-GLM-5.2   # both nodes
   ~/Dev/vLLM-Moet/dspark-port/apply.sh ~/venvs/vllm-moet/lib/python3.12/site-packages/vllm  # both nodes
   
   ~/Dev/vLLM-Moet/spark/start-ray-cluster.sh
   M=$HOME/models/hf/GLM-5.2-FP8
   DSPARK_K=2 MODEL=$M/nvfp4_big3_overlay \
     SPECULATOR=$HOME/models/hf/Sparkulator-GLM-5.2 \
     VLLM_MOE_W2_PREPACKED_DIR=$M/moe_w2_planes_tp2_p208_reap \
     VLLM_NVFP4_DENSE=1 VLLM_NVFP4_TARGETS=o_proj,q_b_proj,kv_b_proj \
     VLLM_ENGINE_READY_TIMEOUT_S=2400 VLLM_MOE_W2_FADVISE_GLOB= \
     ~/Dev/vLLM-Moet/spark/serve-glm52-tp2-dspark.sh --hf-overrides '{"num_experts_per_tok":4}' \
       --max-model-len 98304 --kv-cache-memory-bytes 10737418240bash

   Context: the script defaults to 8192/2G, but KV is heavily under-provisioned there — the trailing --max-model-len/--kv-cache-memory-bytes override it (vLLM takes the last flag). 10G measured free at 96K; 12G/128K costs ~6% decode. K=2 and K=3 measure identically on the 4-bit drafter (0.1σ apart); K=4 loses ~10% because its fourth draft slot is accepted only 1 time in 10. Leave VLLM_DSPARK_CONF_TAU / VLLM_DSPARK_CONF_MIN_K (dynamic K) unset — measured null. First ~10 runs after boot are slow (2→22 tok/s) as the serve script purges page cache and the planes fault back in — never benchmark cold. If you swap between BF16 and W4A16 speculators, clear the draft's compiled graph first (~/.cache/vllm/torch_compile_cache/<hash>/rank_*/eagle_head plus the matching torch_aot_compile/<sha>/) or set VLLM_DISABLE_COMPILE_CACHE=1 — the cache key doesn't include the drafter's quantization and you'll get 'ValueError: too many values to unpack'. PP2 fallback (simpler, slower, no NCCL pin): spark/serve-glm52-pp2.sh --eager.

Key vLLM parameters

API usage

# the served model id comes from GET http://127.0.0.1:8000/v1/models
curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "GLM-5.2",
    "messages": [{"role": "user", "content": "What is 47 × 89?"}],
    "temperature": 0.0
  }'bash

~/Dev/vLLM-Moet/spark/smoke.sh          # coherence
python3 ~/Dev/vLLM-Moet/spark/demo.py   # correct tok/s
python3 ~/Dev/vLLM-Moet/spark/demo_chat.py  # interactive, live tok/s + acceptancebash

Troubleshooting

"Free memory … less than desired GPU memory utilization" at startup

Page cache is squatting. Run `python3 spark/purge-cache.py ~/models/hf` on both nodes (the GLM script does this automatically).

Engine killed at ~60% of load

You're converting at load time (no prepacked planes found — check VLLM_MOE_W2_PREPACKED_DIR), or your per-node memory commitments sum above physical 121 GiB.

`ray status` shows 1 node

Peer raylet died or wrong IP; rerun `start-ray-cluster.sh` (it force-stops both sides first).

Stuck placement groups after a failed launch

`ray stop --force` on both nodes, `pkill -9 -f EngineCore`, then restart the cluster.

nvidia-smi shows no memory numbers

Normal on GB10; watch `free -g` instead. During decode, power.draw >~30 W means working; ~17-20 W means memory-stalled.

Minutes-long cold TTFT / slow first requests

Plane faults from NVMe. Run `spark/warm-planes.sh` after startup or any cache purge; don't mistake a cold prefill for a hang.