GLM-5.2 — 2-bit MoE

No measured runs yet — be the first: install the spark-benchmark skill.

Quick start

1. 1

   Install vLLM-Moet (each node)

   vLLM 0.24.0 has aarch64 wheels — no Docker, no source build. Create the venv, clone the spark-gb10 branch, apply the three patches, pin deps, and build DeepGEMM. Do this on both nodes, or install once and rsync the venv + repo to the peer (identical OS/user).

   python3.12 -m venv ~/venvs/vllm-moet
   ~/venvs/vllm-moet/bin/pip install vllm==0.24.0 "ray[default]==2.56.0"
   
   git clone -b spark-gb10 https://github.com/Sapid-Labs/vLLM-Moet ~/Dev/vLLM-Moet
   cd ~/venvs/vllm-moet/lib/python3.12/site-packages
   git apply ~/Dev/vLLM-Moet/patch/vllm-moet-v0.24.0.patch
   git apply ~/Dev/vLLM-Moet/spark/spark-unified-memory.patch
   git apply ~/Dev/vLLM-Moet/spark/sync-pp-spec-sched-fix.patch
   
   # dep pins (see docs/v024-port.md for why)
   ~/venvs/vllm-moet/bin/pip uninstall -y flashinfer-cubin
   ~/venvs/vllm-moet/bin/pip install flashinfer-python==0.6.14
   git clone https://github.com/deepseek-ai/DeepGEMM ~/Dev/DeepGEMM
   cd ~/Dev/DeepGEMM && git checkout a6b593d2826719dcf4892609af7b84ee23aaf32a
   git submodule update --init --depth 1 third-party/cutlass third-party/fmt
   ~/venvs/vllm-moet/bin/python setup.py bdist_wheel
   ~/venvs/vllm-moet/bin/pip install --no-deps dist/deep_gemm-*.whlbash

   Second node: rsync -a ~/venvs/vllm-moet peer:~/venvs/ and rsync -a ~/Dev/vLLM-Moet peer:~/Dev/ is sufficient.

2. 2

   Lift the memlock limit (one root one-liner, per node)

   DGX OS ships an 8 MB memlock limit that breaks NCCL-over-RDMA (ibv_reg_mr: Cannot allocate memory). Without this the scripts still run over TCP, ~15% slower.

   sudo tee /etc/security/limits.d/99-rdma-memlock.conf <<< $'youruser soft memlock unlimited\nyouruser hard memlock unlimited'bash

   Applies to new login sessions; the cluster script self-SSHes for this reason.

3. 3

   Get the pruned TP2 planes (each node downloads only its shard)

   Never let the server convert weights at load time — on unified memory that OOMs the box. Download the exact expert planes we serve: 2-bit, expert-pruned (208/layer), TP2-sharded, ~79 GB per rank. Both nodes also need the base GLM-5.2-FP8 checkpoint (config/tokenizer/non-expert weights).

   hf download zai-org/GLM-5.2-FP8 --local-dir ~/models/hf/GLM-5.2-FP8
   
   # on .1 (rank 0):
   hf download sapidlabs/GLM-5.2-2bit-MoE-planes-pruned208-tp2 --include 'rank0/*' \
     --local-dir /tmp/planes && mv /tmp/planes/rank0 ~/models/hf/GLM-5.2-FP8/moe_w2_planes_tp2_p208
   # on .2 (rank 1): same command with 'rank1/*'bash

   Or produce them yourself: spark/prepack_planes.py (TP2 variant) then spark/routing/prune_planes.py with spark/routing/keep208.json (row-select, no requant, ~15 min).

4. 4

   Start the Ray cluster and serve GLM-5.2 (from the head node .1)

   Needs NCCL ≥ 2.30.7 staged on both nodes — Torch's bundled 2.28.9 deadlocks TP2 FULL cudagraph replay on GB10+CX7. start-ray-cluster.sh brings up raylets on both nodes, delivers the NCCL pin via VLLM_NCCL_SO_PATH, and resolves each node's RoCEv2 GID index (they drift across reboots — check its '== RoCEv2 GID indexes ==' banner if NCCL dies at init).

   ~/Dev/vLLM-Moet/spark/start-ray-cluster.sh    # raylets on both nodes, memory-capped
   VLLM_MOE_W2_PREPACKED_DIR=$HOME/models/hf/GLM-5.2-FP8/moe_w2_planes_tp2_p208 \
     MTP_K=1 ~/Dev/vLLM-Moet/spark/serve-glm52-tp2-mtp.sh
   ~/Dev/vLLM-Moet/spark/warm-planes.sh ~/models/hf/GLM-5.2-FP8/moe_w2_planes_tp2_p208 --peerbash

   Boot log must show POOL-PRUNED 256->208 (once per MoE layer per rank) or the pruned planes weren't picked up. Expect ~15 tok/s single-stream after a few hundred tokens settle. Re-run warm-planes.sh after any cache purge.

5. 5

   (Optional) Fast build — 25.8 tok/s single-stream (current best)

   Stack four levers on the base config: NVFP4 on just the big-3 attention linears (o_proj, q_b_proj, kv_b_proj — the full attention+shared cut degrades quality, so builds ship big-3 only; all other dense stays FP8, experts stay 2-bit), REAP-calibrated planes, top-k=4 routing (num_experts_per_tok: 4, the s12 win), and the W4A16 dspark speculator at K=2. Needs the dspark venv port applied on both nodes (dspark-port/apply.sh) and the speculator at the SAME path on both nodes. The 4-bit speculator is a drop-in for the BF16 one — same weights, same acceptance, 4.57 GiB instead of 7.09 — but it only works with the quantized-drafter fixes in the port, which is why the port has to be re-applied.

   hf download sapidlabs/Sparkulator-GLM-5.2 --local-dir ~/models/hf/Sparkulator-GLM-5.2   # both nodes
   ~/Dev/vLLM-Moet/dspark-port/apply.sh ~/venvs/vllm-moet/lib/python3.12/site-packages/vllm  # both nodes
   
   ~/Dev/vLLM-Moet/spark/start-ray-cluster.sh
   M=$HOME/models/hf/GLM-5.2-FP8
   DSPARK_K=3 MODEL=$M/nvfp4_big3_overlay \
     SPECULATOR=$HOME/models/hf/Sparkulator-GLM-5.2 \
     VLLM_MOE_W2_PREPACKED_DIR=$M/moe_w2_planes_tp2_p208_reap \
     VLLM_NVFP4_DENSE=1 VLLM_NVFP4_TARGETS=o_proj,q_b_proj,kv_b_proj \
     VLLM_ENGINE_READY_TIMEOUT_S=2400 VLLM_MOE_W2_FADVISE_GLOB= \
     ~/Dev/vLLM-Moet/spark/serve-glm52-tp2-dspark.sh --hf-overrides '{"num_experts_per_tok":4}' \
       --max-model-len 98304 --kv-cache-memory-bytes 10737418240bash

   Context: the script defaults to 8192/2G, but KV is heavily under-provisioned there — the trailing --max-model-len/--kv-cache-memory-bytes override it (vLLM takes the last flag). 10G measured free at 96K; 12G/128K costs ~6% decode. K=3 is the peak (24.28 vs 23.15 at K=2 once acceptance was measured on real content); K=4 loses ~10% because its fourth draft slot is accepted only 1 time in 10. Leave VLLM_DSPARK_CONF_TAU / VLLM_DSPARK_CONF_MIN_K (dynamic K) unset — measured NEGATIVE, not merely null: tau=0.5 costs 23%. First ~10 runs after boot are slow (2→26 tok/s) as the serve script purges page cache and the planes fault back in — never benchmark cold. If you swap between BF16 and W4A16 speculators, clear the draft's compiled graph first (~/.cache/vllm/torch_compile_cache/<hash>/rank_*/eagle_head plus the matching torch_aot_compile/<sha>/) or set VLLM_DISABLE_COMPILE_CACHE=1 — the cache key doesn't include the drafter's quantization and you'll get 'ValueError: too many values to unpack'. PP2 fallback (simpler, slower, no NCCL pin): spark/serve-glm52-pp2.sh --eager.

Revision history

What has changed on this page since it was published, and what it measured. Newest first. Entries marked arena are promoted wins from the public engine arena — measured, not authored.

1. 2026-07-28

   docs

   Backfilled this changelog. The recipe predates the `updates` convention, so its history — swapping the MTP head for the dspark speculator, quantizing that speculator to W4A16, and settling on K=3 — was written into the prose but never recorded as dated entries. That left it with no verifiable-since date, which ranked it maximally stale in refresh-queue.py despite being the most-worked recipe on the site.

   No configuration change; this entry records the state the page already documents. Current serve: W4A16 dspark speculator (sapidlabs/Sparkulator-GLM-5.2, 4.57 GiB total) at DSPARK_K=3 — K=3 measured 24.28 tok/s vs 23.15 at K=2 once acceptance was taken on real content, and K=4 loses ~10% because its fourth draft slot is accepted about 1 time in 10. 25.8 tok/s single-stream at 96K context. The drafter quantization came from the quantized-drafter case study (2026-07-20), now recorded there as outcome `adopted`.

2. 2026-07-20

   performance

   Quantized the dspark speculator to W4A16 and adopted it as what this recipe serves. Measured against the BF16 drafter with the target model, speculation depth (K=2) and workload held fixed, 15 repeats per arm back to back in one session.

   +8.7% decode (20.21 → 21.97 tok/s mean, 2.8σ, 95% CI +0.55…+2.97) at no measurable acceptance cost (0.5166 → 0.5084, 0.7σ) — almost exactly the +9% the weight-read model predicted. Quantizing lm_head as well was measured and NOT adopted: 0.2σ gain against a 2.5σ acceptance cost, because it carries the highest quantization error in the model and emits the draft logits directly. K=2 beat K=3 on the BF16 drafter and ties it at W4A16; K=4 loses 9.6% because its fourth draft slot is accepted one time in ten.