Field Notes

Software discoveries and gotchas from running LLM inference on the DGX Spark (GB10, aarch64, 128 GB unified LPDDR5x, DGX OS / Ubuntu 24.04). Raw material for future site content — append as things come up, with dates.

OS & drivers

• DGX OS updates can strand the GPU driver (2026-07-05). The system updater installed the 6.17 kernel and removed the NVIDIA modules for the running 6.11 kernel. Until the next reboot into 6.17, nvidia-smi fails with "couldn't communicate with the NVIDIA driver" and /dev/nvidia* is gone. Diagnosis: compare uname -r against dpkg -l | grep linux-modules-nvidia. Fix: sudo apt full-upgrade, reboot, confirm the new kernel.
• No passwordless sudo on a default DGX OS install; automation must hand privileged steps back to a human terminal.

aarch64 (ARM) surprises

• curl | bash installers often ship x86-64 binaries. firebase.tools does — "cannot execute binary file: Exec format error". Anything Node-based installs fine via npm install -g instead. Check for arm64 builds before trusting any install script.
• System Node is ancient (18.x from Ubuntu). Use nvm; Node 20.11+ is needed for import.meta.dirname.
• PEP 668: Ubuntu 24.04 blocks pip install --user; every Python tool needs a venv.
• npm optional-dependency bug bites native ARM bindings (2026-07-06). Tailwind 4 dies with Cannot find module '@tailwindcss/oxide-linux-arm64-gnu' if node_modules was installed under a different Node setup. Fix: rm -rf node_modules && npm install, and clear .next if the error persists in cached build chunks.

GB10 GPU specifics

• CUDA arch is sm_121 — build llama.cpp with -DCMAKE_CUDA_ARCHITECTURES=121 (CUDA toolkit 13 ships with DGX OS at /usr/local/cuda, present even when the driver is broken).
• nvidia-smi cannot report memory usage on unified memory ("Not Supported"). Read /proc/meminfo (MemTotal − MemAvailable) instead. power.draw and utilization.gpu work fine.
• Unified memory changes engine defaults: vLLM's --gpu-memory-utilization defaults to 0.9 — of the whole 128 GB pool, which would starve the OS. Set it explicitly (0.5 leaves ~25 GB KV cache for a 35 GB model).
• OOM on unified memory takes the whole machine down (2026-07-06). DGX OS ships with no swap, so when a big vLLM run (72 GB BF16 weights, --gpu-memory-utilization 0.75 ≈ 91 GiB target) coincided with first-run Triton kernel compilation (cicc — NVIDIA's CUDA JIT compiler — was what invoked the oom-killer), the kernel spent 5 hours killing desktop processes and the box hard-reset. Rules of thumb: leave ≥20 GiB headroom for model load + JIT compile on first run of a large model, remember GPU allocations and the OS fight for the same pool, and consider a small swapfile purely as an OOM shock absorber.
• A ~59 GB checkpoint disables vLLM's weight prefetch on unified memory (2026-07-27). Loading Nemotron-3-Nano-30B-A3B-BF16 (58.82 GiB, 13 shards), vLLM logged: Auto-prefetch is disabled because the filesystem (EXT4) is not a recognized network FS (NFS/Lustre) and the checkpoint size (58.82 GiB) exceeds 90% of available RAM (54.33 GiB). On a Spark weights and system memory come out of the same pool, so "available RAM" is already small relative to a checkpoint that would be comfortable on a discrete-GPU host. Consequence: Loading weights took 397.00 seconds — 6.6 minutes, vs ~2 minutes for the 18 GiB NVFP4 build of the same model. Budget 7.5 minutes per restart on any BF16 checkpoint over ~50 GB, and get the serve command right the first time.
• The KV cache dtype can come from the checkpoint, not your flags (2026-07-27). Two builds of the same model, same 6 full-attention layers with 2 KV heads at head_dim 128, same --kv-cache-memory-bytes 3221225472, same vLLM 0.24.0 — different pools: NVFP4 gave 982,061 tokens (3,281 B/token), BF16 gave 504,123 tokens (6,390 B/token). The NVFP4 checkpoint ships an hf_quant_config declaring kv_cache_quant_algo: FP8 and vLLM silently honours it (kv_cache_dtype=fp8_e4m3 in the startup config); the BF16 repo declares nothing and gets a BF16 cache. So a quantized checkpoint may be quantizing your KV too. Read the pool size out of the log rather than computing it from head geometry, and use --kv-cache-dtype fp8 if you want the cheap cache without the quantized weights.
• --max-num-seqs is the hidden memory lever in vLLM. Startup profiling simulates the maximum batch (default 256 sequences); on a 72 GB model the resulting activation spike ate 30+ GiB beyond the configured target within seconds. If you're serving one user or benchmarking small concurrency, set it to ~2× your real concurrency and the spike shrinks proportionally.
• Multimodal models hide a second profiling spike. Qwen3.5 is image-text-to-text; vLLM profiles the vision encoder with maximum-size dummy images on top of the LLM profiling pass. For text-only serving of a large VLM, pass --limit-mm-per-prompt '{"image": 0, "video": 0}' — it skips encoder profiling entirely and saves GBs of transient memory.
• vLLM's torch.compile cache can go stale across config changes (0.24.0). After adding --limit-mm-per-prompt '{"image": 0}' to a model previously started without it, the cached compiled graph loads and crashes with AttributeError: 'NoneType' object has no attribute 'size'. The cache key evidently doesn't cover multimodal limits — rm -rf ~/.cache/vllm/torch_compile_cache after changing such flags.
• Throttle compile parallelism when weights nearly fill the pool. Inductor and ninja default to one compile worker per core — 20 on GB10 — and each cicc instance holds 3–4 GiB, so first-run JIT can burst ~70 GiB on top of resident weights. TORCHINDUCTOR_COMPILE_THREADS=2 MAX_JOBS=4 makes first startup slower but keeps the burst to ~10 GiB. This was the common root cause behind repeated OOM crashes loading a 72 GB model.
• A memory watchdog for big runs must SIGKILL, not SIGTERM — a vLLM engine mid-allocation shrugs off SIGTERM and the memory stays pinned. Poll ≤5s: the profiling spike can go from 40 GiB free to 0 in under 10 seconds.

Quantizing on the Spark (llmcompressor)

Self-quantizing Qwen3.6-35B-A3B (a multimodal, 256-expert MoE with hybrid linear attention) BF16 → FP8 → INT4 → NVFP4 with llmcompressor 0.12 in ~/venvs/vllm surfaced a cluster of unified-memory gotchas (2026-07-19):

• device_map="auto" OOMs on load. It puts the full 67 GB BF16 model on the GPU, and on the unified pool that plus conversion buffers blows past 121 GB — silent SIGKILL, no traceback. Load with torch_dtype=bfloat16, low_cpu_mem_usage=True, device_map="cpu" so weights stay pageable via safetensors mmap; FP8/INT4 weight-only quant needs no GPU residency.
• save_pretrained OOMs on the default 50 GB shard gather. With offloaded modules the save step tries to gather a 50 GB shard at once and dies. Quantize without output_dir, then model.save_pretrained(out, max_shard_size="4GB") — peak drops to ~the compressed model size.
• Quantizing a multimodal model via AutoModelForCausalLM silently drops the vision tower and flattens the config to the text-only *TextConfig. vLLM then refuses to serve it ("Expected Qwen3_5MoeConfig, found Qwen3_5MoeTextConfig") because it routes the arch through its multimodal processor. Load with AutoModelForImageTextToText, keep the vision tower in the ignore list (stays BF16), and the saved checkpoint carries the full config vLLM accepts.
• torch.accelerator.get_memory_info() throws CUDA OOM on GB10 — the same unified-memory quirk that makes nvidia-smi report [N/A] for memory. It breaks compressed-tensors' dispatch_model, which the NVFP4 (MoE-linearizing) path calls. Workaround: run that quant CPU-only with CUDA_VISIBLE_DEVICES="".
• Full w4a4 NVFP4 doesn't fit for a big MoE on one GB10. llmcompressor linearizes all 256 experts/layer to calibrate them; the calibration forward pass then CUDA-OOMs (~80 GiB). Weight-only NVFP4A16 (data-free) works and serves fine, but performs like W4A16 — both dequant to BF16, so the FP4 tensor-core win needs the w4a4 path (expert-offloaded / multi-node calibration).
• Result shape: on one Spark, quantization scaled decode 30→43 tok/s and weights 67→22 GiB, but power stayed flat at 36–41 W (4-bit drew slightly more — dequant overhead). Efficiency gains are all throughput, not wattage.

Engines

• vLLM 0.26.0 removed FLASHINFER_CUTLASS for NVFP4 MoE on sm_121, and it costs 13% decode (2026-07-28, Qwen3-Coder-Next-NVFP4-GB10, Spark-2, three builds measured back to back with identical flags). 0.24.0 and 0.25.1 both log Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend; 0.26.0 logs Using 'VLLM_CUTLASS' for the same checkpoint on the same box. Decode at a 2,042-token prompt with an EAGLE3 draft at k=1: 74.5 → 72.4 → 64.7 tok/s across 0.24.0 / 0.25.1 / 0.26.0, prefill 4,746 → 4,756 → 3,625 tok/s, TTFT 430 → 429 → 563 ms — and 0.26.0 draws more power for the slower result (42.0 W vs 36.6 W average). Everything else is identical: 43.22 GiB residency, a 318,547-token KV pool, and 63.3% draft acceptance on both 0.24.0 and 0.26.0, which is what isolates the MoE kernel as the cause. Forcing it back with --moe-backend flashinfer_cutlass fails in ~90 s with "NvFp4 MoE backend 'FLASHINFER_CUTLASS' does not support the deployment configuration since kernel does not support current device cuda" — the device-level rejection, not the checkpoint-level "does not support quantization scheme" one. Practical rule: after any vLLM upgrade, grep the startup log for the MoE backend line before you trust a published throughput number. 0.25.1 is currently the newest build that keeps the fast NVFP4 MoE path on GB10.

• The HUMMING NVFP4 MoE backend cannot initialize on GB10 — NVML has no memory clock (2026-07-28, vLLM 0.26.0, Spark-2). HUMMING is new to the candidate list in 0.25.1/0.26.0 and it is not rejected by the device check: --moe-backend humming passes selection, loads all 43 GiB of weights, and only then dies with pynvml.NVMLError_NotSupported in humming/utils/device.py:calculate_gpu_bandwidth(), which calls pynvml.nvmlDeviceGetMaxClockInfo(handle, NVML_CLOCK_MEM) to estimate bandwidth for its tuning heuristics. GB10 does not expose one — nvidia-smi -q -d CLOCK prints Memory : N/A next to SM : 3003 MHz, the same unified-memory blind spot that makes --query-gpu=memory.used return [N/A]. So unlike the fast-failing backends above, this one costs a full five-minute boot to discover; don't budget a sweep around it.

• method: "draft_model" is a request, not a guarantee — vLLM 0.26.0 will build an MTP module instead if the draft repo happens to ship MTP weights (2026-07-28, Agents-A1-FP8 + sakamakismile/Agents-A1-4B-NVFP4, vLLM 0.26.0, Spark-2). Asking for {"model": ".../Agents-A1-4B-NVFP4", "method": "draft_model", "num_speculative_tokens": 1} died at startup with a bare AssertionError, no message, from vllm/model_executor/parameter.py:153 (assert self.data.shape == loaded_weight.shape) — reached through vllm/model_executor/models/qwen3_5_mtp.py:305, i.e. the MTP loader, which is not what was asked for. Cause: that draft repo carries a model-mtp-bf16.safetensors beside its NVFP4 weights and an mtp layer count in its config, so vLLM classified it as a Qwen3.5 MTP model and then handed a BF16 MTP tensor to a 4-bit-packed parameter of half the logical width. Two things to carry: list the draft repo's files before writing the spec config (a stray extra checkpoint changes which model class you get), and a half-width shape assert during a draft load is a quantization mismatch, not a sharding one — the same trap as elsewhere in these notes, now reachable without any TP at all.

• FP8 MoE on sm_121 auto-selects the TRITON backend on vLLM 0.26.0 (2026-07-28, Agents-A1-FP8, compressed-tensors float-quantized). Logged as Using TRITON Fp8 MoE backend out of potential backends: ['AITER', 'FLASHINFER_TRTLLM', 'FLASHINFER_CUTLASS', 'DEEPGEMM', 'VLLM_CUTLASS', 'TRITON', 'MARLIN', …]. That is a different pick from the NVFP4 path on the same engine and the same box (which lands on VLLM_CUTLASS), so the backend menu is per quant format as well as per version and per checkpoint. Served correctly unmodified. All five alternatives were then tried, and the two failure messages mean different things: VLLM_CUTLASS and FLASHINFER_TRTLLM die with "kernel does not support current device cuda" — no sm_121 build, a wait-for-upstream answer — while DEEPGEMM and FLASHINFER_CUTLASS die with "kernel does not support quantization scheme QuantKey(f8e4m3fn,scale(f32,static,per_channel),symmetric)x…", which is about the checkpoint (they want block-wise scales; this export has per-channel static ones) and would be fine on a different FP8 export on the same box. Only MARLIN also boots, and it measured 39.08 tok/s against TRITON's 38.77 at a 2,042-token prompt — not separable. Two practical notes: the CLI spelling differs from the log spelling (--moe-backend deep_gemm|cutlass|flashinfer_cutlass|marlin|triton; passing DEEPGEMM is an argparse error), and all four failures happen in seconds, before any weight loading, so probing the whole menu costs about a minute, not a boot each.

• A quantization ignore list can name modules that do not exist, and it fails silently open (2026-07-28, InternScience/Agents-A1-FP8). Its list is 10,641 entries, of which 10,240 are model.language_model.layers.N.mlp.experts.N — the parent expert module, not the gate_proj/up_proj/down_proj leaves that compressed-tensors actually matches against. So the entries matched nothing and every routed expert is FP8 regardless, which the header confirms (F8_E4M3 weights with BF16 weight_scale). Reading the list at face value predicts a checkpoint that is 86% BF16 and 37.7 GB, which is impossible. Settle "what got quantized" from the safetensors header's dtypes, never from the ignore list alone — the list is a statement of intent and the header is the outcome.

• Greedy determinism on sm_121 is a BATCH-SIZE property, not an engine property (2026-07-28, Laguna-XS-2.1, measured on both engines with the same tester). Supersedes the vLLM 0.25.1 note below, which was right that greedy output varies but wrong about the cause. Measured, 6 identical requests at temperature: 0, seed: 0:

  	cold (1st request)	warm, batch=1	warm, concurrent
  llama.cpp (b25-7ad9bd2, Q4_K_M)	differs	24/24 identical	identical
  vLLM 0.26.0 (NVFP4)	differs	12/12 identical	splits

  Sweeping ONLY the concurrent batch size on vLLM, same prompt, warm server: n=1 → 1 distinct, n=2 → 2, n=3 → 2, n=4 → 2, n=6 → 5, n=8 → 2. Batch=1 is deterministic; any n>1 is not, including n=2. Divergence appears at the first sampled token: batching changes GEMM shapes and reduction order enough to flip the argmax, after which completions differ entirely. The concurrent split is reproducible (same hashes, same 1-vs-5 ratio across runs), so it is systematic numerics, not randomness.

  Two consequences:

  • A token-identity correctness check IS available on the Spark, on either engine, provided the harness pins batch size 1 and discards the first request. This is presumably why the mlxfast challenge ranks a serial track only — it is the one regime where exact-output comparison is stable.
  • The first request after server start is never comparable to the rest, on either engine. bench/harness.py already discards warmup; anything doing output comparison must too, or it will report a phantom mismatch.

• Greedy output on vLLM 0.25.1 / sm_121 is not reproducible run-to-run (2026-07-27, Laguna-XS-2.1-NVFP4, vLLM 0.25.1). Four identical requests (temperature: 0, seed: 0, 300 max_tokens) to the same server returned three distinct completions. Consequence: the standard speculative-decoding correctness check — "greedy spec output must be token-identical to greedy no-draft output" — cannot be run on this build; a mismatch proves nothing. Gate a draft on acceptance rate from /metrics instead, which was stable to ~0.5 pt across the same runs. Suspects are prefix caching, varying batch/cudagraph shapes, and FlashInfer reduction order; not investigated further.

• llama.cpp: builds clean from source with CUDA on aarch64 (~15 min, -j 20). Compiling needs only the toolkit, not a working driver — useful while waiting on a reboot.

• vLLM: pip wheels for aarch64+CUDA 13 exist and work (0.24.0, torch 2.11). Gotcha: Triton JIT-compiles helpers at import time with the system gcc and needs python3.12-dev (apt) or every model load fails with a misleading "model architecture failed to be inspected" error — the real error (Python.h: No such file or directory) is buried in a subprocess.

• vLLM FP8 MoE on GB10 needs VLLM_USE_DEEP_GEMM=0 (2026-07-05, vLLM 0.24.0). DeepGEMM targets datacenter Blackwell; on GB10 (sm_121) loading an FP8 MoE model crashes with Assertion error ... Unknown SF transformation. vLLM auto-disables DeepGemm for the dense layers but the MoE backend still selects DEEPGEMM — disable it globally and it falls back to CUTLASS/Triton.

• Invoke vLLM with its venv on PATH, not just by absolute path. flashinfer JIT-compiles sampling kernels at startup and shells out to ninja; calling ~/venvs/vllm/bin/vllm directly leaves the venv's bin off PATH and the engine dies at KV-cache profiling with FileNotFoundError: 'ninja'. Use PATH="$HOME/venvs/vllm/bin:$PATH" vllm serve ....

• Gemma 4 assistant (speculative decoding) on llama.cpp, 2026-07-06 state: support landed in master this week and has sharp edges. (1) Community GGUFs may carry arch gemma4_assistant (underscore) while llama.cpp expects gemma4-assistant — convert from Google's safetensors yourself. (2) The assistant conditions on target hidden states: use --spec-type draft-mtp, not draft-simple (which dies with "requires ctx_other"). (3) Acceptance rates are implausibly low even on trivially predictable output (27% on number sequences; should be 90%+), making spec decoding a net 38% slowdown. Likely immature upstream wiring — re-test after llama.cpp updates before concluding anything about speculative decoding on this hardware. Confirmed via vLLM (same draft, same day): vLLM's gemma4_assistant support gets 45–51% acceptance and a 28–46% decode speedup (--speculative-config '{"model": ..., "num_speculative_tokens": 4}'). Speculative decoding works on Spark; llama.cpp's brand-new assistant path doesn't yet. Acceptance counters come from vLLM's /metrics endpoint (vllm:spec_decode_num_{draft,accepted}_tokens_total).

• "NVFP4" is not one thing — the ignore-list is a decode-speed knob (2026-07-06). Two NVFP4 checkpoints of the same Qwen3.6-35B-A3B arch, same nvfp4-pack-quantized format, same W4A4 scheme, near-identical memory (67.5 vs 69.6 GB): official NVIDIA decoded at 77 tok/s, a community quant (AEON-7 heretic) at 42 — 46% slower — while prefill was identical (~6,230). Cause: the community recipe leaves the hybrid model's linear-attention projections (in_proj_a/b/z, out_proj) in BF16 via a longer ignore list. Those unquantized layers run a slower per-token path that throttles decode but is hidden in (parallel, compute-bound) prefill. Lesson: when benchmarking a quant, diff quantization_config.ignore in config.json — same format label can mean very different decode speed. Not just a quality difference.

• vLLM 0.24 can't run external-draft spec decoding on hybrid models (2026-07-06). DFlash (and any draft_model/eagle-style external draft) against Qwen3.6-35B-A3B dies at engine init with AssertionError in unify_kv_cache_spec_page_size. Cause: Qwen3.6 is hybrid — its linear-attention layers hold a fixed-size recurrent state whose page_size_bytes doesn't scale with block size, so it can't be padded to match the attention pages the draft adds. vLLM does support DFlash — just not with a hybrid target here. Corollary: this is likely why Qwen ships MTP variants (self-speculation reuses the model's own layers, no foreign page size to reconcile). Workaround: use llama.cpp (--spec-type draft-dflash), which handles the hybrid target; convert the DFlash drafter with convert_hf_to_gguf.py --target-model-dir <target>.

• Thinking models (Qwen3 family): tokens stream as reasoning_content before content — benchmark clients that only watch content measure TTFT wrong. See bench/README.md for the other benchmarking pitfalls (prefix caching, ignore_eos, per-slot context).

• DFlash speculative decoding is Qwen3-only in vLLM 0.24.0 (2026-07-12, trying poolside/Laguna-XS-2.1-DFlash). The generic DFlash plumbing is there (DFlashProposer, method="dflash" accepted) but the model registry only maps DFlashDraftModel → qwen3_dflash.DFlashQwen3ForCausalLM — no Laguna draft class. Serving with method=dflash fails: "Model architectures ['DFlashLagunaForCausalLM'] are not supported." The Laguna draft is upstream PR vllm-project/vllm#46853, unmerged here. Back-porting the PR onto 0.24.0 by hand loads and generates coherently but gets 0% draft acceptance (the diff assumes newer internals; one KV-norm hunk needed a manual merge and the result is numerically off) — so no speedup, and the greedy-acceptance gate correctly rejects it. Lesson: for a brand-new arch, spec-decode support usually lags base support by a release; gate any spec-decode benchmark on spec_decode_num_accepted_tokens_total > 0 before trusting it. (llama.cpp is worse off for Laguna — build 15/ee445f9 has draft-dflash but no laguna architecture at all, so it can't serve the base model either; needs the model's llama.cpp PR #25165.)

• vLLM 0.24.0 crashes on mixed-precision grouped WNA16 MoE (2026-07-12, serving poolside/Laguna-XS-2.1-INT4). The checkpoint is compressed-tensors mixed precision — some MoE experts 4-bit, some 8-bit, both group-quantized (group_size=128). check_moe_marlin_supports_layer approves group_size=128 and routes the MoE to Marlin, but CompressedTensorsWNA16MarlinMoEMethod then asserts 8-bit experts must be channel-wise (assert self.group_size == -1) and dies at load with a bare AssertionError / "Engine core initialization failed". The support check and the kernel disagree — a vLLM bug, not a hardware limit. Workaround: force the non-Marlin (Triton) CompressedTensorsWNA16MoEMethod, which handles grouped 8-bit and generates coherent output (verified), by monkeypatching check_moe_marlin_supports_ layer to return False. Ship it as a sitecustomize.py on PYTHONPATH so it also loads in vLLM's spawned engine/worker processes — see bench/patches/laguna-int4/ and the serveCommand in bench/configs/laguna-xs-2-1-int4--vllm.json. The Triton MoE path is a bit slower than Marlin, so INT4 decodes marginally below FP8/NVFP4 on the same model. FP8 and NVFP4 quants of the same model load with no workaround.

Models larger than RAM (disk streaming)

• Unified memory doesn't help you load a too-big model — it hurts (2026-07-08). A 557 GB checkpoint (Hy3, 295 B params) can't be loaded on a 128 GB Spark, and because CPU and GPU share one pool, the usual "load to CPU RAM, move blocks to GPU" trick fails two ways: loading the whole thing to cuda OOMs, and even device_map="cpu" overflows the pool. Loading a big model to cuda while something else holds the pool can also hang the CUDA allocator, not just OOM.
• Stream from the NVMe instead. For layer-wise workloads (REAP calibration, and its prune), build the model on meta (0 resident) and materialize each decoder layer's weights from the on-disk safetensors shards only while that layer is processed, then release it back to meta. Peak RAM is ~one layer (~7 GB for Hy3) regardless of the 557 GB total — the 3.7 TB NVMe is the real capacity limit, not the 128 GB RAM. See the Training page for the REAP recipe.
• Two Sparks (242 GB) still can't hold a 557 GB model — combining memory helps only in the 128–242 GB band (model-parallel). Above that, disk streaming is mandatory regardless of node count; the second Spark buys throughput (data-parallel calibration) or, with pipeline-parallel, more disk capacity.
• Clear __pycache__ after editing library code (2026-07-08). A long cuda/cpu device-mismatch hunt turned out to be stale .pyc: a running process kept executing old bytecode and ignoring source fixes. find src -name __pycache__ -type d -exec rm -rf {} + when a fix "isn't taking."

Clustering: two-Spark pipeline-parallel inference (vLLM)

Running the 157 GB REAP-pruned Hy3 (too big for one Spark, fits across two) with vllm serve --pipeline-parallel-size 2 over Ray + the 200G fabric. The model loads fine; the friction is all environment (2026-07-08):

• Copy the venv, don't rebuild. The Sparks are mirror-image (same aarch64/GB10, same /home/joemuller paths, Python 3.12), so rsync -a of the working ~/venvs/vllm to the peer gives a functioning vLLM+Ray+torch with no compile — the .so/paths all resolve. Saved a from-scratch vLLM build.

• vLLM 0.24.0 supports HYV3ForCausalLM natively (in the model registry).

• NCCL over RoCE isn't turnkey for vLLM — PP group init died with "NCCL error: unhandled system error". NCCL_IB_DISABLE=1 + NCCL_SOCKET_IFNAME= the fabric netdev forces TCP over the 200G link, which is plenty for PP's light inter-stage traffic. (RoCE ib_write_bw works; wiring it into NCCL is a separate GID/HCA tuning job not worth it for a functional check.)

• ninja must be on the Ray worker's PATH — flashinfer JIT-compiles a CUTLASS fused-MoE kernel (gen_cutlass_fused_moe_sm120) on first forward and shells out to ninja (needs nvcc too — both present under /usr/local/cuda

  • a pip install ninja). Ray raylets inherit the login environment, and actors get PATH from there — not from the ray start shell's exports. Fix: symlink ninja into ~/.local/bin (the stock ~/.profile adds it when the dir exists) and start raylets via login shells (bash -lc). Verify with a @ray.remote probe of shutil.which("ninja") on each node.

• Disable Ray's memory monitor on unified memory — RAY_memory_monitor_refresh_ms=0. It counts the 75 GB of GPU weights as system RAM, decides the node is full, and OOM-kills a worker after the model already loaded and KV cache allocated.

• Hard-reset Ray between failed attempts — killed vLLM runs leak placement groups that keep the GPUs "reserved", so the next launch hangs on "Waiting for creating a placement group". ray stop --force + pkill -9 -f 'EngineCore| RayWorkerProc' on both nodes, then confirm ray status shows 0.0/2.0 GPU with no "reserved in placement groups".

• Gloo needs the same interface pinning as NCCL for multi-node vLLM (2026-07-09). PP2 launch died with Gloo connectFullMesh failed ... Connection refused, remote=[127.0.0.1] — the Sparks resolve their own hostnames to loopback, so torch's CPU (gloo) process group advertises 127.0.0.1 to the peer. Fix: GLOO_SOCKET_IFNAME=enp1s0f1np1 + per-node VLLM_HOST_IP=<fabric ip> on both raylets and the vllm serve process (raylets pass env to Ray actors; setting it only on the server is not enough).

• SM120 SASS cubins run unmodified on GB10/sm_121 (2026-07-09). The vLLM-Moet hand-written kernels (2-bit MoE QMMA.SF GEMMs, assembled by cubit with a mercury compat stub) load via cuModuleLoad and op-validate bit-deterministically at rel err 2-3e-3 on GB10 — CUDA treats 12.0/12.1 as one SASS family. Don't assume "SM120-only" kernel projects exclude Spark; test with a 30-line driver-API loader first.

• DeepGEMM nv-dev builds clean on aarch64 (a6b593d2, ~7 min, needs only pybind11/wheel + system CUDA 13 headers). The vLLM-0.24-vendored copy lacks family-120 host paths; the site-packages wheel shadows it.

• flashinfer-jit-cache cu130 wheels are x86-only — on Spark, flashinfer 0.6.14 falls back to JIT-compiling every kernel on first use (slow first request; same ninja/nvcc PATH caveats as the Ray note above).

• Pinned-host "offload" tiers double-allocate on unified memory — designs that keep a pinned-host weight store + a GPU cache pool (vLLM-Moet's delta & base-cache tiers) allocate the same physical LPDDR twice on GB10. Run such stacks in fully-resident mode with the offload tiers disabled (VLLM_MOE_W2_DELTA_GB=0, no BASE_CACHE_GB); "GPU-resident" already means "in the one memory pool".

• The GB10 unified-memory load war (2026-07-09, vLLM-Moet port; full details in ~/Dev/vLLM-Moet branch spark-gb10). Getting a 149 GiB checkpoint converted+loaded on a 121 GiB unified box surfaced six distinct platform behaviors, each capable of OOMing the load on its own:

  1. cudaMalloc takes system RAM but does not trigger page-cache reclaim — streaming a big checkpoint fills the cache and starves the allocator ("754 MiB free" CUDA OOM with 40+ GiB of clean cache). Fix: posix_fadvise(DONTNEED) consumed shards during load.
  2. Pageable H2D copies are CPU-executed writes into UVM — destination pool pages become host-resident and device-backed, and allocator churn accumulates them (~2-3 GiB/layer). Fix: pinned bounce + one persistent device buffer per transfer shape.
  3. cudaHostAlloc (pinned) is shmem-backed with power-of-2 rounding — a 20 GiB pinned ask costs ~33 GiB physical.
  4. Plain pageable host memory is readable by GPU kernels via ATS at full bandwidth — the winning place to keep big read-only weights.
  5. glibc arena free-lists retain tens of GiB of ~10 MB loader buffers; MALLOC_MMAP_THRESHOLD_=65536 returns them.
  6. The kernel OOM killer will take the desktop/session with it — run model engines under systemd-run --user --scope -p MemoryMax=... (a cgroup MemoryHigh throttle can stall a loader indefinitely; prefer a hard MemoryMax). The durable architecture: convert weights offline once (prepacked planes on disk) and serve-load into pageable host memory — no staging, no requant, no churn.

• vLLM KV sizing on GB10 counts system-wide usage: budget = gpu_memory_utilization × total − used, where "used" includes host allocations (even other processes). Tune util to the system picture, not the model's CUDA footprint.

• DGX OS ships memlock hard-limited to 8 MB — this is why NCCL-over-RoCE "mysteriously" fails (2026-07-10). ibv_reg_mr (RDMA memory registration) needs locked pages; NCCL surfaces the 8 MB ceiling as "unhandled system error" at init. Fix: limits.d memlock unlimited + start the process tree from a FRESH login session (inherited shells keep the old limit; self-ssh works). Solved the hy3-era mystery.

• DGX OS also starts sessions with PR_SET_THP_DISABLE — madvise'd transparent hugepages silently no-op (THP_enabled: 0 in /proc/self/status). One unprivileged prctl(41, 0) per process re-enables. Also: CONFIG_READ_ONLY_THP_FOR_FS is not set, so file-backed THP (MADV_COLLAPSE on mmap'd weights) is unavailable on the stock kernel.

• vLLM decode-step profiling on 2-node PP (--profiler-config '{"profiler":"torch",...}' + VLLM_SERVER_DEV_MODE): under CUDA graphs, in-graph kernels don't emit individual trace events — bucket totals still attribute wall time. GLM-5.2 PP2 decode: ~30% expert GEMMs, ~70% comm/pipeline-wait — which is why kernel-side tuning moved little and RoCE/batching moved more.

Speculative decoding

• Speculation depth k has a knee — past it, tok/s drops (2026-07-16). Observed while tuning num_speculative_tokens on Spark: throughput climbs, peaks at a small k, then falls. Mechanism: the target verifies the draft in one pass and keeps only the longest correct prefix, so the first rejection discards the rest. With per-token acceptance α, expected accepted tokens = (1 − α^(k+1))/(1 − α), which saturates at 1/(1−α) (α=0.7 → ~3.3 tokens, no matter how big k is). Marginal accepted tokens decay as α^k while cost keeps rising — an autoregressive drafter (EAGLE / draft-model) costs ∝ k sequential passes, and the verify pass tips compute-bound at large k. So the useful window is small (~k=3–5 here); overshoot is pure draft-and-discard.
• Why the knee is early on GB10 specifically: decode is hard memory-bandwidth-bound (~273 GB/s unified LPDDR5x), and the drafter shares that same starved bus. Every rejected high-k draft token is wasted bandwidth stolen from useful decode, so the crossover to net-negative arrives sooner than on a bandwidth-rich discrete GPU. α also drops on high-entropy text (creative prose) → lower ceiling → even smaller optimal k; it's high on repetitive text (code, RAG-with-quoting, edits), which is where n-gram / prompt-lookahead drafters shine with zero training.
• DSpark sidesteps the fixed-k problem via a confidence head that predicts per-position acceptance and schedules verification — it cuts the block where confidence drops instead of committing to one k. Its semi-AR shape (parallel backbone proposing the whole block + a lightweight serial Markov head so each token glances one back) is a direct answer to purely parallel drafters' tail-drift. On 2× DGX Spark, DeepSeek-V4-Flash-DSpark code-gen went ~40–45 → ~60–67 tok/s single-stream (see models/ bring-up log). Served via vllm serve … --speculative-config '{"method":"dspark","num_speculative_tokens":7}'.

Unified-memory overcommit: the swap-wedge and how to get out (2026-07-19)

Serving Hy3-NVFP4 (169 GB checkpoint, TP2 across both Sparks) while pushing the KV pool. The failure mode and the recovery are both worth knowing:

• Per-cgroup MemoryMax does not bound the sum. systemd-run --scope -p MemoryMax=... -p MemorySwapMax=0 protects one scope, but the raylet, the vLLM serve driver and the OS are separate scopes. Caps of 110G (raylet) + 8G (driver) on a 121 GiB box still permit a global overcommit — and the box wedged. Budget them as a sum: head raylet 104G + driver 8G + OS ~4G; worker raylet 110G + OS ~3G. MemorySwapMax=0 is what converts a miss into a clean kill instead of a thrash, so always set it.
• A wedged node is pingable but SSH-dead. The kernel network stack keeps answering ICMP while userspace (sshd, and tailscaled) is starved — so the node drops off Tailscale entirely. Diagnose from the peer over the fabric/LAN IP (192.168.100.x / 192.168.0.x): ssh returning "connection timed out during banner exchange" means thrashing, not crashed.
• It can recover without a hard reboot. Killing the job on the surviving peer (ray stop --force, then pkill -9 -f 'ray::') released the cross-node pressure, and the wedged Spark came back on its own within a minute or two. Try that before power-cycling.
• RuntimeError: cancelled in shm_broadcast means a TP worker died — it is not a comms bug. The head logs "Engine core initialization failed" while the actual cause is on the other machine (ours was at 120 GB used / 1 GB free). Always check the peer's free -g first.
• RoCEv2 GID indexes differ per node, not just across reboots: on 2026-07-19 Spark-1's v2 IPv4 GID was index 3 while Spark-2's was 6, at the same time. Resolve per node from sysfs at start time; never hardcode.
• vLLM's --gpu-memory-utilization is a weak lever on GB10 because the utilization math counts system-wide usage on unified memory. Pin the KV pool explicitly with --kv-cache-memory-bytes when you want a specific budget.
• Raising --max-model-len raises the startup peak, not just steady state. A config with 23 GiB free at 64K did not survive pinning 23 GiB/rank of KV at 256K — profiling and activation transients need headroom on top. Step up in stages and keep several GiB spare.
• Read memory at steady state, not during weight load. The single most expensive mistake of the day. While shards are loading, the KV pool has not been allocated yet, so free -g flatters you by roughly the size of the KV budget you are about to request. Hy3-NVFP4 TP2 showed ~98 GB used / ~23 GB free mid-load and 116 GB used / 4 GB free once serving — we sized a KV climb against the former and wedged both boxes. Always re-read after the API answers /v1/models.
• KV pool size and --max-model-len are different knobs — divide before you grow. vLLM's "maximum concurrency" is just KV pool ÷ max-model-len. Since the Sparks are single-user workstations, any concurrency above ~1.0x is KV we paid for and can't use. Hy3-NVFP4 TP2 ran at 64K against a ~195K-token pool (3.03x) purely because 64K was the upstream default; moving to 160K gave 2.5× the context at identical memory (116 GB/node) and identical decode speed (13.56 vs 13.51 tok/s — noise). Always divide the two numbers before concluding a model is context-limited. Leave margin above 1.00x though: the pool varies 1-2% between boots (195,376 and 198,608 on the same config) and the server refuses to start if max-model-len exceeds it.
• MTP speculative decoding is broken for HYV3 at TP>1 (vLLM 0.24.0) (2026-07-19). Hy3-NVFP4 ships a real MTP layer — num_nextn_predict_layers: 1, a complete MoE layer at model.layers.80 (1745 tensors vs 1741 for a normal layer) with eh_proj/enorm/hnorm — and vLLM resolves it correctly (Resolved architecture: HYV3MTPModel, speculative.py:514 maps hy_v3 → hy_v3_mtp). But the drafter fails to load at TP2 with The size of tensor a (2048) must match the size of tensor b (4096): hy_v3_mtp.py:102 declares eh_proj as a plain nn.Linear(hidden*2, hidden) instead of a TP-aware parallel layer, so the draft params are built half-width while the checkpoint holds full width. Fails cleanly at load (no wedge). Would need a patched hy_v3_mtp.py, a newer vLLM, or PP instead of TP.
• MTP speculative decoding on Hy3-NVFP4: +43% single-stream, and the vLLM fix it needed (2026-07-19, supersedes the "broken at TP>1" note above). MTP does work at TP2 once one vLLM bug is patched. HYV3SharedHead builds its ParallelLMHead with no prefix, so ModelOptNvFp4Config.is_layer_excluded never matches "lm_head"; on a checkpoint that keeps lm_head in BF16 the MTP head is therefore built as a packed NVFP4 param (hidden/2 = 2048) and then handed the target's full-width BF16 lm_head weight (4096) — the 2048-vs-4096 error was 4-bit packing, not a TP shard. Fix: pass prefix=maybe_prefix(prefix, "lm_head"), mirroring hy_v3.py's own head. Patch: vLLM-Moet/patch/hyv3-mtp-shared-head-quant.patch. Upstream main has the identical defect. Result at k=1: 19.45 tok/s vs 13.56 (+43%), 61.3% draft acceptance, 1.61 tokens/step, TTFT 176→252 ms, KV pool 195,376→176,784 tokens (draft costs ~1.3 GiB/rank out of KV, total residency unchanged), 160K context preserved.
• Debugging tip: trace a failing traceback line to its owning function (grep -rn the literal source line under site-packages) rather than inferring the layer from shapes — it named ParallelLMHead immediately after two wrong guesses. And instrument the node the traceback names: each Spark has its own venv, so patching only the head node means the diagnostic never fires.

2026-07-19 — cudagraph capture is coupled to max-num-seqs

Serving configs that derive --max-cudagraph-capture-size from max-num-seqs (e.g. MAX_NUM_SEQS × (spec_tokens+1)) make max-num-seqs a decode knob, not just a scheduler knob. The standard single-user advice — drop it to 1–2 so the scheduler isn't constrained — backfires: on DeepSeek-V4-Flash-DSpark, 6→2 shrank capture 24→8 and decode fell ~1/3 (code 62→47 tok/s, prose 44→26) with run-to-run spread widening from a few percent to ~2x. Check whether capture size is derived from it before lowering it.

2026-07-19 — docker compose logs replays dead containers

docker compose logs (and the log tails in most bring-up scripts) replay the full history of previous containers in the same compose project. A fresh start can therefore print a stack trace from days ago before it prints anything from this run. We burned time chasing c10::Error frames that turned out to carry a W716 timestamp on 07-19. Use docker logs --since 10m <container> when diagnosing a startup, and always check timestamps before believing a traceback.

2026-07-19 — when a long-context config is prefill-bound

For very long context, decode tok/s can be nearly flat while TTFT dominates completely. On DeepSeek-V4-Flash-DSpark decode stayed 39–50 tok/s from a 1.4K prompt to a 275K prompt, but TTFT went 1.2 s → 199 s (prefill 1,260 → 1,382 tok/s). Two consequences: (1) benchmarking only short prompts tells you nothing about the long-context experience, and (2) decodeTps is the wrong figure to optimize for such a recipe — measure prefill at length as well. Note also that raising max-num-batched-tokens to chase prefill costs KV pool and, if capture size is coupled to max-num-seqs, costs decode too.

2026-07-19 — FlashInfer CUTLASS runs NVFP4 MoE on sm_121 (marlin is not the only option)

Standing guidance from the Hy3 work was that --moe-backend marlin (dequant FP4→FP16) is the only NVFP4-MoE backend that runs on GB10, because flashinfer ships no sm_120/121 cubins. That is no longer true on vLLM 0.24.0 + torch 2.11.0+cu130. Serving unsloth/Qwen3.6-35B-A3B-NVFP4-Fast with no --moe-backend flag at all, vLLM auto-selects:

Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend out of potential backends:
  ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED',
   'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'EMULATION']
Using FLASHINFER attention backend out of potential backends: ['FLASHINFER', 'TRITON_ATTN']

and it serves correctly. The TRT-LLM fused-MoE autotuner does run and reports Skipped N unsupported tactic(s) for trtllm::fused_moe::gemm1/gemm2 — i.e. it prunes the tactics that have no sm_121 kernel and keeps the ones that do, rather than failing. So don't hardcode --moe-backend marlin on new work; let the auto-selection run and check the log line for what it actually picked. The marlin-only claim should be read as specific to the Hy3 checkpoint/vLLM build it was learned on, not a property of GB10.

2026-07-19 — qwen3_5_mtp is silently rewritten to mtp

vLLM 0.24.0 accepts --speculative-config '{"method":"qwen3_5_mtp",...}' but logs method 'qwen3_5_mtp' is deprecated and replaced with mtp and rewrites it. Write "method": "mtp" directly. The MTP layer of unsloth/Qwen3.6-35B-A3B-NVFP4-Fast is left unquantized by the checkpoint (re:^mtp.* in the quant config's ignore list) and loads with no vLLM patch — unlike Hy3, whose MTP shared head needed a one-line quantization fix. Cost of the draft layer: model residency 20.58 GiB → 22.15 GiB (~1.6 GiB).

2026-07-19 — hybrid linear attention makes KV nearly free

Qwen3.6-35B-A3B is a hybrid: full_attention_interval: 4, so only 10 of 40 layers keep a real KV cache, and those have num_key_value_heads: 2 at head_dim: 256. Measured on one Spark at --gpu-memory-utilization 0.85: 75.12 GiB of KV → 7,634,689 tokens, i.e. ~10.3 KiB/token and 29.1x concurrency at the model's full 262,144-token context. Capping the pool with --kv-cache-memory-bytes 4294967296 (4 GiB) still yields 360,197 tokens = 1.37x at full context, freeing ~71 GiB.

The usual Spark advice — "raise --max-model-len until concurrency approaches 1.0x, KV is the thing you fight for" — inverts here. On a hybrid-attention MoE this small, context is free and the real question is what to do with the ~71 GiB you get back. Watch instead for Setting attention block size to 2128 tokens to ensure that attention page size is >= mamba page size: the linear-attention (mamba-style) state forces a large attention page, which is why the per-token KV figure is what it is.

2026-07-19 — the LAN is not what ping suggests: a 10/100 switch, and an admin-down port

Chasing "can a remote drafter reach the Sparks fast enough", the network turned out to be three separate faults, none of them the cable:

1. Spark-2's RJ45 (enP7s7) was administratively down — flags 0x1002 (BROADCAST,MULTICAST, no IFF_UP). A port in that state powers down the PHY, so ethtool reports Link detected: no and Speed: Unknown! even with a good cable plugged in. Don't diagnose a cable from ethtool until you've confirmed IFF_UP is set; check /sys/class/net/<if>/flags, not just ip -brief addr.

2. Don't attribute a link partner to the wrong box. Spark-2's enP7s7 reported its partner advertising 10baseT/100baseT only, and I first read that as "there's a 10/100 switch in the path." Wrong: that cable runs to the GPU rig's second NIC, which has no driver bound (see below), and an unbound PHY happily autonegotiates in a degraded 10/100 default. Before concluding anything about a switch, confirm which device is on the other end. Note also that the igc driver (Intel I226-V) reports no Link partner section at all, so absence of that data is not evidence of anything.

   Separately: a cable with damaged/missing pairs negotiates 100 Mb/s cleanly — 1000BASE-T needs all four pairs, 100BASE-TX needs two — so zero CRC/error counters is fully consistent with a bad cable. Speed alone can't distinguish cable from port; swap-test to localize it.

3. Bringing up a second interface on an already-used /24 blackholes it. enP7s7 came up with 192.168.0.6/24 at metric 0, beating the WiFi route's metric 600, so Spark-2 preferred a 100 Mb wired path whose segment has no uplink (ip neigh → INCOMPLETE for the gateway; 100% loss to gateway, to Chonky, and inbound). RoCE on 192.168.100.x is a different subnet and was unaffected — GLM serving never noticed. If you dual-home a Spark on one subnet, set an explicit metric or use a separate subnet.

Practical upshot for off-box inference work: the only fast path between these machines is the existing 200 Gb ConnectX/RoCE pair. Note also that both Sparks have a second CX port up at 200000 Mb/s (enP2p1s0f1np1) carrying no IPv4 — a spare fabric link, already cabled.

4. ROG Crosshair X870E Hero has two NICs; Ubuntu 24.04 only drives one. lspci -nnk shows 0c:00.0 Intel I226-V (2.5 GbE, driver igc, becomes eno1) and 0d:00.0 Realtek RTL8126 (5 GbE, no kernel driver in use). Mainline r8169 gained RTL8126 support around kernel 6.13; this box runs 6.8.0, and modinfo r8169 | grep 10EC.*8126 returns 0 matches. So the 5 GbE port is electrically dark — a cable plugged into it shows carrier at the far end but no netdev exists locally and nothing routes. Fixes: Realtek's out-of-tree r8126 via DKMS (lower risk on a GPU rig — r8126-dkms is NOT in Ubuntu's repos, only r8125-dkms for the different RTL8125 chip), or the HWE kernel (linux-generic-hwe-24.04, currently 7.0.0) which has it natively.

2026-07-20 — the first cuda.synchronize() in a step absorbs the whole queue

Timing vLLM's dspark proposer stages (VLLM_DSPARK_PROFILE=1, which wraps each stage in a _sync_ms() = torch.cuda.synchronize() + wall clock) initially reported:

draft_fwd       10.97 ms
ctxkv+inputs   362.74 ms     <- against an ~89 ms total cycle
markov_sample    4.88 ms

362 ms inside an 89 ms cycle is impossible. The cause is generic to any sync-based GPU timing: the first synchronize() of an iteration drains everything already queued, so it bills the previous step's entire verify forward to whichever stage happens to sync first. Stages that sit between two syncs (here draft_fwd, markov_sample) are fine — they measure clean intervals. Only the leading one is contaminated.

Fix is one line: torch.cuda.synchronize() before starting the first timer, to drain the queue first. After that:

draft_fwd        9.91 ms
ctxkv+inputs     1.23 ms     <- was 99.7% other people's work
markov_sample    4.75 ms

So when a profile shows one suspiciously enormous stage, suspect sync attribution before believing the stage is slow. Corollary for reading any existing GPU profile: trust interior stages, distrust the first one.

Related: profiling mode is heavily self-distorting here (three extra syncs per step drop throughput several-fold), so use it for the ratio between stages, never for absolute throughput — the same warning spark/MTP-DRAFT-COST.md records for the torch profiler.

2026-07-20 — quantizing a speculator: three things that block it

Quantizing the dspark drafter (not the target) on the vLLM-Moet fork ran into three separate obstacles. All three are generic to speculative decoding, not to this model.

1. The draft's ModelConfig honors quantization, but only from the checkpoint. SpeculativeConfig builds it with quantization=self.quantization (config/speculative.py), and the draft model calls get_draft_quant_config(vllm_config) — so a draft checkpoint carrying its own quantization_config in config.json Just Works through the normal compressed-tensors path. What does not work is the --quantization fp8_per_channel style shorthand: resolve_quantization_config() is only called from engine/arg_utils.py for the target, so the draft ModelConfig gets quantization="fp8_per_channel" with quantization_config=None, and get_quant_config then falls through to file-based loading and dies with "Cannot find the config file". Online quantization for a drafter needs ~3 lines plumbing resolve_quantization_config into SpeculativeConfig.

Worth knowing this fork has online quantization at all (model_executor/layers/quantization/online/): it quantizes bf16 weights during loading, so FP8 needs no pre-quantized checkpoint. Linear methods are 8-bit only (fp8 per-tensor / per-block / per-channel, mxfp8); INT4 linear still requires a real checkpoint.

2. DFlash bypasses quantized dispatch in two places. qwen3_dflash.py did F.linear(hidden_states, self.qkv_proj.weight, ...) in the per-step forward, and sliced qkv_proj.weight[q_size:] in _build_fused_kv_buffers. Under a quantized checkpoint .weight either doesn't exist (marlin stores weight_packed) or is transposed (fp8 PTPC transposes in process_weights_after_loading) — so the first raises and the second silently returns garbage. Route the forward through self.qkv_proj(...) and reconstruct the fused KV rows by unpacking.

Useful ordering fact: _build_fused_kv_buffers() is called at the end of load_weights, i.e. before process_weights_after_loading, so the weights are still in on-disk compressed-tensors layout there (weight_packed [out, in // 8] int32 packed along the input dim). Both the q/k/v fusion and the [q_size:] row slice work on that layout, because the output dim is dim 0.

3. Speculators often have no HF modeling class. The dspark speculator's config.json declares architectures: ["DSparkDraftModel"] and ships no modeling_*.py (the model lives in vLLM), so AutoModelForCausalLM fails and llmcompressor cannot be used at all. The workaround is to quantize the raw safetensors directly — group-wise symmetric RTN is data-free, so no model and no calibration set are needed. See .claude/skills/case-study/scripts/quantize_raw_w4a16.py.

2026-07-20 — marlin W4A16 is ~4x at drafter shapes on sm_121

Micro-benchmark before building anything (GB10, TP2 shard shapes, M=3..64, group 128). The large layers — gate_up, fc, lm_head — come in at 3.8-4.1x BF16 at essentially identical GB/s (220-233 either way), which is the textbook bandwidth-bound signature: same bandwidth, 4x fewer bytes.

The small layers (qkv_proj, down_proj at 6144x6144) reported 14-18x and

1000 GB/s — above GB10's memory bandwidth, which is the tell that the 18 MiB int4 weight is sitting in L2 while the 75 MiB BF16 one isn't. That speedup is an artifact of benchmarking one layer in a loop; in the real model five layers stream and evict each other. Discount any per-layer microbenchmark whose implied bandwidth exceeds the machine's.

2026-07-20 — vLLM's compile cache ignores the drafter's quantization

Swapping a speculator from BF16 to a W4A16 checkpoint and restarting produced:

ValueError: too many values to unpack (expected 10)
  .../vllm/compilation/caching.py, line 217, in __call__
  .../vllm/v1/worker/gpu_model_runner.py, line 6597, in _dummy_run
      self.drafter.dummy_run(...)

The compiled graph for the draft is cached under ~/.cache/vllm/torch_compile_cache/<hash>/rank_N_0/**eagle_head** (the target's lives in the same <hash> dir under a different tag), plus an AOT artifact under torch_compile_cache/torch_aot_compile/<sha>/. Neither key changes when the draft checkpoint's quantization changes — verified directly: quantized and BF16 drafters produced the same AOT sha 2ebd355e78e0…. So vLLM reloads a graph compiled against the BF16 parameter list, while the quantized module now exposes weight_packed + weight_scale where weight used to be — hence the arity mismatch.

Fix without nuking 13 GB of target cache (recompiling the 92-layer target is the expensive part — the draft is ~2 min):

C=~/.cache/vllm/torch_compile_cache
mv $C/<hash>/rank_*/eagle_head       $C/<hash>/rank_*/eagle_head.w4a16
mv $C/torch_aot_compile/<sha>        $C/torch_aot_compile/<sha>.w4a16

Keep one copy per drafter variant and swap them when switching back and forth — the collision bites in both directions, so an A/B between a BF16 and a quantized drafter has to swap caches on every flip, on every node.

Caveat learned the hard way: swapping the old cache back is not enough if you also edited the model source. Restoring the pre-patch BF16 graph after patching qwen3_dflash.py gave a different stale-cache error —

AssertionError: Expected tensors only, but got: <class 'int'>

— because self.qkv_proj(...) returns a (tensor, bias) tuple where F.linear returned a bare tensor, so the traced graph changed shape for the BF16 path too. The source edit is not in the cache key either. For an A/B where the model file was patched, both arms must recompile from the patched source — otherwise the two arms differ by more than the variable under test, which quietly invalidates the comparison.

Symptom to recognize generally: an arity/unpack error inside caching.py or aot_compile.py right after changing what a model's parameters are (quantizing, fusing, adding an adapter) rather than changing shapes. That is a stale compiled graph, not a bug in the model code.

2026-07-20 — flashinfer 0.6.13 JIT-builds sm_121a kernels on first boot (~15 min, looks like a hang)

vLLM 0.25.1 pulls flashinfer-python==0.6.13 / flashinfer-cubin==0.6.13, which ship no prebuilt cubins for sm_121a. On the first serve of a given model/dtype/head-dim combination flashinfer compiles them from source with ninja/nvcc into ~/.cache/flashinfer/0.6.13/121a/cached_ops/…, one op at a time (sampling, then each batch_prefill_* variant).

It looks exactly like a wedge and will bait you into killing it:

• the log sits frozen for 10-20 minutes after torch.compile took NN s
• nvidia-smi shows 12-13 W and 0% GPU — nothing on the accelerator
• the engine's main thread blocks in anon_pipe_read, all threads S, 0% CPU

The tell that it is working, not hung: pgrep -af "ninja|cicc|nvcc" shows an active build whose -gencode reads arch=compute_121a,code=sm_121a, and ~/.cache/flashinfer/<ver>/121a grows. vLLM eventually reports it as part of the profiling step — ours logged Initial profiling/warmup run took 894.32 s and init engine (profile, create kv cache, warmup model) took 1051.00 s.

Subsequent boots with the same shapes reuse the cache and start in ~3 minutes, so budget the long boot once per venv/model-shape, not per experiment. Bumping the flashinfer version invalidates it (the version is a path component). Also note py-spy dump needs root here, so when you want to prove liveness without sudo, read /proc/<pid>/wchan and the child process list instead.

2026-07-20 — spec-decode downgrades cudagraph_mode with the FlashInfer backend

Enabling --speculative-config on a model served with the FlashInfer attention backend logs:

CUDAGraphMode.FULL_AND_PIECEWISE is not supported with spec-decode for attention
backend FlashInferBackend (support: AttentionCGSupport.UNIFORM_SINGLE_TOKEN_DECODE);
setting cudagraph_mode=PIECEWISE

It is a warning, not an error, and serving proceeds — but it means the speculative config is not being compared against the same graph-capture regime as the no-draft baseline. Worth stating whenever a spec-decode speedup is published: the baseline gets FULL graphs, the draft arm gets PIECEWISE, and the measured speedup is therefore net of that handicap (i.e. conservative).

2026-07-20 — quantizing the experts can optimize the wrong thing on a sparse MoE

On a high-sparsity MoE, "how big is the checkpoint" and "how many bytes does one token read" come apart hard, and only the second sets the decode ceiling. On poolside/Laguna-XS-2.1-NVFP4 (256 routed experts, top-8, 40 layers, moe_intermediate_size 512) the routed experts are 82% of the file but 16% of per-token reads, while attention — left BF16 by the checkpoint's ignore list — is 13% of the file but 82% of per-token reads.

Consequence: all three poolside quants (FP8 / NVFP4 / INT4) benchmark within 2 tok/s of each other (42.1 / 42.9 / 40.7), because they share that ignore list and are bandwidth-bound on the same unquantized attention. A llama.cpp Q4_K_M of the same model quantizes attention too and decodes ~87 tok/s — the 2× is quantization scope, not engine or kernel quality (both engines land at ~50% of their own bandwidth ceiling).

Generalizable check before blaming a kernel for slow MoE decode: python3 bench/ckpt_bytes.py <checkpoint> — sum bytes by category, then weight by top-k/num-experts. If attention dominates the per-token read, no MoE backend flag will help; you need a checkpoint that quantizes attention.

2026-07-22 — hf upload/download xet path hangs on the Sparks; disable it

The hf CLI's default xet transfer path stalls on these boxes — a 0.82 GB hf upload sat for ~30 min with only .gitattributes committed and no error, and a hf download retried endlessly at the same shard. Set HF_HUB_DISABLE_XET=1 and the classic path works (the same upload finished in ~3 min at ~5 MB/s). Applies to both upload and download. Also: hf upload <repo> <local> <path> buffers its progress bar, so an empty log ≠ stuck — check the repo's file list via the API, not the log. And two large transfers to/from the Hub at once starve each other; run them one at a time.

2026-07-23 — the flashinfer NVFP4-MoE JIT burst masquerades as a profile_run forward OOM

Bringing up a fresh NVFP4-MoE port (Solar-Open2-250B, 320 experts, TP2) on vLLM 0.25.1, every serve died at determine_available_memory → profile_run with the worker "killed by the OS (out of memory)" and the head logging RuntimeError: cancelled. It reads exactly like the profiling forward allocating tens of GiB on top of the 71.48 GiB/rank weights — and a day was spent chasing that (sweeping --max-model-len, --max-num-batched-tokens, --max-num-seqs, --kv-cache-memory-bytes — none helped, because none of them was the cause).

It is not the forward. Instrumenting DecoderLayer.forward with torch.cuda.memory_allocated()/max_memory_allocated() (env-gated) showed the 320-expert MoE forward peaks ~0.3 GiB above weights at 2048 tokens and does not accumulate across layers. The tens of GiB is the FLASHINFER_CUTLASS NVFP4-MoE backend JIT-compiling its sm120 cutlass kernel on the FIRST forward — which happens inside profile_run. ninja runs nproc (=20) parallel nvcc, ~3–4 GiB each, on a worker already holding the weights.

The trap: you'll instrument the forward, find it cheap, and be confused — because the memory is spent by the compiler subprocesses, which never show up in torch.cuda.* accounting. The tell is pgrep -c 'cicc|nvcc|ptxas' on the worker node spiking during profile_run, not any CUDA allocation.

Fix (same MAX_JOBS lesson as the compile-parallelism note above, but the failure mode is disguised): MAX_JOBS=4 on the raylet env (Ray workers do the compile, so it must be on the raylet, not the driver). With it, the same boot held ~36 GiB free on the worker and sized the KV pool cleanly. Cached under ~/.cache/flashinfer/ afterwards, so later boots skip it. Also confirm ninja (~/.local/bin) + nvcc (/usr/local/cuda/bin) are on the raylet PATH (login shell has them; a non-login launch fails with FileNotFoundError: 'ninja' deep inside gen_cutlass_fused_moe_sm120_module().build_and_load()).

Sanity: don't over-shrink batch/seq/context chasing this — the profiling forward is cheap; it's the compiler that needs bounding.

2026-07-23 — speculative decode (ngram/draft) does NOT work on hybrid KDA linear-attention yet

Tried n-gram / prompt-lookup spec decode on Solar-Open2 (hybrid KDA linear-attn + MoE) to beat its ~10 tok/s: --speculative-config '{"method":"ngram","num_speculative_tokens":5,...}'. It INITIALIZES fine (engine logs SpeculativeConfig(method='ngram', num_spec_tokens=5), no arch rejection) — then the engine dies fatally on the FIRST generated token:

kimi_gdn_linear_attn.py:358, in _forward
    assert non_spec_state_indices_tensor is not None
AssertionError   (worker ip=192.168.100.2, EngineDeadError, HTTP 500)

The Kimi-GDN (KDA) linear-attention kernel HAS a spec-decode path, but it needs the KDA recurrent state partitioned into speculative vs non-speculative slots (non_spec_state_indices_tensor) — plumbing that the hybrid model's attention-metadata / KV-cache-spec builder must populate. A basic out-of-tree port that reuses KimiGatedDeltaNetAttention does not set this up, so any spec method that verifies K>1 tokens per target forward (ngram, EAGLE, draft-model, MTP) asserts here. This is the same wall a trained draft would hit — it's a state-management gap, not a draft-availability one.

Lesson: spec decode is NOT a free lever on hybrid linear-attention (KDA/GDN/mamba) models the way it is on plain-attention MoEs. Enabling it requires wiring the spec-decode state indices through the linear-attn layer's metadata builder (real vLLM-internals port work). Check for non_spec_state_indices_tensor handling before promising a spec-decode win on any mamba/GDN/KDA arch. (The ngram config itself is harmless to try — it fails fast and cleanly, no box wedge.)

2026-07-24 — KAT-Coder V2.5 NVFP4 (Qwen3.5-VL-MoE): three findings

Serving sakamakismile/KAT-Coder-V2.5-Dev-NVFP4 (a Qwen3.6-35B-A3B fine-tune, Qwen3_5MoeForConditionalGeneration) on one Spark, vLLM 0.24.0:

• Qwen3_5MoeForConditionalGeneration (the Qwen3.5/3.6 VL-MoE class) is natively registered in vLLM 0.24.0 AND 0.25.1 — no port needed even though the checkpoint carries a 27-block vision tower. Text-only serving loads the vision tower to memory (~0.83 GiB BF16) but decode never touches it. The GDN linear-attention path (qwen_gdn_linear_attn.py) auto-selects; MoE backend auto-selects FLASHINFER_CUTLASS, attention FLASH_ATTN.

• An uncapped KV pool on a hybrid model can cgroup-OOM-kill the serve on the first long prefill, and it looks like a crash, not an OOM. At util 0.85 with no --kv-cache-memory-bytes, this model's cheap-KV hybrid gave a 77.95 GiB pool (15.36x at 262K). Weights 20.5 + KV 78 = ~98 GiB resident; the first 8192-token benchmark prefill's activations pushed past MemoryMax=110G and systemd killed it with MemorySwapMax=0 — no CUDA traceback, memory freed cleanly, port then refusing. Easy to misread as an engine crash. Fix: pin KV small (6 GiB → 308,955 tok = 1.18x) before benchmarking. Corollary to the existing "read KV at steady state" rule: on a cheap-KV hybrid, also cap KV before you send a long prompt, or the long-prompt activation spike on top of a huge pool wedges/kills the box.

• Full-NVFP4 checkpoints that quantize attention with per-projection global scales trip a vLLM accuracy warning (compressed_tensors_w4a4_nvfp4.py:102): "the weight global scale is different for parallel layers (e.g. q_proj, k_proj, v_proj) ... likely result in reduced accuracy." vLLM fuses q/k/v into one QKV kernel that wants a single shared scale. It serves and coding output looks fine, but it is a real risk — note it in the recipe and eval-verify before trusting it. Prefer a shared-global-scale NVFP4 build if one exists. (Weight-only NVFP4 quants that leave attention FP8 don't hit this.)

• ngram/prompt-lookup DOES run on the Qwen3.5/3.6 GDN hybrid (no non_spec_state assert, unlike the Solar KDA port) but is sharply workload-dependent. Measured warm at 2048/256 greedy: k=3 gave ~47 tok/s on the open-ended harness (−27% vs 64.1 no-draft) yet ~167 tok/s on a verbatim code-echo (2.6x). So the qwen3_5 GDN path has the speculative-state plumbing the bare KDA port lacked — ngram is a real lever for edit-heavy traffic, but a net loss for open-ended generation. Ship it off; document both.

2026-07-26 — vLLM 0.26.0's memory-profiling assert is unsound on unified memory

Symptom: the engine dies during startup with no traceback at all, right after torch.compile took NN s. Memory looks fine (peaked ~36 of 121 GiB), kern.log shows no OOM kill, no core dump appears, and the parent only prints resource_tracker: There appear to be 1 leaked semaphore objects. Every instinct says segfaulting kernel — an sm_121 marlin/flashinfer/GDN kernel fault.

It is none of those. vLLM 0.26.0's gpu_worker.determine_available_memory() ends its profiling with:

assert self.init_snapshot.free_memory >= free_gpu_memory, (
    "Error in memory profiling. "
    f"Initial free memory {...} GiB, current free memory {...} GiB. "
    "This happens when other processes sharing the same container "
    "release GPU memory while vLLM is profiling during initialization. ...")

i.e. it assumes free GPU memory can only decrease between the initial snapshot and the end of profiling. That holds for discrete VRAM. On GB10 it does not: mem_get_info reports the single system-wide unified pool, so anything the OS does — page-cache reclaim, a service restarting, another shell running free/du — makes free memory go up and trips the assert. Ours failed with Initial free memory 89.17 GiB, current free memory 91.27 GiB — a 2 GiB gain.

Two things follow:

• Fix: always pass --kv-cache-memory-bytes on 0.26.0. It is an early return before the profiling block (gpu_worker.py:462), logging "reserved N GiB memory for KV Cache as specified by kv_cache_memory_bytes config and skipped memory profiling". It still runs profile_run() so kernels/graphs still compile — you lose nothing but the fragile auto-sizing. This is the same flag the recipe method already prefers for determinism on unified memory, so there is no downside.
• Don't poll the box while a Spark serve is profiling. Repeatedly SSHing in to run free -g / du -sh / ps during startup perturbs page cache and can cause this. Read memory at steady state after the API is up, which is the correct time anyway.

Diagnostic tell, in order: no traceback + no kern.log oom-kill + no fresh /var/crash file + death immediately after torch.compile ⇒ suspect this assert, not a kernel. The traceback only reaches the log on some runs (the API server's 35 s startup timeout can win the race and truncate it), so absence of the AssertionError text is not evidence against it — grep the log for Error in memory profiling explicitly.

2026-07-26 — two spec-decode traps: concurrency kills the engine, and throughput variance grows with k

Both found while measuring nvidia/Qwen3.6-27B-NVFP4 (native MTP head) on vLLM 0.26.0.

Speculative decoding does not survive concurrent requests. Four simultaneous requests at k=5 killed the engine outright:

torch.AcceleratorError: CUDA error: an illegal memory access was encountered
  flashinfer.py:1232 build → backend.py:510 seq_lens_cpu
      self._seq_lens_cpu = self.seq_lens.to("cpu")
→ EngineDeadError; every subsequent request 500s; server shuts down

CUDA errors are reported asynchronously, so the .to("cpu") is merely where the previous fault surfaces — the real one is an earlier kernel, and the dumped config names the eagle_head (draft) cache dir. It is the draft, not concurrency per se: with identical server flags minus --speculative-config, four concurrent requests all completed (HTTP 200, clean log). Concurrency 1 was flawless across a dozen serves and hours of sweeps. Practical consequences: a spec-decode config is single-user only — do not put one behind anything that fans out — and the harness standard profile cannot be completed on such a config, because its fourth scenario is concurrency 4. That is a second, independent reason the spec-decoding profile deliberately has no concurrency row.

Spec-decode throughput variance grows with k, and one benchmark pass is not enough to rank two k values. Decode under speculation depends on how many drafted tokens survive verification, which depends on the text being generated — and the harness gives every request a unique cache-busting prefix, so each request generates different text. Measured per-request stdev at 2048/256 on this model:

k	stdev (tok/s)	per-request range
0 (no draft)	0.01	~0%
3	0.93	11.3%
4	2.08	25.3%
5	2.15	20.2%
6	2.71	24.4%

The no-draft baseline is essentially noiseless because its work per token is fixed; everything above it inherits content-dependent variance that compounds with k. Four separate passes of the identical k=5 config on the identical node gave 31.18, 33.87, 33.89 and 35.69 tok/s. The existing "run the harness twice, keep the second pass" rule guards against cold kernels and does nothing about this — worse, keeping pass 2 selects a sample at random from a wide distribution, which handed us the highest of four and a "peak at k=5" conclusion the data did not support. Fix: one scenario with repeats: 10, pool the per-request samples, and compare medians with confidence intervals. Pooled at n=13/setting the curve was 12.14 ±0.01, 20.00 ±0.14, 26.48 ±0.30, 29.61 ±0.50, 33.10 ±1.13, 31.75 ±1.17, 35.58 ±1.48 for k=0…6 — clean and monotonic to k=3, then a plateau in which the ordering is not resolvable. A non-monotonic pooled result (here k=5 below both k=4 and k=6) is the tell that you are reading spread, not structure.

2026-07-26 — before porting an env flag from someone else's stack, read the default AND the code path

Surveying ciprianveg/gb10-glm-5.2 (GLM-5.2 on 8× GB10, the local-inference-lab "Gilded Gnosis v18" vLLM fork + B12X) for levers to port into our 2-Spark GLM-5.2 recipe, three of the flags that looked most promising turned out to be dead code on our stack — findable by reading the installed venv, before spending a single ~40 min TP2 boot. The general lesson: a foreign recipe's env block is an artifact of its own fork, not a list of knobs. Grep envs.py for the default and then grep for the consumer, in that order.

The three, on vLLM 0.24.0 (~/venvs/vllm-moet):

• VLLM_USE_RAY_V2_EXECUTOR_BACKEND already defaults to 1. Note the trap: envs.py:64 annotates it bool = False, but the actual resolver at envs.py:859 is bool(int(os.getenv(..., "1"))). The dataclass annotation is not the default — the lambda is. Setting this flag changes nothing; we were already on RayExecutorV2.
• VLLM_USE_RAY_COMPILED_DAG_OVERLAP_COMM is unreachable from the V2 executor. It is consumed only at v1/executor/ray_executor.py:619 (experimental_compile(_overlap_gpu_communication=…)), i.e. inside the V1 RayDistributedExecutor's compiled-DAG path. ray_executor_v2.py contains zero references to the compiled DAG. So on any Ray deployment that takes the (default) V2 executor, this flag is inert — and a recipe setting both it and VLLM_USE_RAY_V2_EXECUTOR_BACKEND=1 is self-cancelling.
• VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=0 is inert whenever --kv-cache-memory-bytes is set. v1/worker/gpu_worker.py:412 early-returns the pinned byte count (after still running profile_run() so kernels/graphs compile) long before cudagraph_memory_estimate_applied is computed at line 467. Since pinning KV is close to mandatory on GB10's unified memory anyway, the two are mutually exclusive: if you pin KV you do not get to tune the profiler, and you do not need to.

Also dropped by inspection, from the same recipe's NCCL block: NCCL_NVLS_ENABLE=0 (no NVLink between Sparks), NCCL_CROSS_NIC=1 (single NIC), NCCL_NET=IB (equivalent to the NCCL_IB_DISABLE=0 + NCCL_IB_HCA we already set), and NCCL_BUFFSIZE=16777216 — that last one is plausibly net-negative here, since 4× the default registered-buffer size comes out of the same pool that has to keep 79 GB of expert planes page-cache resident.

The two survivors were then measured, and both are null. Three paired same-session arms on the GLM-5.2 TP2 ship config, full 52-prompt sparkbench each, with the lever on the raylet (Ray actors take the raylet env, not the driver's):

arm	mean tok/s	paired Δ vs baseline	t (n=52)
baseline	25.576	—	—
PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True	25.489	−0.34%	−0.94
NCCL_MIN/MAX_NCHANNELS=4 + NCCL_CUMEM_ENABLE=0	25.558	−0.07%	−0.61

The null is structural rather than an averaging artefact: flat across all 11 prompt categories, and speculative acceptance is untouched (the NCCL arm reproduced baseline's pos-0 / pos-1 / tok-per-verify to three decimals — 0.739 / 0.521 / 2.628). Worth knowing for its own sake that on GB10 unified memory, expandable_segments — which is a real lever on discrete-VRAM fragmentation — buys nothing measurable at concurrency 1.

Method warning found the hard way in the same session: the monitor for this sweep initially polled the box over SSH every 10 s. On a config whose speed depends entirely on keeping 79 GB of planes page-cache resident, that is the churn the recipe skill already warns about — and it would have applied unequally across arms, which is worse than applying uniformly. Caught before any prompt was measured; re-polled at 180 s. On these boxes the poll interval is a measurement parameter, not a convenience.

2026-07-26 — NVFP4 MLA KV cache is a B12X read path, not a flashinfer one

Long-standing note in models/glm-5.2.md had the NVFP4 KV cache "blocked on the flashinfer AOT sparse_mla_sm120.so work". That is the wrong tree. From local-inference-lab/vllm PR #82 (merged 2026-07-14), the feature is a distinct cache dtype nvfp4_ds_mla — 432 B/token/layer against fp8_ds_mla's 656 (256 B FP4 NoPE

• 32 B E4M3 group scales + 16 B pad + 128 B BF16 RoPE), measured −32% bytes and +47.8% KV pool — and it is B12X_MLA_SPARSE only. FLASHMLA_SPARSE deliberately continues to canonicalize to fp8_ds_mla. The decode/extend read path is a ScaleFormat.NVFP4_E4M3 branch in b12x's sparse-MLA kernels; deleting a flashinfer AOT .so was never going to produce it.

Two details worth keeping even if you don't adopt B12X:

• It is quality-neutral, and unusually well evidenced. Same checkpoint both sides: GPQA-Diamond 174/198 (nvfp4) vs 175/198 (fp8); MMLU-Pro 54/60 vs 52/60; NIAH 30/30 including depths the fp8 pool cannot hold at all; decode speed-neutral at matched context (it is expert-bandwidth-bound, not KV-bound). So "4-bit KV costs quality" is answered, and answered no — for MLA latents with per-group scales.
• The writer can ship as an overlay, no main-extension rebuild. _custom_ops will lazy-load a companion vllm/_nvfp4_mla_cache_C.so if the built extension lacks the op. That matters for wheel-installed stacks like ours, which have no source build.
• Follow-on PR #95 stacks two more: a calibrated per-layer outer scale (teacher-forced KLD 0.184 → 0.152, free) and an fp8-RoPE KV cell that takes the record 432 → 368 B (+15.8% pool).

Cost to actually use it on our stack is a sparse-MLA backend migration to B12X, not a cherry-pick — and our GLM-5.2 recipe is plane-residency-bound rather than KV-bound, so the win is roughly 3 GiB handed back to page cache. Recorded, not chased.

2026-07-26 — vLLM 0.26.0 vs DeepSeek-V4-Flash-DSpark on GB10: closer, still blocked

Re-tested the DSpark checkpoint against stock vLLM 0.26.0 (~/venvs/vllm-026, flashinfer 0.6.14) TP2 over Ray, to see whether the pinned ghcr.io/anemll image is still load-bearing. It is — but the reason has moved, and the new one is narrow.

All three vLLM 0.24.0 walls are gone. DeepGEMM now lays out the ue8m0 block scales (DeepGEMM E8M0 enabled on current platform); the in-checkpoint drafter loads (DSpark draft model loaded: 96 params); and the MoE backend auto-selects to DEEPGEMM_MXFP4 rather than dying in CUTLASS. Weights come to 79.22 GiB per rank. Corollary: the --moe-backend flashinfer_b12x pin in our recipe is image-specific and must not be ported — stock picks something else and works.

The blocker turned out to be OUR OWN --block-size, not a vLLM defect. First decode dies with

tvm.error.InternalError: Check failed: num_tokens > 64 (5 vs. 64)
  in _trtllm_batch_decode_sparse_mla_dsv4_sm120

which reads like vLLM routing decode into the prefill orchestrator. It isn't. flashinfer's mla/_sparse_mla_sm120.py auto-dispatches decode vs prefill, and _decode_dsv4_dispatchable() requires page_block_size == _DECODE_DSV4_PAGE_BLOCK_SIZE, which is 64 (plus d_qk == 512, num_tokens <= 64, and (num_heads, topk) in a fixed table). Our recipe serves --block-size 256, so the decode kernel is not dispatchable, the call falls through to the prefill orchestrator, and that orchestrator's first act is to assert num_tokens > 64 — which single-user decode never satisfies. Fix: --block-size 64 on any stock-vLLM serve of this checkpoint.

Generalizable: an assert about a quantity you did not set (num_tokens) can be a symptom of a dispatch predicate you did set (page_block_size). When a kernel-side check fails on a value you have no direct control over, go read the dispatch predicate that chose that kernel before blaming the engine — the tunable is usually one of its other arguments.

But --block-size 64 then hits a SECOND wall, and the two are mutually exclusive. With block 64 the engine never reaches a forward pass; it dies earlier in KV-cache-group setup:

vllm/v1/core/kv_cache_utils.py:1659, in _get_kv_cache_groups_uniform_groups
    assert max(sm_page_sizes) <= max(all_page_sizes)
AssertionError

DSv4 carries heterogeneous KV — a sliding-window cache, the compressed MLA cache, and an FP8 indexer cache for the Lightning Indexer — and at block 64 their page sizes stop satisfying that uniform-group assertion. So: the sm120 decode kernel dispatches only at page_block_size 64, and 0.26.0's uniform KV grouping rejects 64 for this model. That is a genuine bind, not a flag we haven't found. Untried lead: 0.26.0 added blocks_per_chunk for heterogeneous KV groups (#48878), which is the right area.

Net verdict for now: the pinned ghcr.io/anemll image stays load-bearing, but for a sharper reason than "stock vLLM can't load it" — stock 0.26.0 loads it fine and dies on a block-size bind between two subsystems.

nvfp4_ds_mla is still image-only. 0.26.0's CacheDType has fp8_ds_mla and nvfp4, no nvfp4_ds_mla. But fp8_ds_mla still reaches 1M context: measured ~10.3 KB/token vs nvfp4's ~7.2 (a 1.43× penalty, not 2× — the BF16 RoPE half doesn't shrink). An 18 GiB pin gave 1,873,590 tokens = 1.79× at the full 1,048,576 context. So the third-party image is not what buys 1M; it buys ~6 GiB of headroom.

The k=3 we ship is a correctness bug. The checkpoint declares dspark_block_size: 5 and 0.26.0 hard-rejects anything smaller: "DSpark requires num_speculative_tokens >= dspark_block_size (5); got 3. Smaller values produce incorrect output." The 0.25.2 image has no such guard. Note the memory coupling: capture size is max_num_seqs × (k+1), so 3→5 pushes it 24→36 (truncated to 32), and at an 18 GiB KV pin that combination tripped oom-guard.sh (MemAvailable=5193 MiB) plus NVRM NV_ERR_NO_MEMORY. 12 GiB KV (1,249,060 tokens, 1.19×) fits.

Don't purge page cache before serving this checkpoint. Its loader makes multiple full passes over the 155 GiB tree, so warm boot reached KV sizing in ~4.5 min while the same config after purge-cache.py took 45+ min. Purging only exists to satisfy the startup free-memory check, and --kv-cache-memory-bytes skips that check outright (skipped memory profiling). Pin KV → don't purge.

Ray forwards worker logs out of order. A shard progress bar that appears to jump backwards (88% → 19%) is usually stale buffered output, not a restart. We chased that for a while; the engine had in fact already failed minutes earlier. Check timestamps before diagnosing from a Ray-forwarded log.

GID drift, third sighting. RoCEv2 IPv4 GID index resolved to .1=3 / .2=5 today, where it was 3/6 previously. Keep resolving it at start time; never hardcode.

2026-07-26 (later) — DSpark k=5 re-measurement: correctness fix, mixed throughput effect

Re-ran DeepSeek-V4-Flash-DSpark on the pinned anemll image at k=5 (the checkpoint's dspark_block_size), replacing the k=3 config that vLLM 0.26.0 classifies as producing incorrect output. Warm, harness run twice, second pass kept.

workload	k=3 (old)	k=5 (measured)	acceptance k=3 → k=5	tok/step k=3 → k=5
prose	44.2 @8K	43.1 @8K (42.0 @2048)	0.526 → 0.338	2.58 → 2.69
code	62.5 @8K	73.2 @8K (76.0 @2048)	0.847 → 0.825	3.54 → 5.12

The lesson: acceptance RATE and tokens-per-STEP move in opposite directions as k rises, and only the second one drives throughput. Raising k from 3 to 5 dropped prose acceptance by a third (0.526 → 0.338) — which looks like a regression and isn't. More positions means each individual position is likelier to miss, but the expected accepted run is longer, and decode tok/s = step-rate × tokens-per-step. Compared at a MATCHED 8192-token prompt, code gained +17.1% (62.5 → 73.2) while prose was flat-to-slightly-down (44.2 → 43.1, −2.5%). Judging a k change on acceptance rate alone would have reverted a win.

Paying for k with utilization, not with context. k=5 forces cudagraph capture to max_num_seqs × (k+1) = 36 (was 24). We bought that by dropping GPU_MEMORY_UTILIZATION 0.85 → 0.78, taking KV 17.21 → 8.54 GiB/rank and the pool 2,555,830 → 1,268,110 tokens. That is free here: the pool is still 1.21× at the full 1,048,576 context, and 1M is max_position_embeddings, so surplus pool can never become more context. Steady state: head 108 GiB, worker 105 GiB; weights measured at 79.17 GiB/rank.

Two operational traps re-confirmed this session. (1) The RoCEv2 GID index drifted again — worker moved 6 → 5 while the head stayed 3 — and a stale literal in .env.dspark failed ~2 min into startup as RuntimeError: NCCL error inside init_model_parallel_group. Resolve it from sysfs at launch; never store it. (2) docker compose logs replayed the previous failed container's traceback into the new run's log, and a monitor watching for "Traceback" fired on history. Always bound by --since and check timestamps.

The two-pass load is real and visible. Loading weights took 160.21 seconds for the main model, then Using Eagle3 auxiliary layers from config: (41, 42, 43) and a second full 0→48 shard pass for the drafter. A progress bar that appears to restart is that second pass.

2026-07-26 — check the vLLM arch registry before scouting a model, not after

Deciding whether a Hugging Face checkpoint is servable on the Sparks does not require downloading it. Ask the installed engine directly:

ssh Spark-1 '~/venvs/vllm-026/bin/python -c "
from vllm.model_executor.models.registry import ModelRegistry as R
archs = R.get_supported_archs()
print(\"MiMoV2ForCausalLM\" in archs)"'

Two things this turned up that a model card would not have:

• The registry key is the architectures[0] string, not model_type, and the two disagree often enough to matter. amd/Instella-MoE-16B-A3B-Think declares architectures: ["InstellaMoEForCausalLM"] (not registered) but model_type: "deepseek_v3" (whose DeepseekV3ForCausalLM is registered) — so the right first move is to try it on the DeepSeek path, not to write it off. Conversely a registered model_type proves nothing if the checkpoint's architecture class is bespoke.

• A config advertising MTP is not a checkpoint containing MTP. Qwen/Qwen-AgentWorld-35B-A3B sets text_config.mtp_num_hidden_layers: 1, and its model.safetensors.index.json contains zero mtp.* tensors. Grep the index, not the config:

  curl -s https://huggingface.co/$REPO/raw/main/model.safetensors.index.json \
    | python3 -c 'import json,sys; w=json.load(sys.stdin)["weight_map"]; print(sum(1 for k in w if "mtp" in k))'
  

  Same command is the fastest way to read a repo's real dtype: FP8 checkpoints carry a weight_scale_inv per quantized tensor (MiMo-V2.5 has 36,159 of them), so summing safetensors bytes and dividing by 2 silently halves the parameter count on any natively-FP8 release.

State of the registry on vLLM 0.26.0 (Spark-1, ~/venvs/vllm-026) as of today: Qwen3_5MoeForConditionalGeneration, MiMoV2ForCausalLM, GlmMoeDsaForCausalLM, DeepseekV3ForCausalLM, Lfm2MoeForCausalLM are supported; MotifForCausalLM, LongcatCausalLM, InstellaMoEForCausalLM and Qwen3_5MoeForCausalLM (the plain causal-LM spelling — Qwen3.5 MoE ships only under the ConditionalGeneration class) are not.

2026-07-27 — a registered arch is not a loadable arch: the unconditional vision tower

Qwen/Qwen-AgentWorld-35B-A3B is a language-only release that still carries a vision_config and declares "language_model_only": true in its config.json. Its architecture, Qwen3_5MoeForConditionalGeneration, is registered in both vLLM 0.24.0 and 0.26.0 — and neither can load the checkpoint:

ValueError: Following weights were not initialized from checkpoint:
{'visual.blocks.25.norm2.weight', 'visual.patch_embed.proj.weight', ...}   # 113 params

Qwen3_5MoeForConditionalGeneration.__init__ in model_executor/models/qwen3_5.py constructs a 27-block Qwen3_VisionTransformer unconditionally, and the weight index holds zero visual.* tensors to fill it.

Three things worth carrying:

• --language-model-only does not do what its name suggests here. It sets MultiModalConfig.language_model_only, whose docstring is explicit that it "disables all multimodal inputs by setting all modality limits to 0". It is never consulted at construction time in qwen3_5.py, so the tower is still built and still fails to load.
• The checkpoint's own language_model_only key is not vLLM's. MultiModalConfig defaults it to False and only the CLI flag sets it — a config key with the same name sitting in the HF config does nothing.
• The fix upstream already wrote, one file over. qwen3_next.py:303 guards with text_only = mm_config is None or mm_config.language_model_only. Porting the same guard into qwen3_5.py (skip the tower, self.visual = None) plus passing the flag is enough; both halves are required, neither works alone. The guard is inert without the flag, so a genuinely multimodal Qwen3.5 checkpoint served from the same venv is unaffected.

Generalizes past this model: get_supported_archs() containing your architecture only proves the class exists, not that your checkpoint's tensor set matches what that class builds. A "ForConditionalGeneration" arch serving a language-only release is a standing trap — check the weight index for tower tensors before you spend the download.

Confirmed from the other side, same day. deepreinforce-ai/Ornith-1.0-35B is the same Qwen3_5MoeForConditionalGeneration arch at the same 35B-A3B shape, and it ships all 333 visual.* tensors for the 27-block tower — so it loads on stock vLLM 0.24.0 with no patch, and must not be given --language-model-only (that would zero the modality limits and throw away a capability the weights paid for). Two checkpoints, one arch, opposite handling. So the pre-flight is not "does this arch need the patch?" but a per-checkpoint count against the index, and it costs one HTTP request:

curl -sL https://huggingface.co/<org>/<model>/resolve/main/model.safetensors.index.json |
  python3 -c "import json,sys; wm=json.load(sys.stdin)['weight_map']; \
    print(sum('visual' in k for k in wm), 'visual tensors of', len(wm))"
# Ornith-1.0-35B  -> 333 visual tensors of 31666   => serve stock, no flag
# AgentWorld-35B  -> 0 visual tensors of 693       => needs the guard + --language-model-only

Run it before the download, not after the failed boot. The same request also settles whether a declared mtp_num_hidden_layers is backed by real weights (grep the map for mtp — both of these models declare 1 and ship none).

2026-07-27 — a 4 GiB KV pin does not fit a 262K window on Qwen3.5-MoE — and weight precision is NOT why (corrected same day)

Same architecture (Qwen3.5-MoE hybrid, full_attention_interval: 4, 10 of 40 layers with KV, 2 KV heads at head_dim 256). A --kv-cache-memory-bytes 4294967296 pin inherited from another recipe dies at startup:

To serve at least one request with the model's max seq len (262144), 5.08 GiB KV cache is
needed, which is larger than the available KV cache memory (4.0 GiB). Based on the
available memory, the estimated maximum model length is 205920.

The first version of this note blamed weight precision — "the KV dtype follows the serve, so halving the weight bits roughly halves the KV bill" — and cited a 4-bit sibling getting ~10.3 KiB/token against BF16's ~20.3. That explanation is wrong, and two measurements on this arch settle it:

• deepreinforce-ai/Ornith-1.0-35B in BF16 and its vendor FP8 export (-FP8), same node, same flags, same 8 GiB pin: both report exactly GPU KV cache size: 412,980 tokens and 1.58x — 20.8 KiB/token, identical. FP8 gave back 30.66 GiB of weights (65.53 → 34.87) and not one byte of KV.
• The published qwen3-6-35b-a3b-nvfp4 recipe on the same arch gets 308,955 tokens from a 6 GiB pin — 20.4 KiB/token — and its own notes record the same "needs 5.08 GiB" refusal at 4 GiB. So the 4-bit sibling never had a halved KV bill either.

The correct rule: per-token KV is set by the architecture and the KV dtype, not by the weight bits. A weights-only quant (kv_cache_scheme: null, which is what these vendor FP8/NVFP4 exports ship) leaves the cache untouched, so the pin transfers across precisions of one architecture unchanged. Only --kv-cache-dtype moves it. The practical warning survives intact — do not inherit a pin from an unrelated recipe, ~5.1 GiB is the floor for a full 262,144-token request here, and 8 GiB is the smallest round number that clears it with margin — but do not expect a smaller checkpoint to want a smaller pin. Corollary for the memory illustration: quantizing weights buys you node headroom, not context.

2026-07-27 — the FP8 MoE backend pick is per-precision, not per-architecture

Same model, same engine (vLLM 0.24.0), same node, two checkpoints of deepreinforce-ai/Ornith-1.0-35B:

• BF16 → Using 'FlashInfer CUTLASS' Unquantized MoE backend
• vendor FP8 → Using TRITON Fp8 MoE backend out of potential backends: ['AITER', 'FLASHINFER_TRTLLM', 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'TRITON', 'MARLIN', 'BATCHED_VLLM_CUTLASS', 'BATCHED_TRITON', 'XPU', 'CPU']

The FlashInfer FP8 entries are offered on sm_121 and not selected. Both serve correctly and both are the right choice to leave alone — the point is that "this arch picks backend X on GB10" is not a portable fact. The scheme in the checkpoint's quantization_config selects the backend menu, and the menu differs per scheme and per vLLM release (see the 0.26.0 NVFP4 note above). Read the log line every time; never carry --moe-backend across a precision change.

2026-07-27 — DeepSeek V4 on stock vLLM 0.26.0 / GB10 is blocked by two hard-coded block sizes, not by tuning

Four TP2 boots of deepseek-ai/DeepSeek-V4-Flash-DSpark on stock vLLM 0.26.0 (~/venvs/vllm-026, Ray over the RoCE link) reduce the "0.26.0 nearly works" story to a single irreconcilable pair of constants. Sweeping --block-size is the whole experiment, because on this checkpoint every remaining wall is a function of it.

--block-size 64 — impossible arithmetic. The previously-reported wall here was an assertion in _get_kv_cache_groups_uniform_groups. That assertion is not the bind: adding --disable-hybrid-kv-cache-manager clears it and the engine reaches KV sizing. It then dies in UniformTypeKVCacheSpecs.max_memory_usage_bytes:

cdiv(spec.max_memory_usage_bytes(vllm_config), spec.page_size_bytes)
ZeroDivisionError: integer division or modulo by zero

MLAAttentionSpec.storage_block_size is block_size // compress_ratio (v1/kv_cache_interface.py:395), and this config's compress_ratios list contains 128. 64 // 128 == 0, so those layers get a zero-byte page. Any --block-size below 128 is arithmetically dead on this checkpoint, whatever the kernels want.

--block-size 128 — no backend accepts it. Gets past KV sizing, dies in select_common_block_size (v1/worker/utils.py:348) with ValueError: No common block size for 128. Reading the installed classes directly explains why:

DeepseekV4IndexerBackend.get_supported_kernel_block_sizes()  -> [256]
DeepseekV32IndexerBackend.get_supported_kernel_block_sizes() -> [64]

The V4 Lightning-Indexer backend is pinned to exactly 256 — a bare int, not a MultipleOf, so 128 has no divisor to fall back to. (V3.2 is pinned to 64; do not carry a V3.2 intuition across.)

--block-size 256 — the only value vLLM accepts, and the one the decode kernel refuses. KV sizes cleanly at 256 (measured: GPU KV cache size: 58,880 tokens, 7.19x concurrency at 8,192/request, 18 GiB pinned with hybrid disabled). But flashinfer's mla/_sparse_mla_sm120.py hard-codes _DECODE_DSV4_PAGE_BLOCK_SIZE = 64, and _decode_dsv4_dispatchable() requires page_block_size == 64. At 256 the decode kernel is not dispatchable, the call falls through to the prefill orchestrator, and that asserts num_tokens > 64 — which single-user decode never satisfies.

And there is no second backend. On GB10 (sm_121 → device_capability.major == 12) vLLM's own priority list in platforms/cuda.py is [TRITON_MLA, FLASHINFER_MLA_SPARSE_SM120]. TRITON_MLA is dense MLA and gets rejected for this sparse model, so FLASHINFER_MLA_SPARSE_SM120 is the only sparse-MLA path on this hardware. Nothing to fall back to, nothing to select with VLLM_ATTENTION_BACKEND.

So: 256 required by the indexer, 64 required by the decode kernel, both hard-coded integers in released code. No flag reconciles them. This is a released-artifact defect to fix upstream (or vendor around), not a serving config to keep tuning — which is worth knowing before someone spends another run sweeping flags.

Side finding: --disable-hybrid-kv-cache-manager is not a KV workaround here either. It unifies the sliding-window and compressed-state layers up to full-length KV, and the bill is brutal — measured, it wants 40.07 GiB of KV for a 131,072-token max_model_len (~320 KiB/token) against the pinned image's 8.54 GiB for 1,268,110 tokens (~7.06 KiB/token). That is ~45x per token; the recipe's 1M context would need ~320 GiB of KV. Use the flag to diagnose group-construction failures, never to serve.

Operational note: a 18 GiB --kv-cache-memory-bytes pin on top of 79.22 GiB of weights per rank drives MemAvailable under 8 GiB and trips Joe's oom-guard.sh, which kills the vLLM and Ray processes with no traceback in the serve log — it looks exactly like a silent engine death. Check journalctl --user | grep oom-guard before debugging vLLM, and restart the Ray cluster before the next attempt. A 10 GiB pin stays clear.

2026-07-27 — vLLM 0.26.0 removes FLASHINFER_CUTLASS for NVFP4 MoE on sm_121, and the fallback is slower

Direct amendment to the 2026-07-19 note above ("FlashInfer CUTLASS runs NVFP4 MoE on sm_121"). That finding was correct for vLLM 0.24.0 and is no longer true on 0.26.0. Measured today on one Spark, same node, same checkpoint (unsloth/Qwen3.6-35B-A3B-NVFP4-Fast), same flags, both warmed identically.

Ask for it explicitly on 0.26.0 and it refuses outright:

--moe-backend flashinfer_cutlass
ValueError: NvFp4 MoE backend 'FLASHINFER_CUTLASS' does not support the deployment
configuration since kernel does not support current device cuda.

Left to auto-select, 0.26.0 falls back one slot down the list, silently:

# vLLM 0.24.0
Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend out of potential backends:
  [..., 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'EMULATION']
# vLLM 0.26.0 (note the new HUMMING entry, and the different choice)
Using 'VLLM_CUTLASS' NvFp4 MoE backend out of potential backends:
  [..., 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'HUMMING', 'EMULATION']

Nothing breaks — it just gets slower, quietly. Everything structural is identical across the two versions: model residency 22.15 GiB, KV pool 343,699 tokens at 1.31x concurrency for a 262,144-token window, MTP acceptance unchanged (mean acceptance length 3.33–3.46 on 0.26.0 vs 3.37 on 0.24.0). Only the kernel throughput moves, and prefill moves more than decode:

prompt	decode 0.24.0	decode 0.26.0	prefill 0.24.0	prefill 0.26.0
512	101.3 tok/s	100.4 tok/s	3,057 tok/s	2,668 tok/s
2048	109.1 tok/s	102.4 tok/s	5,917 tok/s	4,732 tok/s
8192	105.3 tok/s	99.2 tok/s	6,521 tok/s	5,211 tok/s

So ~1–6% off decode and ~13–20% off prefill, for free, with no error and no log warning beyond the one backend line.

Three things to carry forward:

• A newer vLLM is not automatically faster on GB10. sm_121 is a minority target; backends get dropped from it between releases. Measure before upgrading a recipe's engine pin, and treat a version pin as a result worth re-testing rather than inertia to clean up.
• The one log line that matters is the NvFp4 MoE backend line. Grep it on every start. It is the only place the fallback announces itself, and reading "VLLM_CUTLASS" where a recipe says to expect "FLASHINFER_CUTLASS" is the whole diagnosis.
• The 2026-07-19 rule still holds in its useful form: don't hardcode --moe-backend marlin, and don't hardcode flashinfer_cutlass either — on 0.26.0 that one is now a hard startup failure rather than a suboptimal choice. Let auto-selection run, then check what it picked and that it isn't EMULATION.

2026-07-27 — an EAGLE3 draft crosses target precision; a DFlash draft does not

Serving gdubicki/Qwen3-Coder-Next-NVFP4-GB10 (NVFP4, qwen3_next, one Spark) on vLLM 0.24.0, the only published speculator for the model is togethercomputer/Aurora-Spec-Qwen3-Coder-Next-FP8 — trained against the FP8 build. Our standing rule from the Laguna/DFlash work is that a draft only works with the target precision it was made for (a mismatched DFlash draft gave 0% acceptance). That rule does not generalize to EAGLE3.

Measured, NVFP4 body + FP8-target EAGLE3 draft, 2048-token prompt, warm, Spark-1:

k	decode tok/s	acceptance	tokens/step	position-0 rate
0 (no draft)	60.80	—	1.000	—
1	72.00	65.7%	1.657	0.657
2	67.60	50.7%	2.014	0.634
3	71.61	42.5%	2.274	0.645
4	60.11	32.4%	2.296	0.619
5	59.58	26.6%	2.328	0.619

(k=4 and k=5 measured on Spark-2; the shared k=3 point read 71.61 on Spark-1 and 70.98 on Spark-2, 0.9% apart, so the halves are comparable. k=3 acceptance matched to the third decimal across nodes.)

Position-0 acceptance is flat at 0.62–0.66 across every k — the draft is not degraded by the precision mismatch at all. The mechanism: DFlash ships no lm_head and reads the target's hidden states directly, so it is calibrated to them; EAGLE3 carries its own lm_head over a reduced draft vocabulary (draft_vocab_size: 32000 + d2t mapping) and predicts in that space. Check architectures and draft_vocab_size in the draft's config.json before writing a cross-precision pairing off.

Also worth noting: the -FP8 suffix names the target the draft was trained against, not the draft's own dtype. Aurora's weights are BF16 (991 MB, 0.52 GiB resident).

Two secondary confirmations from the same run:

• Qwen3NextForCausalLM + compressed-tensors nvfp4-pack-quantized serves unpatched on vLLM 0.24.0 / GB10, and auto-selects FLASHINFER_CUTLASS out of ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED', 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'EMULATION']. No --moe-backend override needed. Attention backend FLASH_ATTN.
• The variance-grows-with-k rule reproduces cleanly on a new model. Per-request stdev at 2048/256/c1: 0.04 tok/s with no draft, then 1.40 / 1.59 / 4.24 / 7.78 / 8.57 for k=1…5 — a 49.5–71.0 tok/s per-request range at k=5. The no-draft baseline is essentially noiseless, which is why it is the cheap, trustworthy half of any ×-speedup. The k=2 dip below its neighbours in the table above is spread, not structure.

2026-07-27 — W4A16 on an EAGLE3 draft: two engine behaviours worth knowing

From the sparkulator run against Qwen3-Coder-Next NVFP4 on vLLM 0.24.0 / GB10. The Sparkulator itself was a wash and was not published — see models/qwen3-coder-next.md. Two findings are about the engine rather than the model, so they live here.

1. A compressed-tensors W4A16 EAGLE3 draft needs no source patch. The DFlash proposer required the get_cache_scale guard and a BF16 qkv_proj (see the sparkulator skill); the EAGLE3 path has neither the FP8-only get_cache_scale call nor a direct qkv_proj.weight read for a context-KV buffer, so Aurora-Spec-Qwen3-Coder-Next quantized to pack-quantized loaded first try, q/k/v/o included. Do not assume the DFlash patch list applies to every draft arch.

2. vLLM does not expand ~ inside the --speculative-config JSON. It hands the literal string to the Hub client, so the failure surfaces as

ERROR repo_utils.py:68] Error retrieving file list: Repo id must be in the form
  'repo_name' or 'namespace/repo_name': '~/models/hf/Aurora-Spec-...'
pydantic_core._pydantic_core.ValidationError: 1 validation error for SpeculativeConfig

which reads as "your draft repo is invalid" when it is really a path bug. The model path outside the JSON expands fine, which is what makes it easy to miss. Use '$HOME' shell concatenation (as data/recipes/qwen3-coder-next-nvfp4.json already does) or an absolute path. Six bench/configs/qwen3-coder-next-nvfp4--vllm--eagle3-*.json carried the broken ~ form and were corrected in this run.

3. /metrics spec-decode counters are cumulative from process start, so warmup poisons them. The four warmup requests accepted at 5–7% against ~66% for the benchmark itself, dragging the raw cumulative ratio down about 3 points. Snapshot vllm:spec_decode_num_{accepted,draft}_tokens_total after warmup and again after the run and difference them. Cross-checked: the differenced k=2 baseline read 50.96% here against 50.7% measured on 2026-07-27 in the recipe run, while the undifferenced number would have read 41.7%.

2026-07-27 — the NVFP4 MoE backend is chosen by the checkpoint's scheme, not by sm_121

Two earlier notes on this file frame the NVFP4-MoE backend question as a property of the GPU or the engine version: 2026-07-19 said "FlashInfer CUTLASS runs NVFP4 MoE on sm_121, marlin is not the only option", and the 2026-07-27 recipe-refresh entry found 0.26.0 drops FLASHINFER_CUTLASS on sm_121 where 0.24.0 keeps it. Both are true and both are incomplete. The dominant variable is the checkpoint's quantization scheme.

Serving nvidia/Qwen3.6-35B-A3B-NVFP4 on vLLM 0.24.0 — the same node, same venv, same day that unsloth/Qwen3.6-35B-A3B-NVFP4-Fast auto-selects FLASHINFER_CUTLASS:

Using 'MARLIN' NvFp4 MoE backend out of potential backends:
  ['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED',
   'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'EMULATION']
Using FLASH_ATTN attention backend out of potential backends: [...]

Forcing it names the reason precisely:

ValueError: NvFp4 MoE backend 'FLASHINFER_CUTLASS' does not support the deployment
configuration since kernel does not support quantization scheme
QuantKey(u8,scale(f8e4m3fn,static,GroupShape(row=1, col=16)),scale2(f32,static,per_tensor),
symmetric)xNone.

The trailing xNone is the missing activation quantization. NVIDIA's artifact is NVFP4A16: 4-bit weights, BF16 activations, no input_activations block in its compressed-tensors config. The FP4 tensor-core MoE kernels need FP4 activations to feed, so every FP4 backend declines and selection falls through to MARLIN, which dequantizes the weights back up and runs a BF16 GEMM. Unsloth's build quantizes attention q/k/v/o to FP8 with dynamic per-token activation scales, which is what qualifies it for CUTLASS.

What it costs, measured on one Spark, vLLM 0.24.0, thinking off, warm, c1:

	NVIDIA NVFP4A16 (MARLIN)	Unsloth NVFP4-Fast (CUTLASS, MTP k=3)
decode @ 512 / 2K / 8K	42.9 / 42.6 / 41.4 tok/s	101.3 / 109.1 / 105.3 tok/s
prefill @ 2K / 8K	5,872 / 6,431 tok/s	5,917 / 6,521 tok/s
ITL @ 2K	23.5 ms	~9 ms
weights resident	21.88 GiB	20.58 GiB (22.15 with draft)

Prefill is a dead heat — MARLIN's dequantize cost amortizes over a large batch of tokens. The entire loss is decode, where the fallback kernel is paid on every single-token step. So the practical rule: grep the NvFp4 MoE backend line on every new checkpoint, even one from the model vendor, and even at the same nominal precision as one that worked. "NVFP4" on the model card does not tell you which kernel you will get.

2026-07-27 — a config can declare an MTP head the checkpoint does not contain

nvidia/Qwen3.6-35B-A3B-NVFP4 sets mtp_num_hidden_layers: 1 in config.json, and its own recipe.yaml lists re:^mtp.* in the quantization ignore list — both of which read as "the MTP head is here and deliberately left unquantized". It is not there. The weight index holds 93,426 tensors and zero matching mtp:

python3 -c "
import json,sys
ks=list(json.load(open(sys.argv[1]+'/model.safetensors.index.json'))['weight_map'])
print(len(ks),'tensors;','mtp:',[k for k in ks if k.startswith('mtp')])" ~/models/hf/<ckpt>

Unsloth's build of the same base model keeps all 19 mtp.* tensors, BF16, 1.573 GiB. On Qwen3.6 35B-A3B that head is worth +57% decode, so the difference is not cosmetic.

Rule: check the weight index, not the config, before planning a recipe around a checkpoint's MTP/EAGLE head. A quantization pipeline that excludes a module from quantization can also drop it from the export, and the config it copies forward will still advertise it.

2026-07-27 — nohup is not enough: use setsid for vLLM servers started over ssh

Twice in one run a vLLM server launched over ssh as nohup vllm serve ... & died at the moment an unrelated local pkill reaped ssh processes on the client side. nohup blocks SIGHUP for the launched process, but vLLM's engine-core and API-server children are in the same process group as the dead sshd session and take the group signal. The server logged a clean shutdown ("leaked semaphore objects to clean up at shutdown"), so it reads like a graceful exit rather than a kill, which makes it easy to misdiagnose as an engine crash.

Launch with setsid — a new session, so no controlling terminal and no shared group:

setsid nohup vllm serve ... > /tmp/vllm.log 2>&1 < /dev/null &

Same applies to the ssh tunnel used to reach the server from another host: killing a stale forwarder by pattern also matches the shell running the pkill, so prefer a pattern that cannot self-match (pkill -f 'N [-]L 8000') or track the PID.

2026-07-27 — vLLM 0.26.0 regresses the Gemma 4 assistant draft; 0.24.0 works

A speculative draft that works on one engine build is not a property of the checkpoint — re-test it on every bump. google/gemma-4-26B-A4B-it-assistant (0.42B) drafts for gemma-4-26B-A4B-it correctly on vLLM 0.24.0 and cannot be loaded at all on 0.26.0. Byte-identical serve command, same checkpoint, same 262,144-token context; 0.24.0 reaches Application startup complete, 0.26.0 dies during torch.compile:

RuntimeError: a and b must have same reduction dim, but got [s47, 3840] X [5632, 1024]

The shapes identify the bug without reading any source. The draft's input projection is [1024, 5632], and 5632 = 2 × 2816 = twice the target's backbone_hidden_size — the assistant consumes two target hidden states concatenated, which is what it was trained on. 0.26.0 feeds it 3840 = 2816 + 1024, i.e. the target hidden concatenated with the draft's own hidden_size. The engine changed how it assembles the assistant's input. (0.26.0 logs SpeculativeConfig(method='mtp', ...) for this draft, so it routes the assistant through the MTP path.)

Two general lessons:

• Without the draft, 0.26.0 serves this model fine and at identical speed (39.87 / 38.97 / 37.63 tok/s at 512 / 2,048 / 8,192-token prompts, against 39.74 / 38.61 / 37.34 on 0.24.0). So an engine bump can be a no-op on the target path and a hard break on the draft path at the same time. Test the spec-decode config specifically, not just "does the model still serve".
• Read the shape mismatch before you read the changelog. Factorising the two numbers against hidden_size and backbone_hidden_size from the draft's config.json located the defect in a couple of minutes.

2026-07-27 — 25-of-30 sliding-window layers make a 262K context almost free

Gemma 4 26B-A4B is the clearest example we have of context cost being an architecture property rather than a quantization or KV-dtype one. Its layer_types is 25 sliding_attention (window 1,024) to 5 full_attention, and the global layers carry only num_global_key_value_heads: 2. Measured consequence on one Spark at FP8: 24.6 KB/token, so the full max_position_embeddings of 262,144 needs about 6.2 GiB of KV. Weights (25.83 GiB) plus draft (0.78 GiB) plus a 7 GiB KV pin is ~34 GiB — a quarter of the box, at the model's maximum context.

Practical notes:

• Count layer_types before budgeting memory. Counter(config['text_config'] ['layer_types']) is a two-line check that predicts the entire KV budget, and it is the first thing to read on any Gemma-shaped or hybrid checkpoint. A model with all-global attention at the same parameter count would not fit this context.
• Left alone, the engine takes far too much cache. At --gpu-memory-utilization 0.85 vLLM sized the pool at 75.14 GiB = 3,202,672 tokens = 12.22x concurrency at full context. For a single user everything above 1.0x is unusable, and here --max-model-len is already at the model ceiling so it cannot be spent on context. Pin it: --kv-cache-memory-bytes 7516192768 gives 1.23x and decode does not move (63.76 vs 63.42 tok/s at a 2K prompt, same day, same flags). Roughly 68 GiB comes back.

2026-07-27 — ignore_eos + truthiness = fabricated decode speeds (harness bug, fixed)

bench/harness.py reported 471 tok/s on an 8,192-token-prompt scenario for a model whose own e2e_ms and ttft_ms in the same record implied 37.6 — a 12.5x overstatement that two of three repeats hit, so the median reported it too.

Cause: the benchmark method sets ignore_eos so every request generates exactly its output budget. Once the model would have stopped, vLLM keeps emitting and streams those tokens as empty-string content deltas — verified directly on a live server, 121 content deltas of which 27 were "". The timing loop tested if delta.get("content"), and "" is falsy in Python, so the entire tail of the stream was dropped: t_last froze at the last non-empty token while completion_tokens still came from usage. Dividing a full token budget by a fraction of the elapsed time inflates decode_tps by exactly the ratio the model stopped early by. It is worst on scenarios with a large output budget and a prompt that invites a short answer — which is why the 512- and 2,048-token rows looked sane and the 8K row did not.

Fixed by testing for presence rather than truthiness:

if delta.get("content") is not None or delta.get("reasoning_content") is not None:

• The tell is arithmetic, not intuition: (e2e_ms − ttft_ms) ÷ completion_tokens must reconcile with the reported itl_ms. When it doesn't, the stream parser is dropping chunks. Worth checking any suspiciously good decode number this way.
• Caveat on older rows: vLLM spec-decoding/standard rows measured before this fix can be inflated wherever the model stopped well before its output budget. Rows whose itlMs reconciles with (e2eLatencyMs − ttftMs) ÷ outputTokens are unaffected; the committed raw runs under bench/results/ carry the per-request fields needed to check. Not audited across the back catalogue.

2026-07-27 — the vision-tower flag is a two-sided decision; count tensors, not configs

Follow-up to "a registered arch is not a loadable arch: the unconditional vision tower" above, and the reason that entry needs a second half.

That note came from Qwen/Qwen-AgentWorld-35B-A3B: a language-only release carrying an inherited vision_config, whose Qwen3_5MoeForConditionalGeneration.__init__ builds a 27-block Qwen3_VisionTransformer unconditionally and then dies with 113 uninitialized visual.* parameters. Fix: a self.visual = None guard in qwen3_5.py plus --language-model-only. That guard is now installed on Spark-1's vLLM 0.24.0 venv.

t-tech/T-Search-NVFP4 is the same architecture class, the same 35B-A3B shape, the same inherited-looking vision_config — and it actually ships the tower: 333 model.visual.* tensors in model.safetensors.index.json. Serving it with --language-model-only on that patched venv produced the mirror-image failure:

ValueError: There is no module or parameter named 'visual' in
Qwen3_5MoeForConditionalGeneration.

i.e. the guard skipped the tower and the loader had 333 weights with nowhere to put them. Dropping the flag was the entire fix — stock vLLM 0.24.0 serves T-Search unpatched, at quantization=modelopt_fp4, with FLASHINFER_CUTLASS NvFp4 MoE and FLASHINFER attention auto-selected on sm_121.

What to carry forward:

• "This arch needs the language-model-only patch" is not a property of the arch. It is a property of the individual checkpoint, and two checkpoints of the same class one week apart went opposite ways.
• The weight index is the only evidence that settles it, and it is one HTTP fetch: sum('visual' in k for k in index['weight_map']) — >0 → serve with no flag; ==0 → flag and guard. Do this before writing the serve line, not after a 3-minute weight load.
• The two error strings ("Following weights were not initialized" vs "There is no module or parameter named") look unrelated. They are the same question answered in opposite directions, and neither one names the flag that caused it.
• Once the guard is installed on a venv it is inert without the flag, so it does not need reverting between checkpoints — the flag is the whole control.

Same run, worth recording alongside: --speculative-config '{"method":"qwen3_5_mtp","model":"<the target's own directory>","num_speculative_tokens":3}' works out of the box on this checkpoint because its 785 mtp.* tensors live in model-mtp.safetensors and are referenced by the main weight index. qwen3_5_mtp.py already carries both the mtp. → model. rename and an explicit workaround for mtp.fc being stored BF16 inside an NVFP4 checkpoint. Measured 37.78 → 61.72 tok/s at a 2048-token prompt on one Spark (1.63x, same node, warm); k=4 regressed to 52.94. So on this family, "the config declares an MTP head" is worth checking against the index (see the 2026-07-27 AgentWorld entry where it was declared and absent) — and when the tensors are really there, it is a one-flag win costing 1.57 GiB.

2026-07-27 — the payoff from quantizing a speculator scales with the recipe's k, and it is knowable in advance

Two Sparkulator runs a few hours apart, same quantizer (data-free group-128 W4A16 RTN over the raw safetensors), same measurement protocol (12 repeats/cell, both nodes, arm order counterbalanced), opposite outcomes:

target, 1 Spark	tuned k	draft	replicated decode delta
Qwen3-Coder-Next NVFP4, EAGLE3 draft	1	0.52 → 0.32 GiB	+0.7 / +2.8% — inside a 2.4-3.1% noise floor, not shipped
Laguna-S 2.1 NVFP4, DFlash draft	6	2.08 → 0.82 GiB	+2.6 to +7.3%, 6/6 arms positive, shipped

The reason is in the decode arithmetic: the draft's share of a step is draft_read × k, so a recipe already tuned to k=6 reads its draft six times per accepted-token group and a k=1 recipe reads it once. k is published on the recipe page, so the cheapest way to pick a Sparkulator target is to sort existing recipes by their measured num_speculative_tokens and start at the top — before downloading a single byte.

Measured on Laguna-S-2.1-NVFP4 (one Spark, vLLM 0.25.1, k=6, --max-num-seqs 4, KV pinned 12 GiB, temp 0, c1, warm, code workload):

• Spark-1 BF16 draft → W4A16: 44.14 → 45.28 (512), 42.72 → 45.85 (2,048), 44.34 → 47.00 (8,192)
• Spark-2 W4A16 → BF16 (order reversed): 44.79 → 41.80, 44.82 → 43.59, 45.90 → 44.11
• acceptance unchanged: 61.84 → 61.67% (Spark-1), 61.34 → 61.15% (Spark-2)
• resident model memory 69.34 → 68.08 GiB, i.e. the draft's 1.26 GiB of savings shows up exactly where the file sizes predict — unlike an EAGLE3 draft, a DFlash draft carries no shared-with-target embedding table to inflate its on-disk size

2026-07-27 — greedy non-determinism on sm_121 reproduces on a second model

Three byte-identical temperature: 0, seed: 0 chat completions fired at the same Laguna-S-2.1-NVFP4 vLLM 0.25.1 server returned three different completions (they diverge in the first sentence). This is the same behaviour first seen on Laguna-XS earlier the same day, so it is a property of the stack, not of one checkpoint. Consequence for spec-decode work: the "greedy output must be token-identical with and without the draft" correctness check is not runnable on this stack — it fails on engine noise before it can tell you anything about the draft. Use draft acceptance from /metrics as the gate instead.

2026-07-27 — a 92 GB model turns two known startup traps into 10-minute mistakes

Bringing up moonshotai/Kimi-Linear-48B-A3B-Instruct (BF16, 91.53 GiB resident, 20 shards) on one Spark cost two failed boots before a good one, and both failures are cheap to avoid once you know the shape. What makes them worth writing down is the load time: weights take 620 s to read, so every startup failure that happens after weight load costs eleven minutes, and every one that happens before costs thirty seconds. Order your risks accordingly.

1. The vLLM 0.26.0 unified-memory profiling assert fires here, and the model is big enough that you feel it. With --gpu-memory-utilization 0.90 and no KV pin, the engine died with no traceback immediately after torch.compile took 12.55 s in total — the exact signature already documented for 0.26.0's assert init_snapshot.free_memory >= free_gpu_memory. The only trace it leaves is a resource_tracker: There appear to be 1 leaked semaphore objects warning from the multiprocessing shutdown. On a small model this is an annoying 3-minute retry; at 620 s of weight load it is the single most expensive way to learn the rule. So: on 0.26.0, pass --kv-cache-memory-bytes from the first boot, not after the first failure — you cannot afford to discover the assert empirically on a large checkpoint.

2. The startup free-memory gate races the previous engine's teardown. Relaunching 3 s after pkill -f "[v]llm serve" failed in ~30 s with:

ValueError: Free memory on device cuda:0 (95.11/121.69 GiB) on startup is less than desired
GPU memory utilization (0.9, 109.52 GiB). Decrease GPU memory utilization or reduce GPU
memory used by other processes.

This is not a real memory shortage and the remedy the message suggests (lower the utilization) is the wrong one — the dying engine was still holding its 91.53 GiB. Thirty seconds later free -g read 110 GiB free and the identical command started fine. The check compares current free memory against utilization × total, so on unified memory it is sensitive to anything transient: a teardown in flight, page cache from the previous load, a concurrent du. Wait for pgrep -f "[v]llm serve" to go empty and free -g to settle before relaunching, and read the number in the error as evidence about timing rather than about your flags. Lowering --gpu-memory-utilization to satisfy it would have quietly under-provisioned the real run.

3. A hybrid KDA/mamba model rounds the attention block size up to match its mamba page. The same boot logged, before any tuning:

Setting attention block size to 1888 tokens to ensure that attention page size is >= mamba page size.
Padding mamba page size by 0.19% to ensure that mamba page size and attention page size are exactly equal.

So on a hybrid linear-attention model the KV block size is not the familiar 16 — it is derived from the recurrent state's size, and KV accounting is quantized to 1,888-token pages. Worth knowing before you try to predict a pool size from bytes-per-token arithmetic: the per-token figure is right, but the pool is allocated in pages that large.

4. A long prefill can kill the engine outright, and --max-num-batched-tokens is the lever (2026-07-27). Also on Kimi Linear, and this is the finding with the widest reach. Serving at --max-model-len 524288 with no chunk bound, a 131,072-token prompt killed vLLM 0.24.0 mid-prefill — the engine logged prefill progress, reached ~45,000 tokens (KV pool 6.8% full), then died with the same no-traceback / leaked-semaphore signature as the startup deaths above. KV occupancy rules out a pool shortage. Re-serving with --max-num-batched-tokens 4096 got a 98,304-token prompt through cleanly (26.3 tok/s decode, 37.8 s TTFT); a 130,000-token prompt then died the same way. So the chunk bound moves the ceiling from ~45K to somewhere in (98,304, 130,000] without removing it.

Three things generalize:

• A context you never sent a request at is a guess, and on this stack it is often a wrong one. The engine started happily at --max-model-len 524288 and reported a 656,759-token KV pool. Both numbers were real and neither predicted that a quarter of that length would kill it. Verify the claimed context with an actual prompt at that length before it reaches a recipe page.
• When a long prefill crashes, reach for --max-num-batched-tokens before you cut the context. It cost nothing measurable on decode here and roughly doubled the usable prompt length. Cutting --max-model-len instead would have hidden the bug and shipped a worse page.
• The no-traceback death is not one bug. We now have it at startup (0.26.0, twice) and mid-prefill (0.24.0, twice) on the same model. Treat "engine vanished, leaked semaphore warning, nothing in the log" as a symptom class on unified memory, not a fingerprint — and bisect with the cheapest distinguishing test you have, which is usually prompt length.

2026-07-27 — the parameter count is the wrong number: a dense 9B decodes 2.4x slower than a 35B-A3B MoE

Both members of the same family, both BF16, same node (Spark-1), same engine (vLLM 0.24.0), same day, warm, greedy, single-stream at a 2,048-token prompt:

	Ornith-1.0-9B (dense)	Ornith-1.0-35B (MoE, top-8 of 256)
weights on disk	17.53 GiB	65.39 GiB
bytes read per decode token	15.87 GB	5.89 GB
arithmetic ceiling at 273 GB/s	17.2 tok/s	46.3 tok/s
measured decode	12.61 tok/s	30.3 tok/s
fraction of its own ceiling	73%	65%

The small model has 26% of the parameters and is 2.40x slower. The byte ratio is 2.69x, so bandwidth accounts for the gap to within 12% and there is nothing else to look for — both configs are healthy fractions of their own roofline. The mechanism is entirely routing: the 35B's 60.0 GiB of routed experts contribute only 1.88 GiB to a decode step, so its 65 GiB checkpoint reads lighter than the 9B's 17.5 GiB one.

Practical consequences on this hardware:

• Screen candidates by bytes-read-per-token, never by parameter count or file size. It is a safetensors header read (JSON at the front of each shard), so it costs seconds and needs no download beyond the index. Read/token = all weights − vision tower (only runs on image inputs) − embed_tokens (a gather). lm_head is read every token; it is 1.90 GiB here and easy to forget.
• bench/ckpt_bytes.py mis-reports a DENSE checkpoint, silently. Its categories assume an MoE. On Ornith-9B it classified only the attention projections and printed "939.5 MB/token → ceiling at 273 GB/s = 290.6 tok/s" — 17x optimistic — with no error and a routed MoE 0.0 MB line that is easy to read past. Sanity-check that its category table sums to the file size before quoting its ceiling.
• The KV budget inverts as well, and this is the one that bites the serving config. KV cost is full_attn_layers × 2 × kv_heads × head_dim × 2 and is unrelated to parameter count. Ornith-9B keeps 4 KV heads on 8 full-attention layers = 32 KiB/token; the 35B keeps 2 on 10 = 20 KiB/token. So the same 262,144-token window costs just over 8 GiB on the small model against ~5 GiB on the big one, and the --kv-cache-memory-bytes 8589934592 that gives the 35B a comfortable 1.58x lands the 9B at ~1.0x — no margin for the few-percent boot-to-boot pool variation, i.e. an intermittent refusal to start. 12 GiB gave a measured 390,070 tokens / 1.49x. Recompute the pin per checkpoint; never inherit it from a bigger sibling.
• A 4-bit quant pays more on a dense model than on a sparse one, which inverts the 2026-07-20 "quantizing the experts can optimize the wrong thing" note. There, top-k routing meant experts were most of the file and little of the read. Here every byte quantized is a byte off the per-token read, and the dense MLP alone is 9.00 of the 15.87 GB.

Other measurements from the same serve, for the record: weights load in 108 s cold from NVMe, torch.compile 28 s, profiling/warmup 37 s; backends FLASH_ATTN v2 plus Triton/FLA GDN prefill kernel (head_k_dim=128); steady-state free -g 43 of 121 used, so a ~39 GiB working set against the 114 GiB ceiling — three quarters of the Spark idle while serving the full 262,144-token window. Two consecutive harness passes agreed to within 0.04 tok/s, confirming the skill's rule that a no-draft config is essentially noiseless (the variance that needs repeats is a speculative-decode phenomenon).

2026-07-27 — hybrid_override_pattern: count the * before you budget KV, and check the checkpoint's own KV dtype

Third instance of the "hybrid attention makes context nearly free" family (2026-07-19 with full_attention_interval, 2026-07-27 with sliding windows), but with two new spellings worth having by name, both found on nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B-NVFP4 (model_type: nemotron_h, 31.6B, 128 experts, 6/token) on Spark-1 / vLLM 0.24.0.

• nemotron_h encodes its layer plan as a STRING, not a stride. config.json carries hybrid_override_pattern: "MEMEM*EMEMEM*EMEMEM*EMEMEM*EMEMEM*EMEMEMEM*EMEMEMEME" — 52 characters, one per layer: M = Mamba mixer (23), E = MoE block (23), * = full attention (6). python3 -c 'print(p.count("*"))' is the entire KV budget analysis, and it takes seconds. Nothing else in the config announces that only 6 of 52 layers hold a cache — num_hidden_layers is 52 and num_key_value_heads is 2, and multiplying those two overstates KV by 8.7x.
• The checkpoint can specify its own KV precision, and vLLM silently honours it. hf_quant_config.json here declares "kv_cache_quant_algo": "FP8" alongside the NVFP4 weight scheme, and the startup config comes up kv_cache_dtype=fp8_e4m3 with no flag passed. So the per-token KV cost was half what the BF16 arithmetic predicted. Read kv_cache_quant_algo out of hf_quant_config.json before doing the KV arithmetic — it is a separate key from quant_algo and it is easy to skim past.
• Measured result: 982,061 tokens from a 3 GiB pinned pool = 3.20 KiB/token, i.e. 3.75x concurrency at the model's full 262,144-token window, on a node whose total serving working set is 28.3 of 114 GiB. The previously published serve command for this checkpoint used --max-model-len 10240 --gpu-memory-utilization 0.5 and cost 68.3 GiB peak for 10,240 tokens; 262,144 tokens with the pinned pool costs 32.3 GiB peak and decodes at 61.16 tok/s against 62.13, matched at a 2,042-token prompt, same node, same engine, both warm. Twenty-five times the context, half the memory, no measurable speed.
• Decode on this arch is unusually flat with fill level — 61.26 / 61.16 / 60.81 / 59.79 / 56.00 tok/s at 512 / 2,048 / 8,192 / 32,768 / 131,072-token prompts, 8% over a 256x range — because 46 of 52 layers have no growing cache to re-read. The cost lands entirely on prefill: TTFT 123 ms / 242 ms / 902 ms / 3.96 s / 25.9 s across the same points, and a single 260,025-token request took 74.8 s end to end. A prompt past the window returns a clean HTTP 400 in ~0.5 s rather than killing the engine.

2026-07-27 — the [v]llm serve pkill guard fails if the HEREDOC you are shipping contains the string

Known trap: ssh host 'pkill -f "vllm serve"' matches the remote shell's own argv and kills the session, so we write [v]llm serve. The character class protects the pattern — it does not protect the rest of the command line. Sending a serve script over SSH as a heredoc in the same invocation as the pkill puts the literal text vllm serve into that shell's argv, and pkill -f "[v]llm serve" matches it and kills the session mid-write (exit 255, no script written, server already dead). Write the script to a local file and scp it, or make the pkill a separate SSH call.

Corollary, same day, llama.cpp flavour: the character-class guard also fails when the serve command you are about to launch in the same one-liner contains the unguarded path. ssh Spark-1 'pkill -f "[l]lama-server"; setsid ~/Dev/llama.cpp-laguna/build/bin/llama-server -m … &' self-kills, because the argv now holds a literal llama-server from the second half. Two ssh calls, or two scripts (/tmp/killsrv.sh + /tmp/serve-*.sh) shipped by scp, is the only shape that reliably survives. Also note a killed-but-unkillable match: a root-owned wrapper shell left over from an earlier session makes pkill exit non-zero ("Operation not permitted") without meaning your server survived — check pgrep -af for the actual binary rather than trusting pkill's exit code.

2026-07-27 — a quantized MTP draft can load with zero errors and be pure noise

Building the Sparkulator for Gemma 4 26B-A4B turned up a failure mode that no log line announces, on vLLM 0.24.0. It is worth checking for on any MTP/EAGLE draft, not just this one.

Gemma4MTPAttention and Gemma4MTPDecoderLayer construct the draft's q_proj, o_proj and mlp.{gate,up,down}_proj with a hardcoded quant_config=None, so those modules are always built as plain BF16 linears expecting a .weight. Then Gemma4MultiTokenPredictor.load_weights ends its dispatch chain with if name not in params_dict: continue — and vLLM's strict "weights not initialized" check is disabled whenever model_config.quantization is not None (default_loader.py: default_enable_weights_track = model_config.quantization is None).

Serve a compressed-tensors W4A16 draft into that and every weight_packed / weight_scale / weight_shape tensor is silently dropped. The projections keep their random initialisation, startup is clean, Application startup complete appears, answers are still correct (speculation is verified), and the only symptom is speed:

• acceptance 3.3% on Spark-2 and 2.1% on Spark-1 (against 44% for the BF16 draft), with essentially all accepts at draft position 0 — the shape of a coin-flip drafter
• Model loading took identical to the BF16 draft, 26.61 GiB, because BF16-shaped parameters were still allocated — which is the cheapest early warning available

Three practical rules:

1. Read Model loading took for the quantized arm and expect it to drop. If a quantized draft costs exactly what the BF16 one did, its weights did not load. (Correct here: 26.61 → 26.41 GiB, matching the checkpoint's 0.782 → 0.585 GiB.)
2. Demand the kernel line. A genuinely quantized draft logs Using MarlinLinearKernel for CompressedTensorsWNA16. No line, no quantization.
3. Acceptance is the only honest gate, and near-zero acceptance means broken, not over-quantized. Data-free group-128 RTN on this 4-layer draft cost nothing measurable once it actually loaded (62.8% vs 60.5–65.3%).

The one-file fix is committed at scripts/patch_gemma4_mtp_draft_quant.py: pass the layers a quant config resolved from the draft's own quantization_config.

The mirror-image bug is worse, and it is the reason the patch has to be conditional. The obvious version of the fix — hand the draft vllm_config.quant_config — makes the draft inherit the target's config. Against an FP8 target that builds the BF16 draft's MLP as FP8 W8A8, drops its weights by the same silent path, and gives 0% acceptance (measured: 0 accepted of 1,524 drafted, and Model loading took fell to 26.48 GiB, FP8-shaped). Override only when the draft ships a quantization_config of its own.

2026-07-27 — a 122B MoE fits one Spark as a community GGUF, and llama.cpp already speaks qwen35moe

Getting BAAI/AREX-Base (122B-A10B, qwen3_5_moe, 245.1 GB of BF16) onto a single node took no porting and no quantization work of our own — bartowski/BAAI_AREX-Base-GGUF already ships it, and the Q4_K_M cut is 74.9 GB, which loads on one Spark with ~36 GiB spare. Four things worth carrying forward:

• Check llama.cpp's arch table before assuming a brand-new model needs a port. grep -o QWEN35MOE ~/Dev/llama.cpp/src/llama-arch.h on our 2026-07-07 checkout (ee445f9, server reports version: 15) already hits, and that build served a model released 2026-07-24 with no changes at all. The vLLM instinct ("new arch ⇒ out-of-tree plugin") does not transfer: llama.cpp lands architectures early, and the existence of a bartowski GGUF is itself evidence that upstream can run it.
• A GGUF route can be strictly cheaper than a vLLM route, not just a fallback. With no FP8/NVFP4 build on the Hub, the vLLM path for this model is a 245 GB download plus a self-quantization pass — hours. The GGUF path was 74.9 GB and one serve command. When a checkpoint is >114 GiB at BF16 and has no vendor 4-bit build, check GGUF first.
• Spark-2's wifi default route is not automatically slow. The standing advice is to pull large checkpoints on Spark-1 and rsync over RoCE. Measured this run: hf download straight to Spark-2 sustained ~61 MB/s (74.9 GB in ~20 minutes), steady, confirmed against /sys/class/net/wlP9s9/statistics/rx_bytes. A single-stream curl of the same host measured only 32 MB/s, so do not price a download from a one-connection test — xet's parallel chunk fetch roughly doubles it. Rsync-from-Spark-1 is still right when the bytes are already on Spark-1; it is not worth a detour when they are not.
• du -s under-reports an in-flight hf download. Progress looked stalled at 5 MB/s because the glob being watched was *.incomplete and one of the two shards had already finished and been renamed. Watch stat -c %s over all blobs, or just the NIC counter.

2026-07-27 — a reasoning model over llama.cpp returns an empty content

BAAI/AREX-Base on llama-server answers with choices[0].message.content == "" and the entire response in choices[0].message.reasoning_content, until the thinking block closes. With a small max_tokens you get a blank-looking reply and "finish_reason": "length", which reads exactly like a broken serve or a bad chat template. It is neither. Two consequences:

• A client that reads only content (including some benchmark scripts and most naive curl one-liners) sees nothing. bench/harness.py is unaffected because it counts tokens from the server's usage block rather than from the message body — which is the reason that rule exists.
• Budget max_tokens for thinking plus answer when sanity-checking one of these models by hand; 512 is a sane floor for a one-line question.

2026-07-27 — greedy determinism on vLLM 0.26.0 holds within a server, not across boots

A refinement of the two 0.25.1 findings above, measured on Qwen3.6-27B-NVFP4 across five server boots on both Sparks. Within every single boot, three byte-identical temperature: 0, seed: 0 completions came back identical (5/5 boots, 3/3 probes each) — unlike 0.25.1, where the same-server probe already disagreed with itself. But two boots of a byte-identical configuration returned different completions from each other, and so did the two arms of an A/B.

So 0.26.0 is per-process deterministic and not cross-process deterministic. The practical consequence is unchanged — the "greedy output token-identical with and without the draft" check compares two different servers, so it still cannot run, and acceptance from /metrics remains the gate — but the diagnosis differs, so keep firing the 3x same-server probe rather than assuming which regime you are in. If the same-server probe is stable, a cross-server mismatch tells you nothing; if it is unstable, you are on the older behaviour.

2026-07-27 — a bad vLLM boot can fake a 40% regression, and only the per-request samples show it

Benchmarking a quantized MTP draft on Spark-2, one boot measured 34.39 / 27.95 / 15.52 tok/s at 512 / 2,048 / 8,192-token prompts. A re-boot of the byte-identical configuration measured 40.24 / 38.11 / 29.58. Same binary, same checkpoint, same flags, same node, twenty minutes apart.

The medians give you no way to tell which one to believe. The per-request samples do:

• bad boot: individual requests at 8.0 and 17.1 tok/s against ~28 medians; at 8K, five consecutive requests at ~15.5 then one at 29.0 — a sustained stall that recovers, not scatter. Per-request sd 6.5 / 6.0 / 5.1.
• every other boot on either node, either arm: per-request sd 0.5 - 2.4.

Nothing appeared in the vLLM log, there was no OOM, and the node was otherwise idle; the cause is still unknown. Two rules fall out. Read the per-request spread before you believe a median — bench/harness.py commits every sample to bench/results/, so this costs nothing. And when you drop a run, drop it for a pathology you can point at in the samples, never because the number was inconvenient: commit the raw file, say in the notes that you dropped it and why, and do not publish it as a benchmark row, because it measures a fault rather than a configuration.

2026-07-27 — ModelOpt checkpoints carry the quant config twice, and vLLM prefers config.json

A ModelOpt export like nvidia/Qwen3.6-27B-NVFP4 ships the same quantization description in two places: config.json → quantization_config (with quantized_layers and ignore) and the legacy top-level hf_quant_config.json (with quantized_layers and exclude_modules). vLLM 0.26.0's ModelOptMixedPrecisionConfig reads config.json in preference — its own docstring says "inside config.json's quantization_config (preferred) or the legacy hf_quant_config.json".

Edit only the legacy file — the obvious move, since it is the one that looks like a quant config — and every layer you added is silently ignored. The layer gets built unquantized, is then handed packed uint8 weights, and the load dies with a bare

File ".../model_executor/parameter.py", line 153, in load_column_parallel_weight
    assert self.data.shape == loaded_weight.shape
AssertionError

with no layer name anywhere in the traceback. It is the first plain ColumnParallelLinear in load order that reports it (here mtp.fc), which is not necessarily the layer you care about. Patch both files, and remove the module from both exclusion lists (ignore and exclude_modules) — they are not aliases of each other.

2026-07-28 — the kernel JIT, not the model, is what kills a fresh NVFP4-MoE TP2 boot

Bringing up poolside/Laguna-M.1-NVFP4 (139.32 GB, TP2 across both Sparks, vLLM 0.26.0) died three times in a row with no traceback: shards load, both ranks print Model loading took 64.98 GiB memory, host memory jumps by roughly the KV pin, keeps climbing, and the process disappears. The local oom-guard.service named the cause — MemAvailable=4726 MiB < 8192 MiB — but nothing in the vLLM log did.

The instinct is to blame the profiling forward and shrink --max-model-len / --kv-cache-memory-bytes. That is wrong, and shrinking the KV pin from 20 GiB to 16 GiB did not fix it. What fixed it was MAX_JOBS=4 on the raylet environment:

KV pin	max-model-len	max-num-batched-tokens	MAX_JOBS	peak host used	outcome
20 GiB	262,144	8,192	unset	106.4 GB	killed
16 GiB	196,608	4,096	unset	96.6 GB, rising	killed
16 GiB	196,608	4,096	4	106.1 GB	served

Rows 2 and 3 differ in nothing but MAX_JOBS, which settles it: the memory was the compiler. flashinfer's NVFP4-MoE cutlass kernel JIT-compiles on the first forward, which runs inside profile_run, and ninja defaults to nproc — 20 on a Spark — parallel nvcc at 3–4 GiB each. None of that shows up in torch.cuda.* accounting, so the symptom is a host-memory OOM that looks exactly like an activation spike.

This generalizes the Solar-Open2 finding (2026-07-23) from "a fresh out-of-tree port" to any NVFP4 MoE shape the box has not compiled before, including a natively-registered arch on a stock engine. Three practical notes:

• The tell is pgrep -c 'cicc|nvcc|ptxas|ninja' on BOTH nodes. Capped at MAX_JOBS=4 it sampled 7–9 (ninja's 4 jobs plus each nvcc's cicc/ptxas children), which is the healthy reading — not 0.
• It must be on the raylet, not the serve command. Ray actors inherit the raylet's environment; exporting it beside vllm serve does nothing.
• Budget the time, not just the memory. Shards finished at 21:20 and Application startup complete came at 21:35 — ~15 of the ~25-minute first boot was compilation, cached under ~/.cache/flashinfer afterwards. A first boot that looks hung for a quarter of an hour after weight loading is normal here.

2026-07-28 — FLASHINFER_CUTLASS is still available for NVFP4 MoE on 0.26.0 (checkpoint-dependent)

The 2026-07-27 note says FLASHINFER_CUTLASS is "gone for NVFP4 MoE on GB10" under vLLM 0.26.0 and that auto-selection silently drops to VLLM_CUTLASS. That is narrower than it reads. On the same engine build, poolside/Laguna-M.1-NVFP4 auto-selected

Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend out of potential backends:
['FLASHINFER_TRTLLM', 'FLASHINFER_CUTEDSL', 'FLASHINFER_CUTEDSL_BATCHED',
 'FLASHINFER_CUTLASS', 'VLLM_CUTLASS', 'MARLIN', 'HUMMING', 'EMULATION']

on both ranks and served correctly. So the backend menu is a function of the checkpoint's shape and quant layout as well as the engine version — keep grepping the log for what was actually chosen on this model rather than inheriting a verdict from another one.

2026-07-28 addendum — a third data point. t-tech/T-Search-NVFP4 (ModelOpt NVFP4 export, Qwen3_5MoeForConditionalGeneration) goes the other way: it auto-selected FLASHINFER_CUTLASS on 0.24.0 and VLLM_CUTLASS on 0.26.0, on the same node with the same flags. Decode was unaffected — 61.72 tok/s on 0.24.0 vs 62.02 on 0.26.0 at a 2,048-token prompt, warm, greedy, MTP draft at k=3 — so treat the backend name in that log line as informational unless it reads EMULATION.

2026-07-28 — vLLM 0.26.0 deprecates the qwen3_5_mtp speculative method name

Checkpoints in the Qwen3.5/3.6 MoE family that ship their own MTP head were served with --speculative-config '{"method":"qwen3_5_mtp",...}'. On vLLM 0.26.0 that still works but logs:

WARNING [speculative.py:687] method `qwen3_5_mtp` is deprecated and replaced with mtp.

Verified on t-tech/T-Search-NVFP4 (Spark-2, 2026-07-28) that {"method":"mtp"} is a drop-in replacement: same Resolved architecture: Qwen3_5MoeMTP, same 23.81 GiB model load, same 689,341-token KV pool, decode inside run-to-run noise. Prefer mtp in new recipes; the old spelling is on borrowed time.

Related, same session: 0.26.0 still refuses full CUDA graphs under spec-decode with the FlashInfer attention backend (CUDAGraphMode.FULL_AND_PIECEWISE is not supported with spec-decode for attention backend FlashInferBackend ... setting cudagraph_mode=PIECEWISE), so a no-draft baseline on this hardware is still measured with a small graph-mode advantage over the speculative arm.

2026-07-28 — llama.cpp: -md alone does nothing; --spec-type defaults to none

Handing llama-server a draft model with -md (and -ngld, and --spec-draft-n-max) is not enough to turn on speculative decoding on a current build. This build (ee445f9, server version: 15) has a multi-speculator framework whose type list is opt-in:

--spec-type none,draft-simple,draft-eagle3,draft-mtp,draft-dflash,ngram-simple,
            ngram-map-k,ngram-map-k4v,ngram-mod,ngram-cache
                        comma-separated list of types of speculative decoding to use
                        (default: none)

With the default none, common_speculative_init() builds an empty implementation list, logs no implementations specified for speculative decoding at TRC level — invisible at the default verbosity, and invisible even at -lv 3 — and returns nullptr. The server then loads the draft model, allocates its context, and never drafts a single token.

The failure is completely silent from the outside, and every surface you would check to catch it agrees that nothing is wrong:

• the server starts normally and serves correct output;
• the draft GGUF is loaded and validated (a broken draft still fails the boot — ours did, on a missing tensor, before this was fixed);
• the boot log contains no spec/draft line at all, at -lv 3;
• timings.draft_n / timings.draft_n_accepted in a /completion response are simply absent (null), because the server only adds them when n_draft_total > 0 — the same thing you would see from a draft with 0% acceptance.

Measured on AREX-Base Q4_K_M + an AREX-Turbo Q4_K_M draft, Spark-1: with -md … -ngld 99 --spec-draft-n-max 1 and no --spec-type, decode was 24.35 tok/s at a 2,042-token prompt — i.e. exactly the no-draft baseline, which is what a silently-disabled speculator looks like. Adding --spec-type draft-simple is what actually enables an external draft model.

Two rules that follow:

• Never conclude "the draft did not pay off" from tok/s alone on llama.cpp. Require draft_n > 0 in the response timings first — that field is the equivalent of vLLM's spec_decode_num_accepted_tokens_total > 0 gate, and it is the only cheap proof that the speculator ran at all.
• A GGUF draft converted from a config that advertises an MTP head it does not contain will not load. convert_hf_to_gguf.py writes nextn_predict_layers from mtp_num_hidden_layers, so llama.cpp then demands a block the weights never had: error loading model: missing tensor 'blk.32.attn_norm.weight' on a 32-layer model. Convert with --no-mtp when the checkpoint's index has zero mtp.* tensors (see the 2026-07-27 note on configs that lie about MTP heads).

2026-07-28 — llama.cpp speculative decoding on a hybrid linear-attention target pays a recurrent-state checkpoint tax

A draft with 100% acceptance made AREX-Base Q4_K_M 17% slower. That is not a contradiction and it is not the draft's fault — it is what speculative decoding costs on llama.cpp when the target keeps a recurrent state.

Measured on Spark-1, llama.cpp ee445f9, -c 262144 --parallel 1 -fa on, warm, greedy, at a matched 2,042-token prompt (AREX-Base 122B-A10B Q4_K_M target, AREX-Turbo dense 4.5B Q4_K_M draft, --spec-type draft-simple):

arm	decode tok/s	vs baseline	acceptance
no draft	24.24	—	—
k=1	20.16	−16.8%	1.000 (191/191)
k=2	20.48	−15.5%	0.842 (240/285)
k=3	19.38	−20.1%	0.717 (261/364)

Both models are behaving exactly as their byte budgets predict, so the loss is elsewhere:

• target 6.489 GB read per token (74.94 GB of GGUF, 256 experts routed top-8), draft 2.698 GB — r = 0.416, so k=1 at 100% acceptance ought to ceiling at 2/(1+r) = 1.41x
• the draft served alone decodes 65.32 tok/s (15.31 ms/token) against the target's 41.3 ms/token — a ratio of 0.371, within 11% of the byte prediction, i.e. no hidden slowness

At k=1 with perfect acceptance the arm produced 2 tokens per step at 20.16 tok/s = 99.2 ms per step, where verify + one draft step accounts for only 41.3 + 15.3 = 56.6 ms. The missing ~42.6 ms/step — about one whole extra target decode — is the state checkpoint. -lv 4 shows it plainly:

llama_context: n_rs_seq      = 0
llama_memory_recurrent: size = 149.06 MiB ( 1 cells, 48 layers, 1 seqs  0 rs_seq)   # target
llama_memory_recurrent: size =  50.25 MiB ( 1 cells, 32 layers, 1 seqs  0 rs_seq)   # draft
srv  load_model: speculative decoding will use checkpoints
srv  load_model: context checkpoints enabled, max = 32, min spacing = 8192

common_context_can_seq_rm() evals two tokens and tries llama_memory_seq_rm(mem, 0, 1, -1); a recurrent memory cannot drop a suffix, so the context is classified FULL (or RS with a bounded n_rs_seq, which here is 0) and server-context.cpp takes its use_ckpt_tgt path, saving and restoring 199 MiB of recurrent state (149.06 target + 50.25 draft) around each speculative step. 199 MiB is ~1.5 ms at 273 GB/s, so this is not a device-bandwidth cost — it is the host-side state serialize/restore path, and ~40 ms for ~199 MiB puts it in the few-GB/s range, which is consistent.

What to carry forward:

• On a hybrid/recurrent target on llama.cpp, add the checkpoint tax to the draft screen. The useful ratio is not draft_bytes / target_bytes but (draft_bytes + state_checkpoint_cost) / target_bytes. Here the state term alone is ~1.0 target-steps, which means even a free draft loses: 2 / (1 + 1.03) ≈ 0.99x at k=1.
• Read n_rs_seq and the two llama_memory_recurrent: size = lines at -lv 4 before building a draft for any model with linear/gated-delta/mamba layers. Both numbers are printed at boot and together they price the experiment in five minutes.
• High acceptance is not evidence the config is winning. Acceptance measures the draft's agreement with the target, and on this engine/arch pair it can be perfect while throughput falls. Always compare against a same-session, same-node no-draft baseline.
• This is llama.cpp-specific. vLLM's Qwen3.5/3.6 GDN path partitions the recurrent state for speculation properly (2026-07-24, KAT-Coder), so the same pairing may behave completely differently there — untested on this model, which has no vLLM-servable 4-bit build yet.