howtospark

Field Notes

docs/dgx-spark-notes.md

Software discoveries and gotchas from running LLM inference on the DGX Spark (GB10, aarch64, 128 GB unified LPDDR5x, DGX OS / Ubuntu 24.04). Raw material for future site content — append as things come up, with dates.

OS & drivers

  • DGX OS updates can strand the GPU driver (2026-07-05). The system updater installed the 6.17 kernel and removed the NVIDIA modules for the running 6.11 kernel. Until the next reboot into 6.17, nvidia-smi fails with "couldn't communicate with the NVIDIA driver" and /dev/nvidia* is gone. Diagnosis: compare uname -r against dpkg -l | grep linux-modules-nvidia. Fix: sudo apt full-upgrade, reboot, confirm the new kernel.
  • No passwordless sudo on a default DGX OS install; automation must hand privileged steps back to a human terminal.

aarch64 (ARM) surprises

  • curl | bash installers often ship x86-64 binaries. firebase.tools does — "cannot execute binary file: Exec format error". Anything Node-based installs fine via npm install -g instead. Check for arm64 builds before trusting any install script.
  • System Node is ancient (18.x from Ubuntu). Use nvm; Node 20.11+ is needed for import.meta.dirname.
  • PEP 668: Ubuntu 24.04 blocks pip install --user; every Python tool needs a venv.
  • npm optional-dependency bug bites native ARM bindings (2026-07-06). Tailwind 4 dies with Cannot find module '@tailwindcss/oxide-linux-arm64-gnu' if node_modules was installed under a different Node setup. Fix: rm -rf node_modules && npm install, and clear .next if the error persists in cached build chunks.

GB10 GPU specifics

  • CUDA arch is sm_121 — build llama.cpp with -DCMAKE_CUDA_ARCHITECTURES=121 (CUDA toolkit 13 ships with DGX OS at /usr/local/cuda, present even when the driver is broken).
  • nvidia-smi cannot report memory usage on unified memory ("Not Supported"). Read /proc/meminfo (MemTotal − MemAvailable) instead. power.draw and utilization.gpu work fine.
  • Unified memory changes engine defaults: vLLM's --gpu-memory-utilization defaults to 0.9 — of the whole 128 GB pool, which would starve the OS. Set it explicitly (0.5 leaves ~25 GB KV cache for a 35 GB model).
  • OOM on unified memory takes the whole machine down (2026-07-06). DGX OS ships with no swap, so when a big vLLM run (72 GB BF16 weights, --gpu-memory-utilization 0.75 ≈ 91 GiB target) coincided with first-run Triton kernel compilation (cicc — NVIDIA's CUDA JIT compiler — was what invoked the oom-killer), the kernel spent 5 hours killing desktop processes and the box hard-reset. Rules of thumb: leave ≥20 GiB headroom for model load + JIT compile on first run of a large model, remember GPU allocations and the OS fight for the same pool, and consider a small swapfile purely as an OOM shock absorber.
  • --max-num-seqs is the hidden memory lever in vLLM. Startup profiling simulates the maximum batch (default 256 sequences); on a 72 GB model the resulting activation spike ate 30+ GiB beyond the configured target within seconds. If you're serving one user or benchmarking small concurrency, set it to ~2× your real concurrency and the spike shrinks proportionally.
  • Multimodal models hide a second profiling spike. Qwen3.5 is image-text-to-text; vLLM profiles the vision encoder with maximum-size dummy images on top of the LLM profiling pass. For text-only serving of a large VLM, pass --limit-mm-per-prompt '{"image": 0, "video": 0}' — it skips encoder profiling entirely and saves GBs of transient memory.
  • vLLM's torch.compile cache can go stale across config changes (0.24.0). After adding --limit-mm-per-prompt '{"image": 0}' to a model previously started without it, the cached compiled graph loads and crashes with AttributeError: 'NoneType' object has no attribute 'size'. The cache key evidently doesn't cover multimodal limits — rm -rf ~/.cache/vllm/torch_compile_cache after changing such flags.
  • Throttle compile parallelism when weights nearly fill the pool. Inductor and ninja default to one compile worker per core — 20 on GB10 — and each cicc instance holds 3–4 GiB, so first-run JIT can burst ~70 GiB on top of resident weights. TORCHINDUCTOR_COMPILE_THREADS=2 MAX_JOBS=4 makes first startup slower but keeps the burst to ~10 GiB. This was the common root cause behind repeated OOM crashes loading a 72 GB model.
  • A memory watchdog for big runs must SIGKILL, not SIGTERM — a vLLM engine mid-allocation shrugs off SIGTERM and the memory stays pinned. Poll ≤5s: the profiling spike can go from 40 GiB free to 0 in under 10 seconds.

Engines

  • llama.cpp: builds clean from source with CUDA on aarch64 (~15 min, -j 20). Compiling needs only the toolkit, not a working driver — useful while waiting on a reboot.
  • vLLM: pip wheels for aarch64+CUDA 13 exist and work (0.24.0, torch 2.11). Gotcha: Triton JIT-compiles helpers at import time with the system gcc and needs python3.12-dev (apt) or every model load fails with a misleading "model architecture failed to be inspected" error — the real error (Python.h: No such file or directory) is buried in a subprocess.
  • vLLM FP8 MoE on GB10 needs VLLM_USE_DEEP_GEMM=0 (2026-07-05, vLLM 0.24.0). DeepGEMM targets datacenter Blackwell; on GB10 (sm_121) loading an FP8 MoE model crashes with Assertion error ... Unknown SF transformation. vLLM auto-disables DeepGemm for the dense layers but the MoE backend still selects DEEPGEMM — disable it globally and it falls back to CUTLASS/Triton.
  • Invoke vLLM with its venv on PATH, not just by absolute path. flashinfer JIT-compiles sampling kernels at startup and shells out to ninja; calling ~/venvs/vllm/bin/vllm directly leaves the venv's bin off PATH and the engine dies at KV-cache profiling with FileNotFoundError: 'ninja'. Use PATH="$HOME/venvs/vllm/bin:$PATH" vllm serve ....
  • Gemma 4 assistant (speculative decoding) on llama.cpp, 2026-07-06 state: support landed in master this week and has sharp edges. (1) Community GGUFs may carry arch gemma4_assistant (underscore) while llama.cpp expects gemma4-assistant — convert from Google's safetensors yourself. (2) The assistant conditions on target hidden states: use --spec-type draft-mtp, not draft-simple (which dies with "requires ctx_other"). (3) Acceptance rates are implausibly low even on trivially predictable output (27% on number sequences; should be 90%+), making spec decoding a net 38% slowdown. Likely immature upstream wiring — re-test after llama.cpp updates before concluding anything about speculative decoding on this hardware. Confirmed via vLLM (same draft, same day): vLLM's gemma4_assistant support gets 45–51% acceptance and a 28–46% decode speedup (--speculative-config '{"model": ..., "num_speculative_tokens": 4}'). Speculative decoding works on Spark; llama.cpp's brand-new assistant path doesn't yet. Acceptance counters come from vLLM's /metrics endpoint (vllm:spec_decode_num_{draft,accepted}_tokens_total).
  • "NVFP4" is not one thing — the ignore-list is a decode-speed knob (2026-07-06). Two NVFP4 checkpoints of the same Qwen3.6-35B-A3B arch, same nvfp4-pack-quantized format, same W4A4 scheme, near-identical memory (67.5 vs 69.6 GB): official NVIDIA decoded at 77 tok/s, a community quant (AEON-7 heretic) at 42 — 46% slower — while prefill was identical (~6,230). Cause: the community recipe leaves the hybrid model's linear-attention projections (in_proj_a/b/z, out_proj) in BF16 via a longer ignore list. Those unquantized layers run a slower per-token path that throttles decode but is hidden in (parallel, compute-bound) prefill. Lesson: when benchmarking a quant, diff quantization_config.ignore in config.json — same format label can mean very different decode speed. Not just a quality difference.
  • vLLM 0.24 can't run external-draft spec decoding on hybrid models (2026-07-06). DFlash (and any draft_model/eagle-style external draft) against Qwen3.6-35B-A3B dies at engine init with AssertionError in unify_kv_cache_spec_page_size. Cause: Qwen3.6 is hybrid — its linear-attention layers hold a fixed-size recurrent state whose page_size_bytes doesn't scale with block size, so it can't be padded to match the attention pages the draft adds. vLLM does support DFlash — just not with a hybrid target here. Corollary: this is likely why Qwen ships MTP variants (self-speculation reuses the model's own layers, no foreign page size to reconcile). Workaround: use llama.cpp (--spec-type draft-dflash), which handles the hybrid target; convert the DFlash drafter with convert_hf_to_gguf.py --target-model-dir <target>.
  • Thinking models (Qwen3 family): tokens stream as reasoning_content before content — benchmark clients that only watch content measure TTFT wrong. See bench/README.md for the other benchmarking pitfalls (prefix caching, ignore_eos, per-slot context).

Models larger than RAM (disk streaming)

  • Unified memory doesn't help you load a too-big model — it hurts (2026-07-08). A 557 GB checkpoint (Hy3, 295 B params) can't be loaded on a 128 GB Spark, and because CPU and GPU share one pool, the usual "load to CPU RAM, move blocks to GPU" trick fails two ways: loading the whole thing to cuda OOMs, and even device_map="cpu" overflows the pool. Loading a big model to cuda while something else holds the pool can also hang the CUDA allocator, not just OOM.
  • Stream from the NVMe instead. For layer-wise workloads (REAP calibration, and its prune), build the model on meta (0 resident) and materialize each decoder layer's weights from the on-disk safetensors shards only while that layer is processed, then release it back to meta. Peak RAM is ~one layer (~7 GB for Hy3) regardless of the 557 GB total — the 3.7 TB NVMe is the real capacity limit, not the 128 GB RAM. See the Training page for the REAP recipe.
  • Two Sparks (242 GB) still can't hold a 557 GB model — combining memory helps only in the 128–242 GB band (model-parallel). Above that, disk streaming is mandatory regardless of node count; the second Spark buys throughput (data-parallel calibration) or, with pipeline-parallel, more disk capacity.
  • Clear __pycache__ after editing library code (2026-07-08). A long cuda/cpu device-mismatch hunt turned out to be stale .pyc: a running process kept executing old bytecode and ignoring source fixes. find src -name __pycache__ -type d -exec rm -rf {} + when a fix "isn't taking."

Clustering: two-Spark pipeline-parallel inference (vLLM)

Running the 157 GB REAP-pruned Hy3 (too big for one Spark, fits across two) with vllm serve --pipeline-parallel-size 2 over Ray + the 200G fabric. The model loads fine; the friction is all environment (2026-07-08):

  • Copy the venv, don't rebuild. The Sparks are mirror-image (same aarch64/GB10, same /home/joemuller paths, Python 3.12), so rsync -a of the working ~/venvs/vllm to the peer gives a functioning vLLM+Ray+torch with no compile — the .so/paths all resolve. Saved a from-scratch vLLM build.

  • vLLM 0.24.0 supports HYV3ForCausalLM natively (in the model registry).

  • NCCL over RoCE isn't turnkey for vLLM — PP group init died with "NCCL error: unhandled system error". NCCL_IB_DISABLE=1 + NCCL_SOCKET_IFNAME= the fabric netdev forces TCP over the 200G link, which is plenty for PP's light inter-stage traffic. (RoCE ib_write_bw works; wiring it into NCCL is a separate GID/HCA tuning job not worth it for a functional check.)

  • ninja must be on the Ray worker's PATH — flashinfer JIT-compiles a CUTLASS fused-MoE kernel (gen_cutlass_fused_moe_sm120) on first forward and shells out to ninja (needs nvcc too — both present under /usr/local/cuda

    • a pip install ninja). Ray raylets inherit the login environment, and actors get PATH from there — not from the ray start shell's exports. Fix: symlink ninja into ~/.local/bin (the stock ~/.profile adds it when the dir exists) and start raylets via login shells (bash -lc). Verify with a @ray.remote probe of shutil.which("ninja") on each node.

  • Disable Ray's memory monitor on unified memory — RAY_memory_monitor_refresh_ms=0. It counts the 75 GB of GPU weights as system RAM, decides the node is full, and OOM-kills a worker after the model already loaded and KV cache allocated.

  • Hard-reset Ray between failed attempts — killed vLLM runs leak placement groups that keep the GPUs "reserved", so the next launch hangs on "Waiting for creating a placement group". ray stop --force + pkill -9 -f 'EngineCore| RayWorkerProc' on both nodes, then confirm ray status shows 0.0/2.0 GPU with no "reserved in placement groups".

  • Gloo needs the same interface pinning as NCCL for multi-node vLLM (2026-07-09). PP2 launch died with Gloo connectFullMesh failed ... Connection refused, remote=[127.0.0.1] — the Sparks resolve their own hostnames to loopback, so torch's CPU (gloo) process group advertises 127.0.0.1 to the peer. Fix: GLOO_SOCKET_IFNAME=enp1s0f1np1 + per-node VLLM_HOST_IP=<fabric ip> on both raylets and the vllm serve process (raylets pass env to Ray actors; setting it only on the server is not enough).

  • SM120 SASS cubins run unmodified on GB10/sm_121 (2026-07-09). The vLLM-Moet hand-written kernels (2-bit MoE QMMA.SF GEMMs, assembled by cubit with a mercury compat stub) load via cuModuleLoad and op-validate bit-deterministically at rel err 2-3e-3 on GB10 — CUDA treats 12.0/12.1 as one SASS family. Don't assume "SM120-only" kernel projects exclude Spark; test with a 30-line driver-API loader first.

  • DeepGEMM nv-dev builds clean on aarch64 (a6b593d2, ~7 min, needs only pybind11/wheel + system CUDA 13 headers). The vLLM-0.24-vendored copy lacks family-120 host paths; the site-packages wheel shadows it.

  • flashinfer-jit-cache cu130 wheels are x86-only — on Spark, flashinfer 0.6.14 falls back to JIT-compiling every kernel on first use (slow first request; same ninja/nvcc PATH caveats as the Ray note above).

  • Pinned-host "offload" tiers double-allocate on unified memory — designs that keep a pinned-host weight store + a GPU cache pool (vLLM-Moet's delta & base-cache tiers) allocate the same physical LPDDR twice on GB10. Run such stacks in fully-resident mode with the offload tiers disabled (VLLM_MOE_W2_DELTA_GB=0, no BASE_CACHE_GB); "GPU-resident" already means "in the one memory pool".

  • The GB10 unified-memory load war (2026-07-09, vLLM-Moet port; full details in ~/Dev/vLLM-Moet branch spark-gb10). Getting a 149 GiB checkpoint converted+loaded on a 121 GiB unified box surfaced six distinct platform behaviors, each capable of OOMing the load on its own:

    1. cudaMalloc takes system RAM but does not trigger page-cache reclaim — streaming a big checkpoint fills the cache and starves the allocator ("754 MiB free" CUDA OOM with 40+ GiB of clean cache). Fix: posix_fadvise(DONTNEED) consumed shards during load.
    2. Pageable H2D copies are CPU-executed writes into UVM — destination pool pages become host-resident and device-backed, and allocator churn accumulates them (~2-3 GiB/layer). Fix: pinned bounce + one persistent device buffer per transfer shape.
    3. cudaHostAlloc (pinned) is shmem-backed with power-of-2 rounding — a 20 GiB pinned ask costs ~33 GiB physical.
    4. Plain pageable host memory is readable by GPU kernels via ATS at full bandwidth — the winning place to keep big read-only weights.
    5. glibc arena free-lists retain tens of GiB of ~10 MB loader buffers; MALLOC_MMAP_THRESHOLD_=65536 returns them.
    6. The kernel OOM killer will take the desktop/session with it — run model engines under systemd-run --user --scope -p MemoryMax=... (a cgroup MemoryHigh throttle can stall a loader indefinitely; prefer a hard MemoryMax). The durable architecture: convert weights offline once (prepacked planes on disk) and serve-load into pageable host memory — no staging, no requant, no churn.

  • vLLM KV sizing on GB10 counts system-wide usage: budget = gpu_memory_utilization × total − used, where "used" includes host allocations (even other processes). Tune util to the system picture, not the model's CUDA footprint.

  • DGX OS ships memlock hard-limited to 8 MB — this is why NCCL-over-RoCE "mysteriously" fails (2026-07-10). ibv_reg_mr (RDMA memory registration) needs locked pages; NCCL surfaces the 8 MB ceiling as "unhandled system error" at init. Fix: limits.d memlock unlimited + start the process tree from a FRESH login session (inherited shells keep the old limit; self-ssh works). Solved the hy3-era mystery.

  • DGX OS also starts sessions with PR_SET_THP_DISABLE — madvise'd transparent hugepages silently no-op (THP_enabled: 0 in /proc/self/status). One unprivileged prctl(41, 0) per process re-enables. Also: CONFIG_READ_ONLY_THP_FOR_FS is not set, so file-backed THP (MADV_COLLAPSE on mmap'd weights) is unavailable on the stock kernel.

  • vLLM decode-step profiling on 2-node PP (--profiler-config '{"profiler":"torch",...}' + VLLM_SERVER_DEV_MODE): under CUDA graphs, in-graph kernels don't emit individual trace events — bucket totals still attribute wall time. GLM-5.2 PP2 decode: ~30% expert GEMMs, ~70% comm/pipeline-wait — which is why kernel-side tuning moved little and RoCE/batching moved more.