Laguna-M.1 NVFP4 (poolside)

No measured runs yet — be the first: install the spark-benchmark skill.

Quick start

1. 1

   Enable user lingering + free the nodes (preflight, once per box)

   TP2 places ranks with Ray, and the raylets run as systemd --user services. Without lingering, the user manager (and every raylet) is torn down the moment the last SSH session closes, and vLLM then hangs forever in 'Waiting for creating a placement group'. Enable lingering once, then start from a clean slate.

   # once per box
   ssh 192.168.100.1 'loginctl enable-linger $USER'
   ssh 192.168.100.2 'loginctl enable-linger $USER'
   
   # clean slate before serving
   pkill -f '[v]llm serve' 2>/dev/null
   ssh 192.168.100.2 "pkill -f '[v]llm serve' 2>/dev/null; true"bash

   Verify with `loginctl show-user $USER | grep Linger` → Linger=yes on both nodes. Do NOT purge the page cache here: this recipe pins the KV pool with --kv-cache-memory-bytes, which skips vLLM's free-memory check, so a purge buys nothing and only makes the 28-shard load slower.

2. 2

   Stage the NVFP4 checkpoint on both nodes

   Each TP rank reads all shards, so both nodes need the complete tree at an identical path. Spark-2's default route is wifi, so download once on node 1 and mirror over the direct RoCE link rather than pulling 139 GB twice.

   # node 1 — 139.32 GB, 28 shards (~23 min at ~110 MB/s)
   hf download poolside/Laguna-M.1-NVFP4 --local-dir ~/models/hf/Laguna-M.1-NVFP4
   
   # mirror to node 2 over the direct RoCE link (~820 MB/s)
   rsync -aL --partial --inplace \
     -e 'ssh -c aes128-gcm@openssh.com -o Compression=no' \
     ~/models/hf/Laguna-M.1-NVFP4/ 192.168.100.2:~/models/hf/Laguna-M.1-NVFP4/bash

   Verify both nodes report 39 files and zero *.incomplete before serving — a partial tree fails several minutes into the load. There is no draft model to stage: none exists for this target.

3. 3

   Bring up the 2-node Ray cluster (raylets as detached services)

   Start each raylet as a systemd --user *service* (not --scope): pass the fabric env via --setenv, raise LimitMEMLOCK for RDMA, and put ninja + nvcc on PATH. MAX_JOBS=4 is load-bearing and must live here rather than on the serve command — the Ray workers do the kernel JIT, and they inherit the raylet's environment, not the driver's.

   VENV=$HOME/venvs/vllm-026
   ENV='--setenv=PATH='$HOME'/venvs/vllm-026/bin:'$HOME'/.local/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
     --setenv=NCCL_IB_DISABLE=0 --setenv=NCCL_IB_HCA=rocep1s0f1 \
     --setenv=NCCL_SOCKET_IFNAME=enp1s0f1np1 --setenv=GLOO_SOCKET_IFNAME=enp1s0f1np1 \
     --setenv=RAY_memory_monitor_refresh_ms=0 --setenv=MAX_JOBS=4'
   
   # each node's RoCEv2 IPv4 GID index differs and drifts across reboots — resolve it
   gid() { ssh $1 "for g in /sys/class/infiniband/rocep1s0f1/ports/1/gids/*; do i=\${g##*/}; \
     [ \"\$(cat /sys/class/infiniband/rocep1s0f1/ports/1/gid_attrs/types/\$i 2>/dev/null)\" = 'RoCE v2' ] || continue; \
     case \$(cat \$g) in *ffff:$(printf '%02x%02x:%02x%02x' $(echo $2|tr . ' ')) ) echo \$i; break;; esac; done"; }
   GID1=$(gid 192.168.100.1 192.168.100.1); GID2=$(gid 192.168.100.2 192.168.100.2)
   
   ssh 192.168.100.1 "systemd-run --user --collect --unit=lagm1-ray-head $ENV \
     --setenv=VLLM_HOST_IP=192.168.100.1 --setenv=NCCL_IB_GID_INDEX=$GID1 \
     -p MemoryMax=102G -p MemorySwapMax=0 -p LimitMEMLOCK=infinity \
     $VENV/bin/ray start --block --head --node-ip-address=192.168.100.1 --port=6379 --object-store-memory=2000000000"
   sleep 10
   ssh 192.168.100.2 "systemd-run --user --collect --unit=lagm1-ray-worker $ENV \
     --setenv=VLLM_HOST_IP=192.168.100.2 --setenv=NCCL_IB_GID_INDEX=$GID2 \
     -p MemoryMax=110G -p MemorySwapMax=0 -p LimitMEMLOCK=infinity \
     $VENV/bin/ray start --block --address=192.168.100.1:6379 --object-store-memory=2000000000 --node-ip-address=192.168.100.2"
   sleep 12
   $VENV/bin/ray status   # expect 2 nodes, 2.0 GPUbash

   --block keeps `ray start` as the service's main process so the unit stays active. If `ray status` shows 1.0 GPU the worker dropped — check lingering, and that the worker unit is active (`systemctl --user is-active lagm1-ray-worker` on .2).

4. 4

   Serve TP2 with a pinned KV pool

   --enforce-eager is required: cross-node cudagraph replay deadlocks on GB10. --kv-cache-memory-bytes pins the pool deterministically instead of letting vLLM's utilization math guess on unified memory, and 16 GiB/rank is what the 196,608-token window costs with a margin. --max-num-batched-tokens 4096 bounds the chunked-prefill transient, which matters on a box this close to full.

   VENV=$HOME/venvs/vllm-026
   export VLLM_HOST_IP=192.168.100.1
   export NCCL_SOCKET_IFNAME=enp1s0f1np1 GLOO_SOCKET_IFNAME=enp1s0f1np1
   export NCCL_IB_DISABLE=0 NCCL_IB_HCA=rocep1s0f1 NCCL_IB_GID_INDEX=$GID1
   export RAY_memory_monitor_refresh_ms=0
   
   systemd-run --user --scope --collect -p MemoryMax=12G -p MemorySwapMax=0 \
     $VENV/bin/vllm serve $HOME/models/hf/Laguna-M.1-NVFP4 \
     --served-model-name laguna-m1-nvfp4 \
     --distributed-executor-backend ray --tensor-parallel-size 2 \
     --enforce-eager \
     --max-model-len 196608 \
     --gpu-memory-utilization 0.85 \
     --kv-cache-memory-bytes 17179869184 \
     --max-num-seqs 2 --max-num-batched-tokens 4096 \
     --host 0.0.0.0 --port 8000bash

   Budget ~25 minutes for a first start on a fresh venv: 28 shards load in ~8 min per rank ('Model loading took 64.98 GiB memory'), then the NVFP4 MoE kernel JIT-compiles for another ~15. Later boots are much faster (cached under ~/.cache/flashinfer). Expect the log to confirm 'Using FLASHINFER attention backend', "Using 'FLASHINFER_CUTLASS' NvFp4 MoE backend" and 'GPU KV cache size: 239,664 tokens'.

5. 5

   Smoke test, warm up, then read the pool and memory

   Confirm coherence, then warm the kernels before believing any throughput number — a cold pass faults expert planes in from NVMe. Read steady-state memory only once the API is up; mid-load figures flatter you by exactly the KV pool you are about to allocate.

   curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
     -d '{"model":"laguna-m1-nvfp4","messages":[{"role":"user","content":"capital of France? one word"}],"max_tokens":8,"temperature":0}'
   # -> "Paris"
   
   # a couple of 300-token throwaway generations to warm the kernels, then measure
   free -g | sed -n '2p'                       # head ~101 GiB used
   ssh 192.168.100.2 "free -g | sed -n '2p'"   # worker ~96 GiB usedbash

   A 300-token warmup generation takes ~24 s, which is the honest feel of this config: ~12.5 tok/s. If your first benchmark pass shows the 512-token prompt slower than the 2,048-token one, the run paid a JIT cost and the whole pass is suspect — warm and re-run.

Revision history

What has changed on this page since it was published, and what it measured. Newest first.

1. 2026-07-28

   re-measured

   First publication: measured bring-up of poolside/Laguna-M.1-NVFP4 on two Sparks over TP2, vLLM 0.26.0.

   12.52 tok/s single-stream decode at a 2,048-token code prompt (warm), 196,608-token context against a measured 239,664-token FP8 KV pool.